spark-bigquery: fix a few of the common errors #1377
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
ci
client/java
integration/spark
tests
mobuchowski
force-pushed
the
bigquery/do-not-create-relation-on-output
branch
2 times, most recently
from
58b8338
to
33998a8
Compare
integration/spark/app/src/main/java/io/openlineage/spark/agent/OpenLineageSparkListener.java
Show resolved
Hide resolved
...on/spark/app/src/test/java/io/openlineage/spark/agent/lifecycle/SparkReadWriteIntegTest.java
Outdated
Show resolved
Hide resolved
integration/spark/app/src/test/resources/spark_scripts/spark_bigquery.py
Show resolved
Hide resolved
| @@ -45,7 +45,7 @@ configurations { | |||
|
|
|||
| ext { | |||
| assertjVersion = '3.23.1' | |||
| bigqueryVersion = '0.26.0' | |||
| bigqueryVersion = '0.22.0' | |||
There was a problem hiding this comment.
Any reason for that?
There was a problem hiding this comment.
We want to use same version of dependency as baseline everywhere in the code.
There was a problem hiding this comment.
sure, there are just two places within codebase where this version is kept. Upgrading the other location is a preferred solution I guess.
There was a problem hiding this comment.
@pawel-big-lebowski moved to 0.26.0.
...shared/src/main/java/io/openlineage/spark/agent/lifecycle/plan/BigQueryNodeInputVisitor.java
Show resolved
Hide resolved
mobuchowski
force-pushed
the
bigquery/do-not-create-relation-on-output
branch
12 times, most recently
from
136aaf6
to
3ebf91d
Compare
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
mobuchowski
force-pushed
the
bigquery/do-not-create-relation-on-output
branch
3 times, most recently
from
1028869
to
056a017
Compare
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
mobuchowski
force-pushed
the
bigquery/do-not-create-relation-on-output
branch
from
056a017
to
6f008ad
Compare
pawel-big-lebowski
approved these changes
Dec 8, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes few of the common issues with spark-bigquery integration and adds integration test for it, together with the CI configuration for it.
spark-bigquerydependencies -spark-bigqueryitself, andspark-bigquery-with-dependencies. They aren't compatible - for example we usecom.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.connector.common.BigQueryUtilwhich isgoogle.cloud.bigquery.connector.common.BigQueryUtilin the non-with-dependencies version. Previously we mixed those two, now we use with-dependencies version everywhere as it's 10x more popular on maven central and recommended everywhereBigQueryNodeVisitortoBigQueryInputNodeVisitorandBigQueryOutputNodeVisitor. The "output" visitors process only root node, while "input" ones process whole tree. If we have one node visitor who is registered both for input and output events, the root node is double counter. The split prevents that.SaveIntoDataSourceCommandVisitorfrom processingBigQueryRelation.BigQueryOutputNodeVisitordoes not try to createBigQueryRelationto get proper table name, but utilizesBigQueryUtil.friendlyTableName(config.getTableId()). This prevents error when output table does not exist and is created by the spark process.