convert destination-bigquery to kotlin CDK #36899

stephane-airbyte · 2024-04-08T19:14:14Z

No description provided.

vercel · 2024-04-08T19:14:20Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 25, 2024 9:37pm

stephane-airbyte · 2024-04-08T19:14:28Z

convert destination-bigquery to kotlin CDK #36899 👈
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @stephane-airbyte and the rest of your teammates on Graphite

.../main/kotlin/io/airbyte/integrations/base/destination/typing_deduping/DefaultTyperDeduper.kt

stephane-airbyte · 2024-04-08T19:16:45Z

...-bigquery/src/main/java/io/airbyte/integrations/destination/bigquery/BigQueryAsyncFlush.java

@@ -48,7 +48,7 @@ public void flush(final StreamDescriptor decs, final Stream<PartialAirbyteMessag

      stream.forEach(record -> {
        try {
-          writer.accept(record.getSerialized(), record.getRecord().getEmittedAt());
+          writer.accept(record.getSerialized(), Jsons.serialize(record.getRecord().getMeta()), record.getRecord().getEmittedAt());


@gisripa I think this is what I need to do, but I'd like you to confirm that

looks correct to me. One thing to verify is if BigQuery is ok adding extra stuff in the CSV file, we had to pass a flag in snowflake to ignore this meta information until we add the migration code to populate meta in raw table.

I think that my addition of V2_WITH_META means that this value will be ignored when writing the CSV

yeah looks like it. seems ok, when we introduce meta in BQ this will work.

airbyte-ci/connectors/pipelines/tests/test_format/non_formatted_code/java.java

edgao

not sure what's up with most of the test failures (... they're claiming that we failed to write the data correctly, which sounds sketchy)

but at least this one I think is just an unrelated breaking change in the cdk - lmk if you want a hand with it, but you should be able to mostly copy these files from e.g. redshift's test resources

BigQuerySqlGeneratorIntegrationTest > testV1V2migration() FAILED
    org.gradle.internal.exceptions.DefaultMultiCauseException: Multiple Failures (2 failures)
    	java.lang.IllegalArgumentException: resource sqlgenerator/alltypes_v1v2_expectedrecords_raw.jsonl not found.
    	java.lang.IllegalArgumentException: resource sqlgenerator/alltypes_v1v2_expectedrecords_final.jsonl not found.

edgao · 2024-04-09T18:35:00Z

...n/kotlin/io/airbyte/cdk/integrations/destination/buffered_stream_consumer/OnCloseFunction.kt

+ *
+ * The @JvmSuppressWildcards is here so that the 2nd parameter of accept stays a java
+ * Map<StreamDescriptor, StreamSyncSummary>
+ * ```


nit: this formatting looks off? (also TIL JvmSuppressWildcards)

gisripa · 2024-04-09T20:25:27Z

they're claiming that we failed to write the data correctly

The failures seems pretty much what we encountered in Snowflake where adding additional data in CSV (airbyte_meta added in base Writer class) which doesn't have a mapped column in table seems to be failing to insert any of the data. So we are silently failing like snowflake when csv rows doesn't match columns size ?

gisripa · 2024-04-09T20:30:14Z

https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv
--ignore_unknown_values I guess is the temporary fix until we introduce meta ?

stephane-airbyte · 2024-04-25T17:53:26Z

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/8837157547
❌ Publish Java CDK version=0.30.11 failed!

gisripa · 2024-04-25T19:20:41Z

...dk/core/src/main/kotlin/io/airbyte/cdk/integrations/destination/async/AsyncStreamConsumer.kt

+        defaultNamespace = defaultNamespace,
+        flushFailure = FlushFailure(),
+        workerPool = workerPool,
+    )


I aggressively removed the overloaded constructors hoping the 2 usages were already in Kotlin with defaults injected. Here it is yet another Java usage 🤦‍♂️

yep, I noticed your removal 😠 😏

edgao

one nit, otherwise lgtm

.../io/airbyte/integrations/base/destination/typing_deduping/BaseSqlGeneratorIntegrationTest.kt

gisripa

lgtm.
nit: any chance we can avoid that overloaded constructor in AsyncStreamConsumer, even if it means copy-pasting the required ones from StagingConsumerFactory into BQ land ?

stephane-airbyte · 2024-04-25T20:03:38Z

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/8838668760
✅ Successfully published Java CDK version=0.30.11!

stephane-airbyte · 2024-04-25T20:09:39Z

/publish-java-cdk force=true

🕑 https://github.com/airbytehq/airbyte/actions/runs/8838745192
✅ Successfully published Java CDK version=0.30.11!

stephane-airbyte · 2024-04-25T21:46:49Z

Merge activity

Apr 25, 5:46 PM EDT: @stephane-airbyte merged this pull request with Graphite.

octavia-squidington-iii added area/connectors Connector related issues CDK Connector Development Kit connectors/destination/bigquery labels Apr 8, 2024

stephane-airbyte marked this pull request as ready for review April 8, 2024 19:14

stephane-airbyte requested review from a team as code owners April 8, 2024 19:14

stephane-airbyte requested review from colesnodgrass, edgao, gisripa and jbfbell April 8, 2024 19:15

stephane-airbyte commented Apr 8, 2024

View reviewed changes

.../main/kotlin/io/airbyte/integrations/base/destination/typing_deduping/DefaultTyperDeduper.kt Outdated Show resolved Hide resolved

stephane-airbyte commented Apr 8, 2024

View reviewed changes

stephane-airbyte force-pushed the stephane/04-08-convert_destination-bigquery_to_kotlin_cdk branch from 1467569 to a02d678 Compare April 8, 2024 21:52

stephane-airbyte requested a review from a team as a code owner April 8, 2024 21:52

stephane-airbyte force-pushed the stephane/04-08-convert_destination-bigquery_to_kotlin_cdk branch 2 times, most recently from 4fea7a9 to c2074de Compare April 8, 2024 22:49

gisripa reviewed Apr 9, 2024

View reviewed changes

airbyte-ci/connectors/pipelines/tests/test_format/non_formatted_code/java.java Outdated Show resolved Hide resolved

stephane-airbyte force-pushed the stephane/04-08-convert_destination-bigquery_to_kotlin_cdk branch 2 times, most recently from 0d28feb to 3598443 Compare April 9, 2024 18:20

edgao reviewed Apr 9, 2024

View reviewed changes

stephane-airbyte force-pushed the stephane/04-08-convert_destination-bigquery_to_kotlin_cdk branch 6 times, most recently from 9a617ca to 0b9fa0f Compare April 10, 2024 16:16

stephane-airbyte force-pushed the stephane/04-08-convert_destination-bigquery_to_kotlin_cdk branch 3 times, most recently from b456f57 to 41db851 Compare April 25, 2024 17:51

stephane-airbyte force-pushed the stephane/04-08-convert_destination-bigquery_to_kotlin_cdk branch 3 times, most recently from 2584ef3 to 1a27dfe Compare April 25, 2024 18:37

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Apr 25, 2024

vercel bot deployed to Preview April 25, 2024 18:41 View deployment

gisripa reviewed Apr 25, 2024

View reviewed changes

edgao approved these changes Apr 25, 2024

View reviewed changes

.../io/airbyte/integrations/base/destination/typing_deduping/BaseSqlGeneratorIntegrationTest.kt Show resolved Hide resolved

gisripa approved these changes Apr 25, 2024

View reviewed changes

evantahler mentioned this pull request Apr 25, 2024

Exclude zookeeper dependencies #37584

Merged

stephane-airbyte force-pushed the stephane/04-08-convert_destination-bigquery_to_kotlin_cdk branch from 1a27dfe to 226ee35 Compare April 25, 2024 19:50

vercel bot deployed to Preview April 25, 2024 19:54 View deployment

stephane-airbyte force-pushed the stephane/04-08-convert_destination-bigquery_to_kotlin_cdk branch from 226ee35 to d60c192 Compare April 25, 2024 20:07

vercel bot deployed to Preview April 25, 2024 20:11 View deployment

stephane-airbyte force-pushed the stephane/04-08-convert_destination-bigquery_to_kotlin_cdk branch from d60c192 to 5139ad8 Compare April 25, 2024 20:19

vercel bot deployed to Preview April 25, 2024 20:23 View deployment

stephane-airbyte force-pushed the stephane/04-08-convert_destination-bigquery_to_kotlin_cdk branch from 5139ad8 to b5ffe32 Compare April 25, 2024 21:12

vercel bot deployed to Preview April 25, 2024 21:16 View deployment

convert destination-bigquery to kotlin CDK

7a68dfd

stephane-airbyte force-pushed the stephane/04-08-convert_destination-bigquery_to_kotlin_cdk branch from b5ffe32 to 7a68dfd Compare April 25, 2024 21:33

vercel bot deployed to Preview April 25, 2024 21:37 View deployment

stephane-airbyte merged commit c4ad3d9 into master Apr 25, 2024
34 checks passed

stephane-airbyte deleted the stephane/04-08-convert_destination-bigquery_to_kotlin_cdk branch April 25, 2024 21:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

convert destination-bigquery to kotlin CDK #36899

convert destination-bigquery to kotlin CDK #36899

stephane-airbyte commented Apr 8, 2024

vercel bot commented Apr 8, 2024 •

edited

Loading

stephane-airbyte commented Apr 8, 2024

stephane-airbyte Apr 8, 2024

gisripa Apr 8, 2024

stephane-airbyte Apr 25, 2024

gisripa Apr 25, 2024

edgao left a comment

edgao Apr 9, 2024

gisripa commented Apr 9, 2024 •

edited

Loading

gisripa commented Apr 9, 2024

stephane-airbyte commented Apr 25, 2024 •

edited by github-actions bot

Loading

gisripa Apr 25, 2024

stephane-airbyte Apr 25, 2024

edgao left a comment

gisripa left a comment

stephane-airbyte commented Apr 25, 2024 •

edited by github-actions bot

Loading

stephane-airbyte commented Apr 25, 2024 •

edited by github-actions bot

Loading

stephane-airbyte commented Apr 25, 2024

convert destination-bigquery to kotlin CDK #36899

convert destination-bigquery to kotlin CDK #36899

Conversation

stephane-airbyte commented Apr 8, 2024

vercel bot commented Apr 8, 2024 • edited Loading

stephane-airbyte commented Apr 8, 2024

stephane-airbyte Apr 8, 2024

Choose a reason for hiding this comment

gisripa Apr 8, 2024

Choose a reason for hiding this comment

stephane-airbyte Apr 25, 2024

Choose a reason for hiding this comment

gisripa Apr 25, 2024

Choose a reason for hiding this comment

edgao left a comment

Choose a reason for hiding this comment

edgao Apr 9, 2024

Choose a reason for hiding this comment

gisripa commented Apr 9, 2024 • edited Loading

gisripa commented Apr 9, 2024

stephane-airbyte commented Apr 25, 2024 • edited by github-actions bot Loading

gisripa Apr 25, 2024

Choose a reason for hiding this comment

stephane-airbyte Apr 25, 2024

Choose a reason for hiding this comment

edgao left a comment

Choose a reason for hiding this comment

gisripa left a comment

Choose a reason for hiding this comment

stephane-airbyte commented Apr 25, 2024 • edited by github-actions bot Loading

stephane-airbyte commented Apr 25, 2024 • edited by github-actions bot Loading

stephane-airbyte commented Apr 25, 2024

Merge activity

vercel bot commented Apr 8, 2024 •

edited

Loading

gisripa commented Apr 9, 2024 •

edited

Loading

stephane-airbyte commented Apr 25, 2024 •

edited by github-actions bot

Loading

stephane-airbyte commented Apr 25, 2024 •

edited by github-actions bot

Loading

stephane-airbyte commented Apr 25, 2024 •

edited by github-actions bot

Loading