✨ Destinations bigquery+snowflake: Release DV2 by edgao · Pull Request #29783 · airbytehq/airbyte

Edward Gao (edgao) · 2023-08-23T23:19:22Z

⚠️ 💀 DO NOT MERGE UNTIL WE ARE CLEARED FOR TAKEOFF 💀 ⚠️

closes #29728. Markdown/yaml changes largely taken from #29683.

Reviewers: Read each commit in sequence. The first two commits (d9aa4d1 + 4d10768) are basically just git checkout evan/bigquery-snowflake-v2 -- <stuff>; the rest are more interesting.

presumably, if anything in here is broken, CI will fail. Not running any interesting tests locally. I did verify that the docker images no longer contain normalization, though.

github-actions · 2023-08-23T23:19:45Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

This code isn't actually used by destination-bigquery. It's only used by destinations that write actual files (e.g. s3/gcs).

disable dv2 for redshift; enable it for snowflake bigquery isn't on this code path because bigquery doesn't use CSV

Evan Tahler (evantahler) · 2023-08-24T00:32:27Z

Pre-release: https://github.com/airbytehq/airbyte/actions/runs/5957951679

Evan Tahler (evantahler) · 2023-08-28T20:50:56Z

Pre-release: https://github.com/airbytehq/airbyte/actions/runs/6004809490

github-actions · 2023-08-28T21:35:43Z

destination-redshift test report (commit `723a13f313`) - ❌

⏲️ Total pipeline duration: 43mn39s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-redshift docker image for platform linux/x86_64	✅
Build airbyte/normalization-redshift:dev	✅
Java Connector Integration Tests	❌
Validate airbyte-integrations/connectors/destination-redshift/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

github-actions · 2023-08-28T21:45:58Z

destination-snowflake test report (commit `723a13f313`) - ✅

⏲️ Total pipeline duration: 10mn02s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

github-actions · 2023-08-28T22:04:48Z

destination-bigquery-denormalized test report (commit `723a13f313`) - ❌

⏲️ Total pipeline duration: 18mn39s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

github-actions · 2023-08-28T22:18:25Z

destination-bigquery test report (commit `723a13f313`) - ✅

⏲️ Total pipeline duration: 13mn24s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

github-actions · 2023-08-28T22:32:44Z

destination-gcs test report (commit `723a13f313`) - ❌

⏲️ Total pipeline duration: 14mn07s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-gcs docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-gcs/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-gcs test

Joe Bell (jbfbell) · 2023-08-29T17:52:37Z


  public GenericData.Record getAvroRecord(final UUID id, final AirbyteRecordMessage recordMessage) throws JsonProcessingException {
    final ObjectNode jsonRecord = MAPPER.createObjectNode();
-    if (TypingAndDedupingFlag.isDestinationV2()) {


I think the wrong block of this conditional is kept?

this is intentional - this file isn't actually used by dv2, it's only used by real file destinations (s3/gcs). e160ad6

Joe Bell (jbfbell) · 2023-08-29T18:00:48Z

  public StagingDatabaseCsvSheetGenerator() {
-    use1s1t = TypingAndDedupingFlag.isDestinationV2();
-    this.header = use1s1t ? JavaBaseConstants.V2_RAW_TABLE_COLUMN_NAMES : JavaBaseConstants.LEGACY_RAW_TABLE_COLUMNS;
+    this(false);


this is to maintain v1 behavior in old destinations (i.e. redshift I think?). V2 destinations are all using the new constructor and explicitly passing true.

github-actions · 2023-08-29T18:41:11Z

destination-redshift test report (commit `978220df1a`) - ❌

⏲️ Total pipeline duration: 39mn53s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-redshift docker image for platform linux/x86_64	✅
Build airbyte/normalization-redshift:dev	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-redshift/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

github-actions · 2023-08-29T18:55:03Z

destination-gcs test report (commit `978220df1a`) - ❌

⏲️ Total pipeline duration: 13mn41s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-gcs docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-gcs/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-gcs test

Edward Gao (edgao) · 2023-08-29T19:06:14Z

/approve-and-merge reason="preflight checks all green, let's do this"

Octavia The Approver (octavia-approvington) · 2023-08-29T19:06:48Z

Lets do it!

github-actions · 2023-08-29T19:07:52Z

destination-snowflake test report (commit `978220df1a`) - ✅

⏲️ Total pipeline duration: 12mn37s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

Edward Gao (edgao) · 2023-08-29T19:08:06Z

(for real though: intentionally not publishing gcs/s3, and bigquery-denormalized is going away. snowflake+bigquery are green.)

github-actions · 2023-08-29T19:22:11Z

destination-bigquery test report (commit `978220df1a`) - ❌

⏲️ Total pipeline duration: 14mn08s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

github-actions · 2023-08-29T19:41:14Z

destination-bigquery-denormalized test report (commit `978220df1a`) - ❌

⏲️ Total pipeline duration: 18mn51s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

github-actions · 2023-08-29T19:47:24Z

destination-snowflake test report (commit `c8550871dc`) - ❌

⏲️ Total pipeline duration: 03mn42s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

github-actions · 2023-08-29T19:50:53Z

destination-bigquery test report (commit `c8550871dc`) - ❌

⏲️ Total pipeline duration: 03mn13s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	❌
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

github-actions · 2023-08-29T19:55:13Z

destination-redshift test report (commit `c8550871dc`) - ❌

⏲️ Total pipeline duration: 04mn04s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-redshift docker image for platform linux/x86_64	✅
Build airbyte/normalization-redshift:dev	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-redshift/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

github-actions · 2023-08-29T20:08:33Z

destination-bigquery-denormalized test report (commit `c8550871dc`) - ❌

⏲️ Total pipeline duration: 13mn09s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

github-actions · 2023-08-29T20:11:25Z

destination-gcs test report (commit `c8550871dc`) - ❌

⏲️ Total pipeline duration: 02mn35s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-gcs docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-gcs/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-gcs test

Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: Evan Tahler <evan@airbyte.io> Co-authored-by: evantahler <evantahler@users.noreply.github.com>

Edward Gao (edgao) force-pushed the edgao/dv2/release branch from ddfee9e to a78a304 Compare August 23, 2023 23:21

Octavia Squidington III (octavia-squidington-iii) removed the CDK Connector Development Kit label Aug 23, 2023

Edward Gao (edgao) changed the title ~~Destinations bigquery+snowflake: Release DV2~~ ✨ Destinations bigquery+snowflake: Release DV2 Aug 23, 2023

Edward Gao (edgao) added the checklist-action-run label Aug 23, 2023

Edward Gao (edgao) added 8 commits August 23, 2023 16:23

copy spec changes

d9aa4d1

logistics

4d10768

remove normalization from build

5e0f08f

remove unnecessary change

e160ad6

This code isn't actually used by destination-bigquery. It's only used by destinations that write actual files (e.g. s3/gcs).

inject param to stagingcsvgenerator+stagingconsumerfactory

dd0a8db

disable dv2 for redshift; enable it for snowflake bigquery isn't on this code path because bigquery doesn't use CSV

inject to GcsUtils

0e1f518

hardcode snowflake

b5ae4d3

hardcode in bigquery

0c8248d

Edward Gao (edgao) force-pushed the edgao/dv2/release branch from a78a304 to 0c8248d Compare August 23, 2023 23:26

Edward Gao (edgao) marked this pull request as ready for review August 23, 2023 23:27

Edward Gao (edgao) requested a review from a team as a code owner August 23, 2023 23:27

This comment was marked as outdated.

Sign in to view

Evan Tahler (evantahler) mentioned this pull request Aug 24, 2023

Snowflake and BigQuery destination V2 #29683

Closed

Evan Tahler (evantahler) added the needed-for-v2-release label Aug 24, 2023

This comment was marked as outdated.

Sign in to view

Merge branch 'master' into edgao/dv2/release

723a13f

Automated Commit - Formatting Changes

8ca2c27

Edward Gao (edgao) mentioned this pull request Aug 28, 2023

destinations v2: snowflake: single-threaded T+D per stream #29878

Merged

Edward Gao (edgao) added 2 commits August 29, 2023 10:57

Merge branch 'master' into edgao/dv2/release

d76df65

botched merge

978220d

Joe Bell (jbfbell) approved these changes Aug 29, 2023

View reviewed changes

Octavia The Approver (octavia-approvington) approved these changes Aug 29, 2023

View reviewed changes

Octavia The Approver (octavia-approvington) merged commit 44f840d into master Aug 29, 2023

Octavia The Approver (octavia-approvington) deleted the edgao/dv2/release branch August 29, 2023 19:06

Evan Tahler (evantahler) mentioned this pull request Aug 29, 2023

destination-bigquery v2.0.1 #29972

Merged

Conversation

Edward Gao (edgao) commented Aug 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 23, 2023 • edited by edgao Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before Merging a Connector Pull Request

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Evan Tahler (evantahler) commented Aug 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Evan Tahler (evantahler) commented Aug 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 28, 2023

destination-redshift test report (commit 723a13f313) - ❌

Uh oh!

github-actions Bot commented Aug 28, 2023

destination-snowflake test report (commit 723a13f313) - ✅

Uh oh!

github-actions Bot commented Aug 28, 2023

destination-bigquery-denormalized test report (commit 723a13f313) - ❌

Uh oh!

github-actions Bot commented Aug 28, 2023

destination-bigquery test report (commit 723a13f313) - ✅

Uh oh!

github-actions Bot commented Aug 28, 2023

destination-gcs test report (commit 723a13f313) - ❌

Uh oh!

Joe Bell (jbfbell) Aug 29, 2023

Choose a reason for hiding this comment

Uh oh!

Edward Gao (edgao) Aug 29, 2023

Choose a reason for hiding this comment

Uh oh!

Joe Bell (jbfbell) Aug 29, 2023

Choose a reason for hiding this comment

Uh oh!

Edward Gao (edgao) Aug 29, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Aug 29, 2023

destination-redshift test report (commit 978220df1a) - ❌

Uh oh!

github-actions Bot commented Aug 29, 2023

destination-gcs test report (commit 978220df1a) - ❌

Uh oh!

Edward Gao (edgao) commented Aug 29, 2023

Uh oh!

Octavia The Approver (octavia-approvington) commented Aug 29, 2023

Uh oh!

github-actions Bot commented Aug 29, 2023

destination-snowflake test report (commit 978220df1a) - ✅

Uh oh!

Edward Gao (edgao) commented Aug 29, 2023

Uh oh!

github-actions Bot commented Aug 29, 2023

destination-bigquery test report (commit 978220df1a) - ❌

Uh oh!

github-actions Bot commented Aug 29, 2023

destination-bigquery-denormalized test report (commit 978220df1a) - ❌

Uh oh!

github-actions Bot commented Aug 29, 2023

destination-snowflake test report (commit c8550871dc) - ❌

Uh oh!

github-actions Bot commented Aug 29, 2023

destination-bigquery test report (commit c8550871dc) - ❌

Uh oh!

github-actions Bot commented Aug 29, 2023

destination-redshift test report (commit c8550871dc) - ❌

Uh oh!

github-actions Bot commented Aug 29, 2023

destination-bigquery-denormalized test report (commit c8550871dc) - ❌

Uh oh!

github-actions Bot commented Aug 29, 2023

destination-gcs test report (commit c8550871dc) - ❌

Edward Gao (edgao) commented Aug 23, 2023 •

edited

Loading

github-actions Bot commented Aug 23, 2023 •

edited by edgao

Loading

Evan Tahler (evantahler) commented Aug 24, 2023 •

edited

Loading

Evan Tahler (evantahler) commented Aug 28, 2023 •

edited

Loading

destination-redshift test report (commit `723a13f313`) - ❌

destination-snowflake test report (commit `723a13f313`) - ✅

destination-bigquery-denormalized test report (commit `723a13f313`) - ❌

destination-bigquery test report (commit `723a13f313`) - ✅

destination-gcs test report (commit `723a13f313`) - ❌

destination-redshift test report (commit `978220df1a`) - ❌

destination-gcs test report (commit `978220df1a`) - ❌

destination-snowflake test report (commit `978220df1a`) - ✅

destination-bigquery test report (commit `978220df1a`) - ❌

destination-bigquery-denormalized test report (commit `978220df1a`) - ❌

destination-snowflake test report (commit `c8550871dc`) - ❌

destination-bigquery test report (commit `c8550871dc`) - ❌

destination-redshift test report (commit `c8550871dc`) - ❌

destination-bigquery-denormalized test report (commit `c8550871dc`) - ❌

destination-gcs test report (commit `c8550871dc`) - ❌