Snowflake and BigQuery destination V2 #29683

evantahler · 2023-08-21T21:08:41Z

💀 DO NOT MERGE ⚠️

Publish multiple connectors at once with

--name=destination-bigquery --name=destination-snowflake 
--pre-release
CHANGE THE BRANCH

To make a local registry:

download https://storage.googleapis.com/prod-airbyte-cloud-connector-metadata-service/registries/v0/oss_registry.json
replace the values you want
use it locally via CONNECTOR_REGISTRY_BASE_UR=path/to/json.json

Prereleases:

Date	BigQuery	Snowflake	Run
2023-08-21	airbyte/destination-bigquery:2.0.0-dev.b82ab8b780	airbyte/destination-snowflake:3.0.0-dev.b82ab8b780	https://github.com/airbytehq/airbyte/actions/runs/5931864372
2023-08-21	airbyte/destination-bigquery:2.0.0-dev.195ce56d4e	airbyte/destination-snowflake:3.0.0-dev.195ce56d4e	https://github.com/airbytehq/airbyte/actions/runs/5933074788
2023-08-23	airbyte/destination-snowflake:3.0.0-dev.912e60c225	airbyte/destination-bigquery:2.0.0-dev.912e60c225	https://github.com/airbytehq/airbyte/actions/runs/5955875481

github-actions · 2023-08-21T21:09:04Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

evantahler · 2023-08-21T21:09:24Z

docs/integrations/destinations/snowflake.md

+| 3.0.0           | 2023-08-27 | [xxx](https://github.com/airbytehq/airbyte/pull/xxx)       | Destinations V2                                                                                                                                     |
 | 2.0.0           | 2023-08-09 | [\#28894](https://github.com/airbytehq/airbyte/pull/29236) | Remove support for Snowflake GCS/S3 loading method in favor of Snowflake Internal staging                                                           |


Of note: Snowflake + Destinations V2 is actually "v3.0.0"

evantahler · 2023-08-21T21:37:26Z

...ations/bases/base-java/src/main/java/io/airbyte/integrations/base/TypingAndDedupingFlag.java

  public static boolean isDestinationV2() {
-    return DestinationConfig.getInstance().getBooleanValue("use_1s1t_format");
+    //TODO: Refactor this whole class away
+    // return DestinationConfig.getInstance().getBooleanValue("use_1s1t_format");
+    return true;
  }


This is smallest possible change to 'force' V2 destinations to be enabled.

github-actions · 2023-08-21T21:51:43Z

destination-bigquery test report (commit `a4657f1954`) - ❌

⏲️ Total pipeline duration: 40mn53s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	❌
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

github-actions · 2023-08-21T22:24:17Z

destination-snowflake test report (commit `a4657f1954`) - ❌

⏲️ Total pipeline duration: 32mn23s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	❌
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

github-actions · 2023-08-21T22:51:12Z

destination-snowflake test report (commit `b82ab8b780`) - ❌

⏲️ Total pipeline duration: 24mn44s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

edgao

do you also need to update run_with_normalization.sh to never run normalization? Or is that covered by the metadata change

also I think a lot of DATs are going to fail here, since they make assumptions about normalized tables existing.

airbyte-integrations/connectors/destination-bigquery/src/main/resources/spec.json

airbyte-integrations/connectors/destination-snowflake/src/main/resources/spec.json

github-actions · 2023-08-21T23:28:19Z

destination-bigquery test report (commit `b82ab8b780`) - ❌

⏲️ Total pipeline duration: 36mn57s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

evantahler · 2023-08-22T00:24:35Z

do you also need to update run_with_normalization.sh to never run normalization? Or is that covered by the metadata change

Good point! The platform should be good about not running a normalization container now that the destinations no longer have a normalizationConfig metadata (https://github.com/airbytehq/airbyte-platform-internal/pull/8314), but the normal READ container might still get the use_1s1t_format option from an old connection's config file.

edgao · 2023-08-22T00:32:49Z

I just realized that we should make a second copy of that script, if we want to do beta periods for other connectors (so we would have an always-dv2 script, and a script that supports both modes). Problem for next week though!

evantahler · 2023-08-22T00:44:01Z

You know what? The fine folks on Connector Ops already have us covered!

The build script only uses the /airbyte/run_with_normalization.sh entry point if context.connector.supports_normalization and DESTINATION_NORMALIZATION_BUILD_CONFIGURATION[context.connector.technical_name]["supports_in_connector_normalization"]. These new updates to the metadata don't pass those checks any more

github-actions · 2023-08-22T00:48:31Z

destination-snowflake test report (commit `71475e9d00`) - ❌

⏲️ Total pipeline duration: 27mn01s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

github-actions · 2023-08-22T01:29:53Z

destination-bigquery test report (commit `71475e9d00`) - ❌

⏲️ Total pipeline duration: 41mn11s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

github-actions · 2023-08-22T01:58:37Z

destination-snowflake test report (commit `195ce56d4e`) - ❌

⏲️ Total pipeline duration: 26mn18s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

github-actions · 2023-08-22T02:35:56Z

destination-bigquery test report (commit `195ce56d4e`) - ❌

⏲️ Total pipeline duration: 37mn08s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

github-actions · 2023-08-23T20:28:15Z

destination-snowflake test report (commit `912e60c225`) - ❌

⏲️ Total pipeline duration: 27mn42s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

github-actions · 2023-08-23T21:13:08Z

destination-bigquery test report (commit `912e60c225`) - ❌

⏲️ Total pipeline duration: 44mn41s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

evantahler · 2023-08-24T00:30:50Z

Closed in favor of #29783

Snowflake and BigQuery destination V2

a4657f1

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/destination/bigquery connectors/destination/snowflake labels Aug 21, 2023

evantahler commented Aug 21, 2023

View reviewed changes

evantahler marked this pull request as ready for review August 21, 2023 21:09

evantahler requested review from a team as code owners August 21, 2023 21:09

Merge branch 'master' into evan/bigquery-snowflake-v2

6cdf421

evantahler mentioned this pull request Aug 21, 2023

Dev images of v2.0.0 connectors to test #29681

Closed

evantahler added 3 commits August 21, 2023 14:13

Fill in PR number

6254433

add upgrade deadlines

7a9e440

fix migration guide heading

b82ab8b

evantahler commented Aug 21, 2023

View reviewed changes

edgao reviewed Aug 21, 2023

View reviewed changes

airbyte-integrations/connectors/destination-bigquery/src/main/resources/spec.json Outdated Show resolved Hide resolved

airbyte-integrations/connectors/destination-snowflake/src/main/resources/spec.json Outdated Show resolved Hide resolved

remove "Early Access"

71475e9

evantahler added 2 commits August 21, 2023 17:24

update run_with_normalization

8a91a2a

Merge branch 'master' into evan/bigquery-snowflake-v2

443f42a

evantahler force-pushed the evan/bigquery-snowflake-v2 branch from 105f43e to 443f42a Compare August 22, 2023 00:40

revert changes to run_with_normalization

52b8009

simplify dagger build when we don't need normalization

195ce56

evantahler mentioned this pull request Aug 22, 2023

Real V2 releases for BigQuery and Snowflake #29728

Closed

evantahler assigned Hesperide Aug 22, 2023

Merge branch 'master' into evan/bigquery-snowflake-v2

912e60c

edgao mentioned this pull request Aug 23, 2023

✨ Destinations bigquery+snowflake: Release DV2 #29783

Merged

evantahler closed this Aug 24, 2023

evantahler deleted the evan/bigquery-snowflake-v2 branch August 24, 2023 00:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Snowflake and BigQuery destination V2 #29683

Snowflake and BigQuery destination V2 #29683

evantahler commented Aug 21, 2023 •

edited

github-actions bot commented Aug 21, 2023 •

edited by evantahler

evantahler Aug 21, 2023

evantahler Aug 21, 2023

github-actions bot commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

edgao left a comment

github-actions bot commented Aug 21, 2023

evantahler commented Aug 22, 2023

edgao commented Aug 22, 2023

evantahler commented Aug 22, 2023 •

edited

github-actions bot commented Aug 22, 2023

github-actions bot commented Aug 22, 2023

github-actions bot commented Aug 22, 2023

github-actions bot commented Aug 22, 2023

github-actions bot commented Aug 23, 2023

github-actions bot commented Aug 23, 2023

evantahler commented Aug 24, 2023 •

edited

		\| 3.0.0 \| 2023-08-27 \| [xxx](https://github.com/airbytehq/airbyte/pull/xxx) \| Destinations V2 \|
		\| 2.0.0 \| 2023-08-09 \| [\#28894](https://github.com/airbytehq/airbyte/pull/29236) \| Remove support for Snowflake GCS/S3 loading method in favor of Snowflake Internal staging \|

Snowflake and BigQuery destination V2 #29683

Snowflake and BigQuery destination V2 #29683

Conversation

evantahler commented Aug 21, 2023 • edited

💀 DO NOT MERGE ⚠️

github-actions bot commented Aug 21, 2023 • edited by evantahler

Before Merging a Connector Pull Request

evantahler Aug 21, 2023

Choose a reason for hiding this comment

evantahler Aug 21, 2023

Choose a reason for hiding this comment

github-actions bot commented Aug 21, 2023

destination-bigquery test report (commit a4657f1954) - ❌

github-actions bot commented Aug 21, 2023

destination-snowflake test report (commit a4657f1954) - ❌

github-actions bot commented Aug 21, 2023

destination-snowflake test report (commit b82ab8b780) - ❌

edgao left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 21, 2023

destination-bigquery test report (commit b82ab8b780) - ❌

evantahler commented Aug 22, 2023

edgao commented Aug 22, 2023

evantahler commented Aug 22, 2023 • edited

github-actions bot commented Aug 22, 2023

destination-snowflake test report (commit 71475e9d00) - ❌

github-actions bot commented Aug 22, 2023

destination-bigquery test report (commit 71475e9d00) - ❌

github-actions bot commented Aug 22, 2023

destination-snowflake test report (commit 195ce56d4e) - ❌

github-actions bot commented Aug 22, 2023

destination-bigquery test report (commit 195ce56d4e) - ❌

github-actions bot commented Aug 23, 2023

destination-snowflake test report (commit 912e60c225) - ❌

github-actions bot commented Aug 23, 2023

destination-bigquery test report (commit 912e60c225) - ❌

evantahler commented Aug 24, 2023 • edited

evantahler commented Aug 21, 2023 •

edited

github-actions bot commented Aug 21, 2023 •

edited by evantahler

destination-bigquery test report (commit `a4657f1954`) - ❌

destination-snowflake test report (commit `a4657f1954`) - ❌

destination-snowflake test report (commit `b82ab8b780`) - ❌

destination-bigquery test report (commit `b82ab8b780`) - ❌

evantahler commented Aug 22, 2023 •

edited

destination-snowflake test report (commit `71475e9d00`) - ❌

destination-bigquery test report (commit `71475e9d00`) - ❌

destination-snowflake test report (commit `195ce56d4e`) - ❌

destination-bigquery test report (commit `195ce56d4e`) - ❌

destination-snowflake test report (commit `912e60c225`) - ❌

destination-bigquery test report (commit `912e60c225`) - ❌

evantahler commented Aug 24, 2023 •

edited