T&D: Use non-null columns for PKs rather then scanning raw table to ensure all records have PKs by evantahler · Pull Request #30779 · airbytehq/airbyte

Evan Tahler (evantahler) · 2023-09-27T00:50:59Z

This saves a scan of the raw table by swapping a query looking for null PKs each T&D by altering the final tables to have NON-NULL PK columns.
#30812 is merged into this PR, so existing syncs will undergo a one-time soft reset (rebuild) of the final table to add the "NOT NULL" constraint on primary key columns

…nsure all records have PKs

vercel · 2023-09-27T00:51:04Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Oct 3, 2023 4:04pm

github-actions · 2023-09-27T00:51:22Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

…nfo (#30812) Co-authored-by: evantahler <evantahler@users.noreply.github.com> Co-authored-by: Joe Bell <joseph.bell@airbyte.io> Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>

airbyte-oss-build-runner · 2023-09-29T22:21:51Z

destination-bigquery test report (commit `ea9f9b7806`) - ✅

⏲️ Total pipeline duration: 03mn31s

Step	Result
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

airbyte-oss-build-runner · 2023-09-29T22:24:58Z

destination-snowflake test report (commit `ea9f9b7806`) - ✅

⏲️ Total pipeline duration: 02mn55s

Step	Result
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-snowflake	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

Edward Gao (edgao)

lgtm, couple minor comments

airbyte-oss-build-runner · 2023-10-02T19:06:28Z

destination-bigquery test report (commit `d7e5c4c0a4`) - ✅

⏲️ Total pipeline duration: 08mn45s

Step	Result
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

airbyte-oss-build-runner · 2023-10-02T19:16:45Z

destination-snowflake test report (commit `d7e5c4c0a4`) - ✅

⏲️ Total pipeline duration: 10mn05s

Step	Result
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-snowflake	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

…java/io/airbyte/integrations/destination/bigquery/typing_deduping/BigQuerySqlGenerator.java Co-authored-by: Joe Bell <joseph.bell@airbyte.io>

airbyte-oss-build-runner · 2023-10-03T15:43:00Z

destination-snowflake test report (commit `fb0e237872`) - ✅

⏲️ Total pipeline duration: 12mn49s

Step	Result
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-snowflake	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

airbyte-oss-build-runner · 2023-10-03T15:45:08Z

destination-bigquery test report (commit `fb0e237872`) - ❌

⏲️ Total pipeline duration: 01mn55s

Step	Result
Build connector tar	❌
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

airbyte-oss-build-runner · 2023-10-03T16:03:35Z

destination-bigquery test report (commit `b6ecbc2a65`) - ❌

⏲️ Total pipeline duration: 09mn42s

Step	Result
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	❌
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

airbyte-oss-build-runner · 2023-10-03T16:16:07Z

destination-bigquery test report (commit `5454a6baa8`) - ✅

⏲️ Total pipeline duration: 09mn09s

Step	Result
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

airbyte-oss-build-runner · 2023-10-03T16:19:49Z

destination-snowflake test report (commit `5454a6baa8`) - ✅

⏲️ Total pipeline duration: 03mn26s

Step	Result
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-snowflake	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

…ble to ensure all records have PKs (#30779)" This reverts commit 4992fce.

…nsure all records have PKs (#30779) Co-authored-by: evantahler <evantahler@users.noreply.github.com> Co-authored-by: Joe Bell <joseph.bell@airbyte.io> Co-authored-by: jbfbell <jbfbell@users.noreply.github.com> Co-authored-by: Edward Gao <edward.gao@airbyte.io> Co-authored-by: edgao <edgao@users.noreply.github.com>

T&D: Use not-null columns for PKs rather then scanning raw table to e…

09e1754

…nsure all records have PKs

Evan Tahler (evantahler) changed the title ~~T&D: Use not-null columns for PKs rather then scanning raw table to e…~~ T&D: Use not-null columns for PKs rather then scanning raw table to ensure all records have PKs Sep 27, 2023

Octavia Squidington III (octavia-squidington-iii) added area/connectors Connector related issues connectors/destination/bigquery connectors/destination/snowflake labels Sep 27, 2023

Metadata

960a5fe

Evan Tahler (evantahler) marked this pull request as ready for review September 27, 2023 00:54

Evan Tahler (evantahler) requested a review from a team as a code owner September 27, 2023 00:54

Octavia Squidington III (octavia-squidington-iii) added the area/documentation Improvements or additions to documentation label Sep 27, 2023

Evan Tahler (evantahler) commented Sep 27, 2023

View reviewed changes

Comment thread ...ava/io/airbyte/integrations/destination/snowflake/typing_deduping/SnowflakeSqlGenerator.java Outdated

vercel Bot deployed to Preview September 27, 2023 00:55 View deployment

Automated Commit - Formatting Changes

947adfe

This comment was marked as outdated.

Sign in to view

test fixes

0e8e846

This comment was marked as outdated.

Sign in to view

remove custom snowflake error message test

fe417f6

Evan Tahler (evantahler) mentioned this pull request Sep 27, 2023

T&D: Soft reset on PK non-null column change + More Snowflake Table info #30812

Merged

Evan Tahler (evantahler) changed the title ~~T&D: Use not-null columns for PKs rather then scanning raw table to ensure all records have PKs~~ T&D: Use non-null columns for PKs rather then scanning raw table to ensure all records have PKs Sep 27, 2023

Merge branch 'master' into evan/no-t_and_d-pk-check

c5507b5

vercel Bot deployed to Preview September 27, 2023 23:55 View deployment

Automated Commit - Formatting Changes

7081356

This comment was marked as outdated.

Sign in to view

Merge branch 'master' into evan/no-t_and_d-pk-check

7722c7b

vercel Bot deployed to Preview September 28, 2023 14:58 View deployment

Merge branch 'master' into evan/no-t_and_d-pk-check

bad9b6b

Airbyte (airbytehq) deleted a comment from airbyte-oss-build-runner Sep 29, 2023

T&D: Soft reset on PK non-null column change + More Snowflake Table i…

ea9f9b7

…nfo (#30812) Co-authored-by: evantahler <evantahler@users.noreply.github.com> Co-authored-by: Joe Bell <joseph.bell@airbyte.io> Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>

Edward Gao (edgao) approved these changes Sep 29, 2023

View reviewed changes

Edward Gao (edgao) added 2 commits October 2, 2023 11:40

minor tweaks

c8a07c4

Merge branch 'master' into evan/no-t_and_d-pk-check

315fa91

vercel Bot deployed to Preview October 2, 2023 18:44 View deployment

This comment was marked as outdated.

Sign in to view

Automated Commit - Formatting Changes

d7e5c4c

Joe Bell (jbfbell) approved these changes Oct 2, 2023

View reviewed changes

Comment thread .../java/io/airbyte/integrations/destination/bigquery/typing_deduping/BigQuerySqlGenerator.java Outdated

Update airbyte-integrations/connectors/destination-bigquery/src/main/…

fb0e237

…java/io/airbyte/integrations/destination/bigquery/typing_deduping/BigQuerySqlGenerator.java Co-authored-by: Joe Bell <joseph.bell@airbyte.io>

typo

b6ecbc2

Edward Gao (edgao) enabled auto-merge (squash) October 3, 2023 15:55

Automated Commit - Formatting Changes

5454a6b

Edward Gao (edgao) merged commit 4992fce into master Oct 3, 2023

Edward Gao (edgao) deleted the evan/no-t_and_d-pk-check branch October 3, 2023 16:23

Evan Tahler (evantahler) mentioned this pull request Oct 4, 2023

When a Raw record is found missing a PK, we should delete that record #31063

Closed

Edward Gao (edgao) added a commit that referenced this pull request Oct 4, 2023

Revert "T&D: Use non-null columns for PKs rather then scanning raw ta…

5e15f67

…ble to ensure all records have PKs (#30779)" This reverts commit 4992fce.

Evan Tahler (evantahler) mentioned this pull request Oct 9, 2023

Remove Dv2 Null checks #31149

Merged

Evan Tahler (evantahler) mentioned this pull request Oct 27, 2023

Formally decide how the Airbyte Protocol should handle multiple primary keys #31926

Closed

Uh oh!

Conversation

Evan Tahler (evantahler) commented Sep 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel Bot commented Sep 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 27, 2023 • edited by evantahler Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before Merging a Connector Pull Request

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

airbyte-oss-build-runner commented Sep 29, 2023

destination-bigquery test report (commit ea9f9b7806) - ✅

Uh oh!

airbyte-oss-build-runner commented Sep 29, 2023

destination-snowflake test report (commit ea9f9b7806) - ✅

Uh oh!

Edward Gao (edgao) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

airbyte-oss-build-runner commented Oct 2, 2023

destination-bigquery test report (commit d7e5c4c0a4) - ✅

Uh oh!

airbyte-oss-build-runner commented Oct 2, 2023

destination-snowflake test report (commit d7e5c4c0a4) - ✅

Uh oh!

Uh oh!

airbyte-oss-build-runner commented Oct 3, 2023

destination-snowflake test report (commit fb0e237872) - ✅

Uh oh!

airbyte-oss-build-runner commented Oct 3, 2023

destination-bigquery test report (commit fb0e237872) - ❌

Uh oh!

airbyte-oss-build-runner commented Oct 3, 2023

destination-bigquery test report (commit b6ecbc2a65) - ❌

Uh oh!

airbyte-oss-build-runner commented Oct 3, 2023

destination-bigquery test report (commit 5454a6baa8) - ✅

Uh oh!

airbyte-oss-build-runner commented Oct 3, 2023

destination-snowflake test report (commit 5454a6baa8) - ✅

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Evan Tahler (evantahler) commented Sep 27, 2023 •

edited

Loading

vercel Bot commented Sep 27, 2023 •

edited

Loading

github-actions Bot commented Sep 27, 2023 •

edited by evantahler

Loading

destination-bigquery test report (commit `ea9f9b7806`) - ✅

destination-snowflake test report (commit `ea9f9b7806`) - ✅

destination-bigquery test report (commit `d7e5c4c0a4`) - ✅

destination-snowflake test report (commit `d7e5c4c0a4`) - ✅

destination-snowflake test report (commit `fb0e237872`) - ✅

destination-bigquery test report (commit `fb0e237872`) - ❌

destination-bigquery test report (commit `b6ecbc2a65`) - ❌

destination-bigquery test report (commit `5454a6baa8`) - ✅

destination-snowflake test report (commit `5454a6baa8`) - ✅