Skip to content

Destination snowflake: prevent data loss, by only running T+D once at end of sync - #29881

Merged
Edward Gao (edgao) merged 4 commits into
masterfrom
edgao/dv2/snowflake/tdonce
Aug 26, 2023
Merged

Edward Gao (edgao) merged 4 commits into
masterfrom
edgao/dv2/snowflake/tdonce

Conversation

@edgao

@edgao Edward Gao (edgao) commented Aug 26, 2023 •

Copy link
Copy Markdown
Contributor

https://airbytehq-team.slack.com/archives/C03C4AVJWG4/p1693003110775889

our sql queries don't work well in concurrent situations

@octavia-squidington-iii Octavia Squidington III (octavia-squidington-iii) added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Aug 26, 2023
@edgao Edward Gao (edgao) changed the title only td at end of sync Destination snowflake: prevent data loss, by only running T+D once at end of sync Aug 26, 2023
@edgao
Edward Gao (edgao) marked this pull request as ready for review August 26, 2023 00:20
@edgao
Edward Gao (edgao) requested a review from a team as a code owner August 26, 2023 00:20
@edgao
Edward Gao (edgao) enabled auto-merge (squash) August 26, 2023 00:20
@github-actions

Copy link
Copy Markdown
Contributor

destination-snowflake test report (commit fcdbda53ac) - ✅

⏲️ Total pipeline duration: 33mn44s

Step Result
Java Connector Unit Tests ✅
Build connector tar ✅
Build destination-snowflake docker image for platform linux/x86_64 ✅
Build airbyte/normalization-snowflake:dev ✅
Java Connector Integration Tests ✅
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml ✅
Connector version semver check ✅
Connector version increment check ✅
QA checks ✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

@github-actions

Copy link
Copy Markdown
Contributor

destination-snowflake test report (commit f8e2790f2a) - ✅

⏲️ Total pipeline duration: 32mn37s

Step Result
Java Connector Unit Tests ✅
Build connector tar ✅
Build destination-snowflake docker image for platform linux/x86_64 ✅
Build airbyte/normalization-snowflake:dev ✅
Java Connector Integration Tests ✅
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml ✅
Connector version semver check ✅
Connector version increment check ✅
QA checks ✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

@github-actions

Copy link
Copy Markdown
Contributor

destination-snowflake test report (commit 19c9777b0b) - ✅

⏲️ Total pipeline duration: 05mn36s

Step Result
Java Connector Unit Tests ✅
Build connector tar ✅
Build destination-snowflake docker image for platform linux/x86_64 ✅
Build airbyte/normalization-snowflake:dev ✅
Java Connector Integration Tests ✅
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml ✅
Connector version semver check ✅
Connector version increment check ✅
QA checks ✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

@edgao
Edward Gao (edgao) merged commit 4584f1c into master Aug 26, 2023
@edgao
Edward Gao (edgao) deleted the edgao/dv2/snowflake/tdonce branch August 26, 2023 01:49
Harry Touloupas (harrytou) pushed a commit to KYVENetwork/airbyte that referenced this pull request Sep 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/destination/snowflake

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants