🐛 Destination Snowflake: Merge old snowflake work #27935

jbfbell · 2023-07-03T22:12:16Z

Adds data as JsonNode to pass through, running into memory issues so add JVM args to attach VisualVM
JsonNode
Lowers the optimal batch size to see if this improvements movement
Fixes NPE by checking if PartialAirbyteMessage contains a PartialAirbyteRecord
Fixes config switch when config is not explicitly set (config migration needed)
Adds logic to check if queue has elements before getting timeOfLastMessage
Add PartialSerialisedMessage test. (Add PartialSerialisedMessage test. #27452)
🤖 Auto format destination-snowflake code [skip ci]
Cleans up code w/o JVM args & rebase
🤖 Auto format destination-snowflake code [skip ci]
Adds breadcrumb on the STATE message deviation and where the deserialize/serialize is done to unpack
🤖 Auto format destination-snowflake code [skip ci]
Adds back line formatter removed and comment describing rational for lower batchSize
Bumps Snowflake version and type checks
Added note to remove PartialAirbyteRecordMessage with low resource testing
Automated Commit - Format and Process Resources Changes
Fix issue with multiple namespaces in snowflake not writing to the correct staging schema
Fix issue with multiple namespaces in snowflake not writing to the correct staging schema
remove stage name maniuplating method

What

Describe what the change is solving
It helps to add screenshots if it affects the frontend.

How

Describe the solution

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user?

For connector PRs, use this section to explain which type of semantic versioning bump occurs as a result of the changes. Refer to our Semantic Versioning for Connectors guidelines for more information. Breaking changes to connectors must be documented by an Airbyte engineer (PR author, or reviewer for community PRs) by using the Breaking Change Release Playbook.

If there are breaking changes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Actions

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Community member? Grant edit access to maintainers (instructions)
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Connector version is set to 0.0.1
- Dockerfile has version 0.0.1
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- docs/integrations/<source or destination>/<name>.md including changelog with an entry for the initial version. See changelog example
- docs/integrations/README.md

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

…add JVM args to attach VisualVM

…yteRecord

…on needed)

…ssage

* Test deserialise. * Add tests. * Simplify and fix tests. * Format. * Adds tests for deserializeAirbyteMessage * Adds tests for deserializeAirbyteMessage with bad data * Cleans up deserializeAirbyteMessage and throws Exception when invalid message * More code cleanup --------- Co-authored-by: ryankfu <ryan.fu@airbyte.io>

…ize/serialize is done to unpack

…lower batchSize

…sting

…rrect staging schema

octavia-squidington-iii · 2023-07-05T17:56:31Z

source-instagram test report (commit `7f77448fea`) - ❌

⏲️ Total pipeline duration: 10mn14s

Step	Result
Validate airbyte-integrations/connectors/source-instagram/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-instagram docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-instagram test

octavia-squidington-iii · 2023-07-05T18:03:46Z

source-pipedrive test report (commit `7f77448fea`) - ❌

⏲️ Total pipeline duration: 06mn01s

Step	Result
Validate airbyte-integrations/connectors/source-pipedrive/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-pipedrive docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-pipedrive test

octavia-squidington-iii · 2023-07-05T18:03:48Z

source-pinterest test report (commit `7f77448fea`) - ❌

⏲️ Total pipeline duration: 15mn21s

Step	Result
Validate airbyte-integrations/connectors/source-pinterest/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-pinterest docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-pinterest test

octavia-squidington-iii · 2023-07-05T18:05:28Z

source-zendesk-chat test report (commit `7f77448fea`) - ❌

⏲️ Total pipeline duration: 08mn33s

Step	Result
Validate airbyte-integrations/connectors/source-zendesk-chat/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-zendesk-chat docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-zendesk-chat test

octavia-squidington-iii · 2023-07-05T18:09:21Z

source-stripe test report (commit `7f77448fea`) - ✅

⏲️ Total pipeline duration: 13mn37s

Step	Result
Validate airbyte-integrations/connectors/source-stripe/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-stripe docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-stripe test

octavia-squidington-iii · 2023-07-05T18:09:25Z

source-zendesk-talk test report (commit `7f77448fea`) - ✅

⏲️ Total pipeline duration: 05mn24s

Step	Result
Validate airbyte-integrations/connectors/source-zendesk-talk/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-zendesk-talk docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-zendesk-talk test

octavia-squidington-iii · 2023-07-05T18:14:00Z

source-xero test report (commit `7f77448fea`) - ✅

⏲️ Total pipeline duration: 18mn15s

Step	Result
Validate airbyte-integrations/connectors/source-xero/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-xero docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-xero test

octavia-squidington-iii · 2023-07-05T19:04:58Z

destination-e2e-test test report (commit `6921b3ab36`) - ❌

⏲️ Total pipeline duration: 04mn59s

Step	Result
Validate airbyte-integrations/connectors/destination-e2e-test/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-e2e-test docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-e2e-test:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-e2e-test test

octavia-squidington-iii · 2023-07-05T19:08:58Z

source-e2e-test-cloud test report (commit `6921b3ab36`) - ❌

⏲️ Total pipeline duration: 03mn51s

Step	Result
Validate airbyte-integrations/connectors/source-e2e-test-cloud/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build source-e2e-test-cloud docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-e2e-test-cloud:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-e2e-test-cloud test

octavia-squidington-iii · 2023-07-05T19:12:52Z

source-elasticsearch test report (commit `6921b3ab36`) - ❌

⏲️ Total pipeline duration: 03mn44s

Step	Result
Validate airbyte-integrations/connectors/source-elasticsearch/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-elasticsearch docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-elasticsearch:integrationTest	❌
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-elasticsearch test

jbfbell · 2023-07-05T23:46:56Z

...wflake/src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeDestination.java

+  public static boolean useAsyncSnowflake(final JsonNode config) {
+    // this is how we toggle async snowflake on.
+    // final Set<String> stagingLoadingMethods = Collections.emptySet();
+    final Set<String> stagingLoadingMethods = Set.of("standard", "internal staging", "internal-staging", "internal_staging");


@davinchia Should this be on? In the other PR this was set to off

hmm which pr did you previously see this from?

note: from the spec.json, I believe this should only be Internal Staging. We don't want to turn this on for 'standard'.

edgao

rubberstamp 🤷 lgtm assuming other conversations get resolved

davinchia

One comment on turning this on only for Internal Staging.

Otherwise,

jbfbell · 2023-07-06T17:16:30Z

/approve-and-merge

octavia-approvington · 2023-07-06T17:17:04Z

Trying to merge your PR using a bot?
Comment this on the PR:

/approve-and-merge reason="<STATE YOUR REASON HERE>"

Make sure you've reviewed the docs!

davinchia · 2023-07-06T17:26:24Z

@jbfbell if you are trying to use the slash command you are missing a 'reason' parameter

jbfbell · 2023-07-06T17:27:07Z

@jbfbell if you are trying to use the slash command you are missing a 'reason' parameter

@davinchia dang, thank you

jbfbell · 2023-07-06T17:28:18Z

/approve-and-merge reason="Snowflake Destination tests passing, only publishing snowflake destination"

octavia-approvington · 2023-07-06T17:28:50Z

Our work here is done

ryankfu and others added 25 commits June 21, 2023 14:24

Adds data as JsonNode to pass through, running into memory issues so …

4978325

…add JVM args to attach VisualVM

JsonNode

db6f38e

Lowers the optimal batch size to see if this improvements movement

138e1f0

Fixes NPE by checking if PartialAirbyteMessage contains a PartialAirb…

2912811

…yteRecord

Fixes config switch when config is not explicitly set (config migrati…

dc879ca

…on needed)

Adds logic to check if queue has elements before getting timeOfLastMe…

54d0c6f

…ssage

🤖 Auto format destination-snowflake code [skip ci]

437358b

Cleans up code w/o JVM args & rebase

d7fdacc

🤖 Auto format destination-snowflake code [skip ci]

247d081

Adds breadcrumb on the STATE message deviation and where the deserial…

e44b5d6

…ize/serialize is done to unpack

🤖 Auto format destination-snowflake code [skip ci]

5a611bb

Adds back line formatter removed and comment describing rational for …

e9550d3

…lower batchSize

Bumps Snowflake version and type checks

ca1a224

Added note to remove PartialAirbyteRecordMessage with low resource te…

f75c171

…sting

Automated Commit - Format and Process Resources Changes

ba0e810

versioning merge conflicts

1a9118e

Merge branch 'master' into ryan/fix-record-data

615dfe8

Merge branch 'master' into ryan/fix-record-data

667764f

Merge branch 'master' into ryan/fix-record-data

0b2cb08

Merge branch 'master' into ryan/fix-record-data

4bc56c4

Fix issue with multiple namespaces in snowflake not writing to the co…

2ad9be4

…rrect staging schema

Fix issue with multiple namespaces in snowflake not writing to the co…

ef50661

…rrect staging schema

remove stage name maniuplating method

7ed966a

Merge branch 'master' into ryan/fix-record-data

040935b

jbfbell requested review from a team as code owners July 3, 2023 22:12

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/destination/snowflake labels Jul 3, 2023

Merge branch 'master' into joseph.bell/27714/snowflake-merging

6921b3a

jbfbell added 2 commits July 5, 2023 13:59

update other snowflake loading method types

d690384

Merge branch 'master' into joseph.bell/27714/snowflake-merging

fa73220

jbfbell requested a review from evantahler July 5, 2023 23:28

jbfbell commented Jul 5, 2023

View reviewed changes

edgao approved these changes Jul 6, 2023

View reviewed changes

davinchia approved these changes Jul 6, 2023

View reviewed changes

jbfbell added 2 commits July 6, 2023 09:07

Merge branch 'master' into joseph.bell/27714/snowflake-merging

531f12d

remove standard

487c157

octavia-approvington approved these changes Jul 6, 2023

View reviewed changes

octavia-approvington merged commit b03be1b into master Jul 6, 2023
17 of 18 checks passed

octavia-approvington deleted the joseph.bell/27714/snowflake-merging branch July 6, 2023 17:28

jbfbell mentioned this pull request Jul 6, 2023

🐛 Destination Snowflake: fix unit test #28020

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Destination Snowflake: Merge old snowflake work #27935

🐛 Destination Snowflake: Merge old snowflake work #27935

jbfbell commented Jul 3, 2023

octavia-squidington-iii commented Jul 5, 2023

octavia-squidington-iii commented Jul 5, 2023

octavia-squidington-iii commented Jul 5, 2023

octavia-squidington-iii commented Jul 5, 2023

octavia-squidington-iii commented Jul 5, 2023

octavia-squidington-iii commented Jul 5, 2023

octavia-squidington-iii commented Jul 5, 2023

octavia-squidington-iii commented Jul 5, 2023

octavia-squidington-iii commented Jul 5, 2023

octavia-squidington-iii commented Jul 5, 2023

jbfbell Jul 5, 2023

davinchia Jul 6, 2023

davinchia Jul 6, 2023

edgao left a comment

davinchia left a comment

jbfbell commented Jul 6, 2023

octavia-approvington commented Jul 6, 2023

davinchia commented Jul 6, 2023

jbfbell commented Jul 6, 2023

jbfbell commented Jul 6, 2023

octavia-approvington commented Jul 6, 2023

🐛 Destination Snowflake: Merge old snowflake work #27935

🐛 Destination Snowflake: Merge old snowflake work #27935

Conversation

jbfbell commented Jul 3, 2023

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

octavia-squidington-iii commented Jul 5, 2023

source-instagram test report (commit 7f77448fea) - ❌

octavia-squidington-iii commented Jul 5, 2023

source-pipedrive test report (commit 7f77448fea) - ❌

octavia-squidington-iii commented Jul 5, 2023

source-pinterest test report (commit 7f77448fea) - ❌

octavia-squidington-iii commented Jul 5, 2023

source-zendesk-chat test report (commit 7f77448fea) - ❌

octavia-squidington-iii commented Jul 5, 2023

source-stripe test report (commit 7f77448fea) - ✅

octavia-squidington-iii commented Jul 5, 2023

source-zendesk-talk test report (commit 7f77448fea) - ✅

octavia-squidington-iii commented Jul 5, 2023

source-xero test report (commit 7f77448fea) - ✅

octavia-squidington-iii commented Jul 5, 2023

destination-e2e-test test report (commit 6921b3ab36) - ❌

octavia-squidington-iii commented Jul 5, 2023

source-e2e-test-cloud test report (commit 6921b3ab36) - ❌

octavia-squidington-iii commented Jul 5, 2023

source-elasticsearch test report (commit 6921b3ab36) - ❌

jbfbell Jul 5, 2023

Choose a reason for hiding this comment

davinchia Jul 6, 2023

Choose a reason for hiding this comment

davinchia Jul 6, 2023

Choose a reason for hiding this comment

edgao left a comment

Choose a reason for hiding this comment

davinchia left a comment

Choose a reason for hiding this comment

jbfbell commented Jul 6, 2023

octavia-approvington commented Jul 6, 2023

davinchia commented Jul 6, 2023

jbfbell commented Jul 6, 2023

jbfbell commented Jul 6, 2023

octavia-approvington commented Jul 6, 2023

source-instagram test report (commit `7f77448fea`) - ❌

source-pipedrive test report (commit `7f77448fea`) - ❌

source-pinterest test report (commit `7f77448fea`) - ❌

source-zendesk-chat test report (commit `7f77448fea`) - ❌

source-stripe test report (commit `7f77448fea`) - ✅

source-zendesk-talk test report (commit `7f77448fea`) - ✅

source-xero test report (commit `7f77448fea`) - ✅

destination-e2e-test test report (commit `6921b3ab36`) - ❌

source-e2e-test-cloud test report (commit `6921b3ab36`) - ❌

source-elasticsearch test report (commit `6921b3ab36`) - ❌