-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🎉 Snowflake Destination internal staging support #8253
🎉 Snowflake Destination internal staging support #8253
Conversation
vmaltsev seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
/test connector=connectors/destination-snowflake
|
airbyte-integrations/connectors/destination-snowflake/src/main/resources/spec.json
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/destination-snowflake/src/main/resources/spec.json
Outdated
Show resolved
Hide resolved
...a/io/airbyte/integrations/destination/snowflake/SnowflakeInternalStagingConsumerFactory.java
Outdated
Show resolved
Hide resolved
...c/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeStagingSqlOperations.java
Outdated
Show resolved
Hide resolved
...te/integrations/destination/snowflake/SnowflakeInternalStagingDestinatiomAcceptanceTest.java
Outdated
Show resolved
Hide resolved
...a/io/airbyte/integrations/destination/snowflake/SnowflakeInternalStagingConsumerFactory.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The test failure does not look relevant to this PR. The issue should have been fixed already on the master branch. Maybe you just need to merge master and the build should pass.
/publish connector=connectors/destination-snowflake
|
Can not publish new version at the moment. This PR #5814 broke normalization step |
/publish connector=connectors/destination-snowflake
|
* Snowflake Destination: implemented internal staging * fix checkstyle * improved perfomance of sync * added changelog * test refactoring * removed file chunking * add docs * added javadoc, updated spec * refactoring * bump version Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
What
Records can be inserted into snowflake using a standard SQL INSERT statement or by first copying the data to blob storage (e.g: azure, gcs, s3, snowflake internal) then using the COPY command to read that data into snowflake. The staging approach is an order of magnitude faster and should pretty much always be used except for the smallest data volumes/proof of concepts.
The snowflake destination currently supports staging to S3. it should also support internal staging as described here: https://docs.snowflake.com/en/user-guide/data-load-considerations-stage.html
How
Workload :
CREATE STAGE @TEMP_STAGE_NAME
PUT file://local/some_folder/* @TEMP_STAGE_NAME.
COPY FROM @TEMP_STAGE_NAME
DROP @TEMP_STAGE_NAME
Standard Inserts 3 millions records
![snowflake_insert_3m](https://user-images.githubusercontent.com/39538064/144055787-32da1a2e-3be0-4d9c-b675-74250a8eeeca.png)
![snowflake_int_3m_7](https://user-images.githubusercontent.com/39538064/144055848-fce05d64-6b52-445e-8c0b-3f0a96dce3df.png)
Internal Staging 3 millions records
Recommended reading order
x.java
y.python
🚨 User Impact 🚨
Are there any breaking changes? If yes, please make sure to include it here and in any changelogs with the 🚨🚨 emoji
What is the end result perceived by the user?
Pre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/SUMMARY.md
docs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampledocs/integrations/README.md
airbyte-integrations/builds.md
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing./publish
command described hereUpdating a connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing./publish
command described hereConnector Generator
-scaffold
in their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates
then checking in your changes