Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[destination] implement loader for Snowflake with PUT stage #22

Closed
12 of 13 tasks
rudolfix opened this issue Jun 15, 2022 · 1 comment · Fixed by #414
Closed
12 of 13 tasks

[destination] implement loader for Snowflake with PUT stage #22

rudolfix opened this issue Jun 15, 2022 · 1 comment · Fixed by #414
Assignees

Comments

@rudolfix
Copy link
Collaborator

rudolfix commented Jun 15, 2022

Background
Implement Snowflake destination based on PUT stage. It will be quite similar to the BigQuery implementation (just simpler).

Tasks

  • use jsonl as preferred type
  • use PUT to stage
  • allow option to remove files when COPY INTO succeeded.
  • our jsonl uses base64 to store binary so enable BINARY_FORMAT in COPY INTO https://docs.snowflake.com/en/sql-reference/sql/copy-into-table
  • fail on any error on COPY INTO, return up to 100 errors
  • i think you must use TRUNCATE for replace load, do TRUNCATE COPY in one transaction!
  • implement cluster hint with CLUSTER BY

Authentication
Derived from ConnectionStringCredentials:

    • should expose several query parameters as fields warehouse role

Tests

    • we should enable it as destination in tests. all tests using ALL_DESTINATIONS must pass
    • it should pass the same common tests which BigQuery job client and sql client is passing
    • note that to pass sql_client tests you'll need to map DBApi exceptions into 3 categories that dlt needs: relation not found, terminal exceptions, transient exceptions. see bigquery and postgres implementations. that's manual work

all in all writing the test suite is more work than the code here

@rudolfix rudolfix self-assigned this Jun 15, 2022
@rudolfix rudolfix changed the title [loader] implement loader for Snowflake [destination] implement local loader for Snowflake Jun 5, 2023
@rudolfix rudolfix changed the title [destination] implement local loader for Snowflake [destination] implement loader for Snowflake with PUT stage Jun 5, 2023
@rudolfix
Copy link
Collaborator Author

rudolfix commented Jun 5, 2023

@adrianbr if you have any comments to the ticket...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant