Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Snowflake sink #2266

Merged
merged 5 commits into from Jan 4, 2024
Merged

feat: Snowflake sink #2266

merged 5 commits into from Jan 4, 2024

Conversation

abcpro1
Copy link
Contributor

@abcpro1 abcpro1 commented Dec 29, 2023

Introduce the Snowflake sink as a storage destination for Dozer pipeline transformations.
A snowflake table is updated at a configurable interval with the latest values from Dozer.

dozer-log/src/reader/checkpoint.rs Outdated Show resolved Hide resolved
Introduce the Snowflake sink as a storage destination for Dozer pipeline transformations.
A snowflake table will be updated at a configurable interval with the latest values from Dozer.
@abcpro1 abcpro1 force-pushed the snowflake-sink branch 10 times, most recently from 76df65c to 61f147b Compare January 2, 2024 04:23
@abcpro1 abcpro1 force-pushed the snowflake-sink branch 2 times, most recently from 06c7d6e to 2d49b02 Compare January 3, 2024 16:32
odbc crate does not support array parameter binding, odbc-api does.
doing batch inserts with array parameter binding improves query execution time.
@abcpro1 abcpro1 added this pull request to the merge queue Jan 4, 2024
@abcpro1 abcpro1 removed this pull request from the merge queue due to a manual request Jan 4, 2024
@abcpro1 abcpro1 enabled auto-merge January 4, 2024 08:15
@abcpro1 abcpro1 linked an issue Jan 4, 2024 that may be closed by this pull request
@abcpro1 abcpro1 added this pull request to the merge queue Jan 4, 2024
Merged via the queue into getdozer:main with commit 5058b40 Jan 4, 2024
5 checks passed
@abcpro1 abcpro1 deleted the snowflake-sink branch January 4, 2024 08:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Measure and improve snowflake sink performance
2 participants