Reduce the snowflake async buffer size by benmoriceau · Pull Request #30671 · airbytehq/airbyte

Benoit Moriceau (benmoriceau) · 2023-09-21T17:59:00Z

What

Reduce the async buffer size ratio for the snowflake destination from 0.7 to 0.5

vercel · 2023-09-21T17:59:06Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Sep 21, 2023 8:25pm

github-actions · 2023-09-21T17:59:22Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

Ryan Br... (tryangul) · 2023-09-21T18:02:27Z

+                                                      final ParsedCatalog parsedCatalog,
+                                                      final String defaultNamespace,
+                                                      final boolean useDestinationsV2Columns,
+                                                      Optional<Long> BufferMemoryLimit) {


needs final. Also casing

Ryan Br... (tryangul) · 2023-09-21T19:01:47Z

        flusher,
        catalog,
-        new BufferManager(),
+        BufferMemoryLimit.map(memoryLimit -> new BufferManager(memoryLimit)).orElse(new BufferManager()),


can we put this in a variable or method for clarity

Something like getBufferMemoryLimit() { return bufferMemoryLimit.getOrElse(DEFAULT_VALUE)

Akash Kulkarni (akashkulk)

Just trying to understand the tradeoffs here : does this mean that the destination will be slower assuming the source is already producing records faster than the destination can consume?

airbyte-oss-build-runner · 2023-09-21T19:18:14Z

destination-snowflake test report (commit `b4aeb3f022`) - ❌

⏲️ Total pipeline duration: 77mn34s

Step	Result
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	❌
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

Benoit Moriceau (benmoriceau) · 2023-09-21T20:28:29Z

Just trying to understand the tradeoffs here : does this mean that the destination will be slower assuming the source is already producing records faster than the destination can consume?

Not exactly, we will still buffer the main difference is that we will reach the limit of the buffer faster (because it is too small). In most of those case, we would have reach the buffer limit even if we had a bigger buffer, reducing the size will just makes use reaching the limit faster.

airbyte-oss-build-runner · 2023-09-21T20:39:00Z

destination-snowflake test report (commit `5ec7b96795`) - ❌

⏲️ Total pipeline duration: 78mn16s

Step	Result
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	❌
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

Akash Kulkarni (akashkulk) · 2023-09-21T20:53:31Z

Just trying to understand the tradeoffs here : does this mean that the destination will be slower assuming the source is already producing records faster than the destination can consume?

Not exactly, we will still buffer the main difference is that we will reach the limit of the buffer faster (because it is too small). In most of those case, we would have reach the buffer limit even if we had a bigger buffer, reducing the size will just makes use reaching the limit faster.

And just so that I understand - this will just result in more frequent uploads to the destination (Snowflake)?

Approving from my end - but will defer to Ryan Br... (@tryangul) for a more thorough review

airbyte-oss-build-runner · 2023-09-21T20:55:29Z

destination-snowflake test report (commit `f22c6c0fa9`) - ✅

⏲️ Total pipeline duration: 13mn50s

Step	Result
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

Ryan Br... (tryangul) · 2023-09-21T21:02:13Z

And just so that I understand - this will just result in more frequent uploads to the destination (Snowflake)?

It just means we will buffer less before backpressuring. The goal is to buffer enough so our upload threads always have work available as soon as they finish, but not so much we oom. The original limits as they've been set were just a best guess when this was originally designed months ago.

Benoit Moriceau (benmoriceau) · 2023-09-21T21:46:39Z

/approve-and-merge reason="Build is passing"

Octavia The Approver (octavia-approvington) · 2023-09-21T21:47:10Z

Myomoto says it looks good

Reduce the snowflake async buffer size

b4aeb3f

Benoit Moriceau (benmoriceau) requested review from Akash Kulkarni (akashkulk) and Ryan Br... (tryangul) September 21, 2023 17:59

Benoit Moriceau (benmoriceau) requested a review from a team as a code owner September 21, 2023 17:59

Octavia Squidington III (octavia-squidington-iii) added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Sep 21, 2023

vercel Bot deployed to Preview September 21, 2023 17:59 View deployment

Octavia Squidington III (octavia-squidington-iii) added the connectors/destination/snowflake label Sep 21, 2023

Update changelog

5ec7b96

vercel Bot deployed to Preview September 21, 2023 18:01 View deployment

Ryan Br... (tryangul) reviewed Sep 21, 2023

View reviewed changes

Akash Kulkarni (akashkulk) reviewed Sep 21, 2023

View reviewed changes

PR comments

f22c6c0

Benoit Moriceau (benmoriceau) requested a review from a team as a code owner September 21, 2023 20:25

Benoit Moriceau (benmoriceau) requested a review from Ryan Br... (tryangul) September 21, 2023 20:25

Benoit Moriceau (benmoriceau) requested a review from Akash Kulkarni (akashkulk) September 21, 2023 20:28

Akash Kulkarni (akashkulk) approved these changes Sep 21, 2023

View reviewed changes

Ryan Br... (tryangul) approved these changes Sep 21, 2023

View reviewed changes

Octavia The Approver (octavia-approvington) approved these changes Sep 21, 2023

View reviewed changes

Octavia The Approver (octavia-approvington) merged commit f599c1c into master Sep 21, 2023

Octavia The Approver (octavia-approvington) deleted the bmoric/reduce-snowflake-buffer-size branch September 21, 2023 21:47

Joe Bell (jbfbell) pushed a commit that referenced this pull request Sep 22, 2023

Reduce the snowflake async buffer size (#30671)

369d64c

Conversation

Benoit Moriceau (benmoriceau) commented Sep 21, 2023

What

Uh oh!

vercel Bot commented Sep 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 21, 2023

Before Merging a Connector Pull Request

Uh oh!

Ryan Br... (tryangul) Sep 21, 2023

Choose a reason for hiding this comment

Uh oh!

Benoit Moriceau (benmoriceau) Sep 21, 2023

Choose a reason for hiding this comment

Uh oh!

Ryan Br... (tryangul) Sep 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Benoit Moriceau (benmoriceau) Sep 21, 2023

Choose a reason for hiding this comment

Uh oh!

Akash Kulkarni (akashkulk) left a comment

Choose a reason for hiding this comment

Uh oh!

airbyte-oss-build-runner commented Sep 21, 2023

destination-snowflake test report (commit b4aeb3f022) - ❌

Uh oh!

Benoit Moriceau (benmoriceau) commented Sep 21, 2023

Uh oh!

airbyte-oss-build-runner commented Sep 21, 2023

destination-snowflake test report (commit 5ec7b96795) - ❌

Uh oh!

Akash Kulkarni (akashkulk) commented Sep 21, 2023

Uh oh!

airbyte-oss-build-runner commented Sep 21, 2023

destination-snowflake test report (commit f22c6c0fa9) - ✅

Uh oh!

Ryan Br... (tryangul) commented Sep 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Benoit Moriceau (benmoriceau) commented Sep 21, 2023

Uh oh!

Octavia The Approver (octavia-approvington) commented Sep 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

vercel Bot commented Sep 21, 2023 •

edited

Loading

Ryan Br... (tryangul) Sep 21, 2023 •

edited

Loading

destination-snowflake test report (commit `b4aeb3f022`) - ❌

destination-snowflake test report (commit `5ec7b96795`) - ❌

destination-snowflake test report (commit `f22c6c0fa9`) - ✅

Ryan Br... (tryangul) commented Sep 21, 2023 •

edited

Loading