Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flush rules to be cognizant of in-memory database size #89

Merged
merged 15 commits into from
May 8, 2023

Conversation

Tang8330
Copy link
Contributor

@Tang8330 Tang8330 commented May 8, 2023

Addresses #68

This is necessary because Snowflake and BigQuery both have their limits on how big (in kb) a request can be. Snowflake's API docs specify 1 MB src...but the actual limit is significantly higher.

Through this PR, I've tested 25 mb payloads without any issue. However, this does happen...especially for tables with TOAST columns. The current workaround is to just reduce the number of items from the bufferRow but that's not robust.

This PR will allow users to specify an additional merge rule, and our merge rules will now look like:

  1. Flush interval time (default 10s)
  2. Pool rows (default to 15k) - number of rows.
  3. Pool size (default to 25 mb) - size of the in-memory db.

Whichever one is invoked first will trigger a flush cycle.

TODOs:

@Tang8330 Tang8330 changed the title WIP WIP - Enhance our flush rules May 8, 2023
@Tang8330 Tang8330 added the enhancement New feature or request label May 8, 2023
@Tang8330
Copy link
Contributor Author

Tang8330 commented May 8, 2023

Will confirm this alleviates the REQUEST ENTITY error from Snowflake.

@Tang8330 Tang8330 changed the title WIP - Enhance our flush rules Flush rules to be cognizant of in-memory database size May 8, 2023
@Tang8330 Tang8330 merged commit 4975601 into master May 8, 2023
1 check passed
@Tang8330 Tang8330 deleted the third-variable-for-flush branch May 8, 2023 23:39
@Tang8330
Copy link
Contributor Author

Tang8330 commented May 8, 2023

FYI @TamasNo1 - this is now officially supported.

@TamasNo1
Copy link
Contributor

TamasNo1 commented May 9, 2023

Amazing, well done @Tang8330 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants