feat(targets): Targets now accept a batch_size_rows setting to configure how many rows are loaded in each record batch
#2248
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BuzzCutNorman
requested review from
edgarrmondragon and
kgpayne
as code owners
BuzzCutNorman
changed the title
CodSpeed Performance ReportMerging #2248 will not alter performanceComparing Summary
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #2248 +/- ##
==========================================
+ Coverage 88.54% 88.56% +0.01%
==========================================
Files 54 54
Lines 4724 4730 +6
Branches 920 921 +1
==========================================
+ Hits 4183 4189 +6
Misses 380 380
Partials 161 161 ☔ View full report in Codecov by Sentry. |
|
@pnadolny13 and @edgarrmondragon I broke out the code from pr 1876 that gives the user of a target the ability to set the batch size in rows which triggers a drain. Please take a look and let me know what you think. |
Co-authored-by: Edgar Ramírez Mondragón <16805946+edgarrmondragon@users.noreply.github.com>
edgarrmondragon
force-pushed
the
2155-target-set-batch-size-rows
branch
from
ed51495
to
c247f82
Compare
edgarrmondragon
approved these changes
Feb 19, 2024
There was a problem hiding this comment.
Thanks for taking the time for extracting this into its own PR @BuzzCutNorman!
|
@edgarrmondragon Thanks 🙏 for all of the comment and documentation edits. Loads better 🥳 than what I had. 🎉 |
edgarrmondragon
changed the title
batch_size_rows setting to configure how many rows are loaded in each record batch
edgarrmondragon
added a commit
that referenced
this pull request
Mar 6, 2024
…igure how many rows are loaded in each record batch (#2248) * added batch_size_rows tests to target * add batch_size_rows test to test target sql * added batch size row to target capabilities * add batch_size_rows to target as a bulitin config * added code for batch_size_rows * Apply suggestions from code review Co-authored-by: Edgar Ramírez Mondragón <16805946+edgarrmondragon@users.noreply.github.com> * applied documentaion update from review to max_size * Update singer_sdk/sinks/core.py * Update singer_sdk/sinks/core.py * chore: Link to `batch_size_rows` attribute docs * Update `batch_size_rows` docs --------- Co-authored-by: Edgar Ramírez Mondragón <16805946+edgarrmondragon@users.noreply.github.com> Co-authored-by: Edgar Ramírez-Mondragón <edgarrm358@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This feature will allow target users the ability to set the batch size used to triggers a drain via the target's meltano confiig.
batch_size_rows: 100000When a batch is full and a drain is triggered, the message logged will now state the size of the batch so the users has a visual confirmation that the batch size they configured is being utilized.
The code for this PR was pulled out of PR 1876.
Closes #2155
📚 Documentation preview 📚: https://meltano-sdk--2248.org.readthedocs.build/en/2248/