Implement concurrent pushes across batches #3206
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #3206 +/- ##
==========================================
+ Coverage 72.68% 73.24% +0.55%
==========================================
Files 259 259
Lines 19864 19889 +25
==========================================
+ Hits 14439 14567 +128
+ Misses 4521 4400 -121
- Partials 904 922 +18
Flags with carried forward coverage won't be shown. Click here to find out more.
|
olegbespalov
force-pushed
the
cloudv2-concurrency-per-batches
branch
2 times, most recently
from
16be249
to
e2f4dc6
Compare
olegbespalov
force-pushed
the
cloudv2-concurrency-per-batches
branch
from
e2f4dc6
to
459cac9
Compare
olegbespalov
changed the title
olegbespalov
changed the title
mstoykov
reviewed
Jul 24, 2023
oleiade
reviewed
Jul 24, 2023
There was a problem hiding this comment.
LGTM too 🤝
Non-blocking on my part, but I share the same concern regarding the work dispatching to multiple workers as @mstoykov. I think the code could be simplified indeed, to have one input channel, and multiple workers reading from it concurrently, I find it might also make the logic easier to follow.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What?
This change moves the concurrency from flushes to batches. That fixes issue #3192 and generally improves processing.
Why?
Even if the single network request to ingester is fast enough, executing them sequentially increases the time.
Checklist
make ci-like-lint) and all checks pass.make tests) and all tests pass.Related PR(s)/Issue(s)
Closes #3192