run multiple queries per table at once with boltdb-shipper #2656
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #2656 +/- ##
==========================================
+ Coverage 61.19% 61.25% +0.05%
==========================================
Files 172 173 +1
Lines 13363 13390 +27
==========================================
+ Hits 8178 8202 +24
Misses 4438 4438
- Partials 747 750 +3
|
sandeepsukhani
force-pushed
the
boltdb-shipper-multi-query
branch
2 times, most recently
from
d6081db
to
2a17fea
Compare
sandeepsukhani
force-pushed
the
boltdb-shipper-multi-query
branch
from
2a17fea
to
f8fea78
Compare
cyriltovena
pushed a commit
to cyriltovena/loki
that referenced
this pull request
Jun 11, 2021
It broke ingester hand-overs, and is not needed now we use BigChunk. I left in the code which allows non-full-size chunks to be read in: maybe someone has some in a database. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
While investigating query performance I found labels query to be slower because it does a lot of index queries, with our internal cluster it was close to 33k queries. After digging deeper I found we were doing mutex and channel operation in downloads path for each query here which is inefficient.
This PR changes the code to do mutex and channel operation and run all queries per table at once and run. With my local testing I found 7x improvement in query performance with this change.
Checklist