fix: Use a pool of LabelPairs to reduce memory allocs #2838
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We see pretty heavy memory usage when
querier.v1.QueryService/Series
calls are made on blocks with large aindex.tsdb
. For a real world example, see here.In this flamegraph, a ~7.7mb
index.tsdb
file is being used, resulting in using ~680mb of memory.The culprit is this loop in
getUniqueLabelsSets
:pyroscope/pkg/phlaredb/block_querier.go
Lines 1992 to 2006 in e1a13ca
The high memory usage comes from us allocating a slice every iteration and even more egregiously, allocating a new
typesv1.LabelPair
on the heap every time we have a label name/value match.This PR changes
getUniqueLabelsSets
to use a slice pool strategy, where we allocate a pool of the exact size and fill it with pointers to heap memory. We reuse this pool instead of allocating new memory each iteration ofpostings.Next()
. If we need the contents of the pool, we explicitly copy out of the pool. This substantially reduces memory usage.After the change, the new flamegraph shows
getUniqueLabelsSets
using ~56mb of memory:And some benchmarks using the same 7.7mb
index.tsdb
:Of course, now we have the problem of
index.(*Reader).LabelValueFor
using a lot of memory, but this is a significant and simple enough step that I think it's justifiable to merge on its own.