Fix query split logic for LabelNames, LabelValues and ProfileTypes #2852
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While looking at the ingesters memory usage in a high-load environment I noticed that a big portion of it came from
(*singleBlockQuerier) openTSDBIndex
.Looking at the
pyroscopedb_blocks_currently_open
metric, many ingesters have a large number of opened blocks (300+ in some cases). Narrowing that further with traces, the large number of open blocks came from requests toLabelNames
andLabelValues
with long query intervals. Currently these requests are served entirely by ingesters because until recently such requests didn't contain astart
andend
in the body (the client didn't send the range).With grafana/grafana#78861, clients started sending a
start
andend
for the LabelNames and LabelValues APIs. This means we can now properly split the queries between ingesters and store gateways which should in theory reduce the memory pressure on ingesters as described above.Notes:
start
andend
for those requests (a separate Grafana PR to follow)