Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix query split logic for LabelNames, LabelValues and ProfileTypes #2852

Merged
merged 1 commit into from
Dec 15, 2023

Conversation

aleks-p
Copy link
Contributor

@aleks-p aleks-p commented Dec 14, 2023

While looking at the ingesters memory usage in a high-load environment I noticed that a big portion of it came from (*singleBlockQuerier) openTSDBIndex.

Looking at the pyroscopedb_blocks_currently_open metric, many ingesters have a large number of opened blocks (300+ in some cases). Narrowing that further with traces, the large number of open blocks came from requests to LabelNames and LabelValues with long query intervals. Currently these requests are served entirely by ingesters because until recently such requests didn't contain a start and end in the body (the client didn't send the range).

With grafana/grafana#78861, clients started sending a start and end for the LabelNames and LabelValues APIs. This means we can now properly split the queries between ingesters and store gateways which should in theory reduce the memory pressure on ingesters as described above.

Notes:

  • The ProfileTypes API is covered here but the behavior for this endpoint won't change until the clients start sending start and end for those requests (a separate Grafana PR to follow)
  • The issue could also be solved with block plans but I've opted for a simple incremental change for now. If this works we could switch entirely to block plans and simplify queriers.
  • Ingesters are currently not closing old blocks that they've opened, making this problem worse over time. If needed we could also add periodic eviction of blocks from memory, though I don't think this will make a big impact following the change done here

@aleks-p aleks-p requested a review from a team as a code owner December 14, 2023 20:20
@cyriltovena
Copy link
Contributor

If needed we could also add periodic eviction of blocks from memory, though I don't think this will make a big impact following the change done here

We should at some point do that for ingester and store-gateway, I think it's more pressing for store-gateways.

Copy link
Contributor

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cyriltovena cyriltovena merged commit 178e882 into main Dec 15, 2023
19 checks passed
@cyriltovena cyriltovena deleted the fix/split-queries-missing-apis branch December 15, 2023 08:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants