Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: Add compactor.max-compaction-time to MimirCompactorHasNotSuccessfullyRunCompaction runbook #2302

Open
56quarters opened this issue Jul 2, 2022 · 2 comments

Comments

@56quarters
Copy link
Contributor

In some situations, the correct way to resolve a MimirCompactorHasNotSuccessfullyRunCompaction alert is to set the compactor.max-compaction-time flag so that only a smaller (several hours) amount of time is spent during the compaction cycle for a single tenant. Per the docs for the flag:

This can help in multi-tenant environments to avoid single tenant using
all compaction time, but also in single-tenant environments to force new
discovery of blocks more often

One instance of the alert triggering was a large number of "backfilled" blocks for a single tenant, monopolizing compactor time.

@56quarters 56quarters added the type/docs Improvements or additions to documentation label Jul 2, 2022
@pracucci
Copy link
Collaborator

pracucci commented Jul 5, 2022

I would argue we should set a reasonable default for that flag. For example, there's no real performance impact (with regards to object storage API calls and overhead in compaction) if we set it to something between 1h and 2h, but would be a better default.

@osg-grafana osg-grafana removed the type/docs Improvements or additions to documentation label Oct 26, 2023
@osg-grafana
Copy link
Contributor

osg-grafana commented Oct 26, 2023

Removing from Docs Team remit because the documentation is operational in nature, and runbooks are maintained by engineers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants