-
Notifications
You must be signed in to change notification settings - Fork 524
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Limit backfill validation concurrency #4598
Conversation
I tested that mimirtool functions as expected with the first client starts, completes normally
second client starts immediately after first, waits and retries
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you! I like that you also added metric for validations in progress.
Can we also update documentation for API endpoint and mention that it can now return 429? This should also be mentioned in the changelog as [CHANGE]
for the endpoint.
Very nice work on the PR. Thanks also for including example of output from mimirtool and screenshot from testing! |
…ency' into aldernero/backfill-limit-concurrency
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code seems fine; I had a couple of nits, especially one line in CHANGELOG that looks unrelated.
What this PR does
This PR adds a configurable limit to the compactor that sets the maximum number of concurrent backfill validations. This can be used to prevent compactor resources from becoming over-utilized due to the validation phase of block upload which consumes both local storage and CPU resources.
-compactor.max-block-upload-validation-concurrency
, defaults to 1, 0 means unlimitedcortex_block_upload_current_validations
to monitor concurrencyChecklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]