New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Decoupled blocks deletion from compaction #2623
Decoupled blocks deletion from compaction #2623
Conversation
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
|
||
// Ensure an initial cleanup occurred before starting the compactor. | ||
if err := services.StartAndAwaitRunning(ctx, c.blocksCleaner); err != nil { | ||
return errors.Wrap(err, "failed to start the blocks cleaner") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't matter too much in Cortex, but it would be good to call ringSubservices.StopAsync()
if not nil. (Or StopAsync
can be make to work with nil receiver too, which would be nice.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very good point. Done.
Signed-off-by: Marco Pracucci <marco@pracucci.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
What this PR does:
While running the blocks compactor at scale, we reliased that blocks marked for deletion could be potentially never deleted if the compactor is busy all the time compacting blocks.
In this PR I've decoupled blocks deletion from compaction, so that blocks deletion is not blocked by a busy compactor.
Which issue(s) this PR fixes:
N/A
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]