Add estimated compaction jobs based on bucket-index to Mimir dashboards #7449
Conversation
…shboard, and Tenant dashboard. Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
pstibrany
force-pushed
the
compaction-jobs-panels
branch
from
e872edb
to
ab8954e
Compare
| Estimated number of compaction jobs based on latest version of bucket index. Ingesters upload new blocks every 2 hours (shortly after 01:00 UTC, 03:00 UTC, 05:00 UTC, etc.), | ||
| and compactors should process all of them within 2h interval. If this graph regularly goes to zero (or close to zero) in 2 hour intervals, then compaction works as designed. |
There was a problem hiding this comment.
is there any "gotchas" around the estimation? My understanding is that it's only delayed, but in general the number there was true at some point in the past, right?
There was a problem hiding this comment.
My understanding is that it's only delayed, but in general the number there was true at some point in the past, right?
- Yes, "delay" or bucket-index not being 100% up-to-date is one reason.
- Additionally, when computing this number, we only consider jobs that can be executed at given moment. But there are typically more jobs -- however they are blocked. For example job for merging blocks in time range 02:00 -- 04:00 is blocked until all "split" jobs for this time range have finished. Only "split" jobs would be counted toward the estimate and those "merge" jobs would not.
- Another is that when this is computed, there may be ongoing compaction jobs (perhap.
So it's safe to say that computed number was the minimum number of jobs at some recent point in time.
There was a problem hiding this comment.
i was going to suggest to include this in the panel descriptions so it's easier to know how reliable this is and to also explain things like the split/merge jobs shadowing (finished split jobs actually creating more merge jobs)
There was a problem hiding this comment.
Sure, I can update the description.
There was a problem hiding this comment.
Being done here: #7481
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
What this PR does
This PR adds estimated compaction jobs based on bucket-index to Mimir dashboards:
Compactor
Tenant
Top Tenants
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX].about-versioning.mdupdated with experimental features.