-
Notifications
You must be signed in to change notification settings - Fork 513
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ingester: include only owned series in active series stats #7976
Comments
there was some discussion in slack with @pstibrany and @pr00se. I'm leaving here to gist of it. There were two concerns
|
from @pstibrany
after the rollout of zone-a early compaction happened twice more in zone-a than in the other zones; but still overall zone-a is at lower levels of early compaction than before |
Context
With the kafka architecture scaling ingesters is easier and is expected to happen more often. Series change owners when ingersters are added. For 10 minutes (value of
-ingester.active-series-metrics-idle-timeout
) both the new ingester and some old ingesters will count a series as active. This means that for this time a tenant would appear to have increased the number of active series when in reality they haven't.Proposal
Use active series when calculating active series stats. This means the ingester will reevaluate what it considers active series every time the ring changes. This will show as changes in the values of the following metrics
cortex_ingester_active_series
cortex_ingester_active_series_custom_tracker
cortex_ingester_active_native_histogram_series
cortex_ingester_active_native_histogram_series_custom_tracker
cortex_ingester_active_native_histogram_series_custom_tracker
cortex_ingester_active_native_histogram_buckets_custom_tracker
/api/v1/cardinality/active_series
endpointProposed implementation
Head.ForEachSecondaryHash()
in mimir-prometheus to return the ref of the series (storage.SeriesRef) as well as the secondary hashActiveSeries.Delete(r storage.SeriesRef)
which behaves very similar toActiveSeries.purge()
ActiveSeries.Delete
for every series which isn't owned anymore; the call would happen somewhere arounduserTSDB.recomputeOwnedSeries
The text was updated successfully, but these errors were encountered: