Skip to content

Commit

Permalink
Make CortexIngesterReachingSeriesLimit warning less sensitive
Browse files Browse the repository at this point in the history
As it turns out, during normal shuffle-sharding operation, the 70%
mark is often exceeded, but not by much. Therefore, this change sets
the new warning mark at 75%. It also increases the `for` duration to
15m as the expected reaction time for warning alerts is usually in the
order of hours, so we can as well wait a bit longer to see if the
problem is transient.

Signed-off-by: beorn7 <beorn@grafana.com>
  • Loading branch information
beorn7 committed Jul 26, 2021
1 parent 0d48edf commit c6eab23
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions cortex-mixin/alerts/alerts.libsonnet
Original file line number Diff line number Diff line change
Expand Up @@ -255,9 +255,9 @@
(cortex_ingester_memory_series / ignoring(limit) cortex_ingester_instance_limits{limit="max_series"})
and ignoring (limit)
(cortex_ingester_instance_limits{limit="max_series"} > 0)
) > 0.7
) > 0.75
|||,
'for': '5m',
'for': '15m',
labels: {
severity: 'warning',
},
Expand Down

0 comments on commit c6eab23

Please sign in to comment.