From 2c174a28b0fee594a120964f041f654b5085cde1 Mon Sep 17 00:00:00 2001 From: Callum Styan Date: Wed, 16 Jun 2021 15:30:08 -0700 Subject: [PATCH] Add a simple playbook for ingester series limit alert. Signed-off-by: Callum Styan --- cortex-mixin/docs/playbooks.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/cortex-mixin/docs/playbooks.md b/cortex-mixin/docs/playbooks.md index 16511382..5f0734a1 100644 --- a/cortex-mixin/docs/playbooks.md +++ b/cortex-mixin/docs/playbooks.md @@ -26,7 +26,9 @@ If nothing obvious from the above, check for increased load: ### CortexIngesterReachingSeriesLimit -_TODO: this playbook has not been written yet._ +First check the writes resources dashboard and scaling dashboard. The usual target is 1.5M active series per ingester and a max of 2.5M. + +Scaling up the ingesters will help, but it won't resolve the alert immediately, as series are active until the next TSDB Head compaction (every 2h or so). You may also want to temporarily increase the per ingester series limit (this is a runtime reloadable config option) until that next Head compaction occurs. ### CortexIngesterReachingTenantsLimit