From ebb4d088008bde0d280dcc63f3645591c7d6ac95 Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Thu, 1 Jul 2021 17:38:55 +0200 Subject: [PATCH] Added playbook for CortexProvisioningTooManyWrites Signed-off-by: Marco Pracucci --- cortex-mixin/docs/playbooks.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/cortex-mixin/docs/playbooks.md b/cortex-mixin/docs/playbooks.md index 4373bad8..825fa1c6 100644 --- a/cortex-mixin/docs/playbooks.md +++ b/cortex-mixin/docs/playbooks.md @@ -440,7 +440,14 @@ _TODO: this playbook has not been written yet._ ### CortexProvisioningTooManyWrites -_TODO: this playbook has not been written yet._ +This alert fires if the average number of samples ingested / sec in ingesters is above our target. + +How to **fix**: +- Scale up ingesters + - To compute the desired number of ingesters to satisfy the average samples rate you can run the following query, replacing `` with the namespace to analyse and `` with the target number of samples/sec per ingester (check out the alert threshold to see the current target): + ``` + sum(rate(cortex_ingester_ingested_samples_total{namespace=""}[$__rate_interval])) / ( * 0.9) + ``` ### CortexAllocatingTooMuchMemory