From 35fa27fde27ab9f992ad806cd313ed9134b1c86a Mon Sep 17 00:00:00 2001 From: Eliska Romanova Date: Tue, 23 Apr 2024 09:27:58 +0200 Subject: [PATCH] OBSDOCS-961: Update docs to explain persistence of silences when multiple Alertmanager pods are restarted --- modules/monitoring-configuring-persistent-storage.adoc | 9 +++++++-- modules/monitoring-managing-silences.adoc | 5 +++++ observability/monitoring/managing-alerts.adoc | 4 ++++ 3 files changed, 16 insertions(+), 2 deletions(-) diff --git a/modules/monitoring-configuring-persistent-storage.adoc b/modules/monitoring-configuring-persistent-storage.adoc index 279258649f12..f50b48ec41fa 100644 --- a/modules/monitoring-configuring-persistent-storage.adoc +++ b/modules/monitoring-configuring-persistent-storage.adoc @@ -3,10 +3,15 @@ // * observability/monitoring/configuring-the-monitoring-stack.adoc :_mod-docs-content-type: CONCEPT -[id="configuring_persistent_storage_{context}"] +[id="configuring-persistent-storage_{context}"] = Configuring persistent storage -Running cluster monitoring with persistent storage means that your metrics are stored to a persistent volume (PV) and can survive a pod being restarted or recreated. This is ideal if you require your metrics or alerting data to be guarded from data loss. For production environments, it is highly recommended to configure persistent storage. Because of the high IO demands, it is advantageous to use local storage. +Run cluster monitoring with persistent storage to gain the following benefits: + +* Protect your metrics and alerting data from data loss by storing them in a persistent volume (PV). As a result, they can survive pods being restarted or recreated. +* Avoid getting duplicate notifications and losing silences for alerts when the Alertmanager pods are restarted. + +For production environments, it is highly recommended to configure persistent storage. Because of the high IO demands, it is advantageous to use local storage. [id="persistent-storage-prerequisites"] == Persistent storage prerequisites diff --git a/modules/monitoring-managing-silences.adoc b/modules/monitoring-managing-silences.adoc index 15104adcc283..9f69c9f2726b 100644 --- a/modules/monitoring-managing-silences.adoc +++ b/modules/monitoring-managing-silences.adoc @@ -14,3 +14,8 @@ Creating silences is useful in scenarios where you have received an initial aler When creating a silence, you must specify whether it becomes active immediately or at a later time. You must also set a duration period after which the silence expires. After you create silences, you can view, edit, and expire them. + +[NOTE] +==== +When you create silences, they are replicated across Alertmanager pods. However, if you do not configure persistent storage for Alertmanager, silences might be lost. This can happen, for example, if all Alertmanager pods restart at the same time. +==== diff --git a/observability/monitoring/managing-alerts.adoc b/observability/monitoring/managing-alerts.adoc index b32e2025e53d..96f7dd9ca4aa 100644 --- a/observability/monitoring/managing-alerts.adoc +++ b/observability/monitoring/managing-alerts.adoc @@ -38,6 +38,10 @@ include::modules/monitoring-getting-information-about-alerts-silences-and-alerti // Managing silences include::modules/monitoring-managing-silences.adoc[leveloffset=+1] +[role="_additional-resources"] +.Additional resources +* xref:../../observability/monitoring/configuring-the-monitoring-stack.adoc#configuring-persistent-storage_configuring-the-monitoring-stack[Configuring persistent storage] + include::modules/monitoring-silencing-alerts.adoc[leveloffset=+2] include::modules/monitoring-editing-silences.adoc[leveloffset=+2] include::modules/monitoring-expiring-silences.adoc[leveloffset=+2]