From 51ed50926b389d7768439d35562ff6d7b8bf5195 Mon Sep 17 00:00:00 2001 From: Eliska Romanova Date: Tue, 23 Apr 2024 09:27:58 +0200 Subject: [PATCH] OBSDOCS-961: Update docs to explain persistence of silences when multiple Alertmanager pods are restarted --- modules/monitoring-configuring-persistent-storage.adoc | 9 +++++++-- modules/monitoring-managing-silences.adoc | 5 +++++ observability/monitoring/managing-alerts.adoc | 4 ++++ 3 files changed, 16 insertions(+), 2 deletions(-) diff --git a/modules/monitoring-configuring-persistent-storage.adoc b/modules/monitoring-configuring-persistent-storage.adoc index 279258649f12..f50b48ec41fa 100644 --- a/modules/monitoring-configuring-persistent-storage.adoc +++ b/modules/monitoring-configuring-persistent-storage.adoc @@ -3,10 +3,15 @@ // * observability/monitoring/configuring-the-monitoring-stack.adoc :_mod-docs-content-type: CONCEPT -[id="configuring_persistent_storage_{context}"] +[id="configuring-persistent-storage_{context}"] = Configuring persistent storage -Running cluster monitoring with persistent storage means that your metrics are stored to a persistent volume (PV) and can survive a pod being restarted or recreated. This is ideal if you require your metrics or alerting data to be guarded from data loss. For production environments, it is highly recommended to configure persistent storage. Because of the high IO demands, it is advantageous to use local storage. +Run cluster monitoring with persistent storage to gain the following benefits: + +* Protect your metrics and alerting data from data loss by storing them in a persistent volume (PV). As a result, they can survive pods being restarted or recreated. +* Avoid getting duplicate notifications and losing silences for alerts when the Alertmanager pods are restarted. + +For production environments, it is highly recommended to configure persistent storage. Because of the high IO demands, it is advantageous to use local storage. [id="persistent-storage-prerequisites"] == Persistent storage prerequisites diff --git a/modules/monitoring-managing-silences.adoc b/modules/monitoring-managing-silences.adoc index cfc087081bf5..5dd97f8be713 100644 --- a/modules/monitoring-managing-silences.adoc +++ b/modules/monitoring-managing-silences.adoc @@ -11,3 +11,8 @@ You can create a silence to stop receiving notifications about an alert when it When creating a silence, you must specify whether it becomes active immediately or at a later time. You must also set a duration period after which the silence expires. You can view, edit, and expire existing silences. + +[NOTE] +==== +When you create silences, they are replicated across Alertmanager pods. However, if you do not configure persistent storage for Alertmanager, silences might be lost. This can happen, for example, if all Alertmanager pods restart at the same time. +==== \ No newline at end of file diff --git a/observability/monitoring/managing-alerts.adoc b/observability/monitoring/managing-alerts.adoc index a68e50bd32e8..f8d36dd055b8 100644 --- a/observability/monitoring/managing-alerts.adoc +++ b/observability/monitoring/managing-alerts.adoc @@ -38,6 +38,10 @@ include::modules/monitoring-getting-information-about-alerts-silences-and-alerti // Managing silences include::modules/monitoring-managing-silences.adoc[leveloffset=+1] +[role="_additional-resources"] +.Additional resources +* xref:../../observability/monitoring/configuring-the-monitoring-stack.adoc#configuring-persistent-storage_configuring-the-monitoring-stack[Configuring persistent storage] + include::modules/monitoring-silencing-alerts.adoc[leveloffset=+2] include::modules/monitoring-editing-silences.adoc[leveloffset=+2] include::modules/monitoring-expiring-silences.adoc[leveloffset=+2]