mgr/dashboard: Prometheus documentation

The documentation explains how to configure the alertmanager to use the
dashboard receiver and how to connect the use of the alertmanager API.
Further it explains where to find the alerts and what happens if they
are configured and something is happening.

Fixes: https://tracker.ceph.com/issues/36721
Signed-off-by: Stephan Müller <smueller@suse.com>

doc/mgr/dashboard.rst

@@ -368,6 +368,81 @@ To enable SSO::
 
   $ ceph dashboard sso enable saml2
 
+Enabling Prometheus alerting
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Using Prometheus for monitoring, you have to define `alerting rules
+<https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules>`_.
+To manage them you need to use the `Alertmanager
+<https://prometheus.io/docs/alerting/alertmanager>`_.
+If you are not using the Alertmanager yet, please `install it
+<https://github.com/prometheus/alertmanager#install>`_ as it's mandatory in
+order to receive and manage alerts from Prometheus.
+
+The Alertmanager capabilities can be consumed by the dashboard in three different
+ways:
+
+#. Use the notification receiver of the dashboard.
+
+#. Use the Prometheus Alertmanager API.
+
+#. Use both sources simultaneously.
+
+All three methods are going to notify you about alerts. You won't be notified
+twice if you use both sources.
+
+#. Use the notification receiver of the dashboard:
+
+   This allows you to get notifications as `configured
+   <https://prometheus.io/docs/alerting/configuration/>`_ from the alertmanager.
+   You will get notified inside the dashboard once a notification is send out,
+   but you are not able to manage alerts.
+
+   Add the dashboard receiver and the new route to your Alertmanager configuration.
+   This should look like::
+
+     route:
+       receiver: 'ceph-dashboard'
+     ...
+     receivers:
+       - name: 'ceph-dashboard'
+         webhook_configs:
+         - url: '<url-to-dashboard>/api/prometheus_receiver'
+
+
+   Please make sure that the Alertmanager considers your SSL certificate in terms
+   of the dashboard as valid. For more information about the correct
+   configuration checkout the `<http_config> documentation
+   <https://prometheus.io/docs/alerting/configuration/#%3Chttp_config%3E>`_.
+
+#. Use the API of the Prometheus Alertmanager
+
+   This allows you to manage alerts. You will see all alerts, the Alertmanager
+   currently knows of, in the alters listing. It can be found in the *Cluster*
+   submenu as *Alerts*. The alerts in can be sorted by name, job, severity,
+   state and start time. Sadly it's not possible to know when a alert was send
+   out through a notification by the Alertmanager based on your configuration,
+   that's why the dashboard will notify the user on any visible change to an
+   alert and will notify the changed alert.
+
+   Currently it's not yet possible to silence an alert and expire an silenced
+   alert, but this is work in progress and will be coming soon.
+
+   To use it, specify the host and port of the Alertmanager server::
+
+     $ ceph dashboard set-alertmanager-api-host <alertmanager-host:port>  # default: ''
+
+   For example::
+
+     $ ceph dashboard set-alertmanager-api-host 'http://localhost:9093'
+
+
+#. Use both methods
+
+   The different behaviors of both methods are configured in a way that they
+   should not disturb each other through annoying duplicated notifications
+   popping up.
+
 Accessing the dashboard
 ^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -446,6 +521,7 @@ scopes are:
   management.
 - **log**: include all features related to Ceph logs management.
 - **grafana**: include all features related to Grafana proxy.
+- **prometheus**: include all features related to Prometheus alert management.
 - **dashboard-settings**: allows to change dashboard settings.
 
 A *role* specifies a set of mappings between a *security scope* and a set of