prometheus: add multicluster support to alerts

Signed-off-by: Aswin Toni <aswin.toni@cern.ch>
ceph · Aug 17, 2022 · 5cdc1c6 · 5cdc1c6
1 parent b578d2e
commit 5cdc1c6
Show file tree

Hide file tree

Showing 8 changed files with 1,364 additions and 1,176 deletions.
diff --git a/monitoring/ceph-mixin/README.md b/monitoring/ceph-mixin/README.md
@@ -5,7 +5,7 @@ All the Grafana dashboards are already generated in the `dashboards_out`
 directory and alerts in the `prometheus_alerts.yml` file.
 
 You can use the Grafana dashboards and alerts with Jsonnet like any other
-prometheus mixin. You can find more ressources about mixins in general on
+prometheus mixin. You can find more resources about mixins in general on
 [monitoring.mixins.dev](https://monitoring.mixins.dev/).
 
 ### Grafana dashboards for Ceph
@@ -26,11 +26,20 @@ plugin](http://docs.ceph.com/en/latest/mgr/prometheus/) and the
 
 
 ### Prometheus alerts
-In `prometheus_alerts.yml` you'll find a set of Prometheus
+In `prometheus_alerts.libsonnet` you'll find a set of Prometheus
 alert rules that should provide a decent set of default alerts for a
-Ceph cluster. Just put this file in a place according to your Prometheus
+Ceph cluster. After building them with jsonnet put this file in place according to your Prometheus
 configuration (wherever the `rules` configuration stanza points).
 
+### Multi-cluster support
+Ceph-mixin supports dashboards and alerts across multiple clusters. 
+To enable this feature you need to configure the following in `config.libsonnnet`:
+
+```
+showMultiCluster: true,
+clusterLabel: '<your cluster label>',
+```
+
 #### SNMP
 Ceph provides a MIB (CEPH-PROMETHEUS-ALERT-MIB.txt) to support sending Prometheus
 alerts through to an SNMP management platform. The translation from Prometheus

diff --git a/monitoring/ceph-mixin/alerts.jsonnet b/monitoring/ceph-mixin/alerts.jsonnet
@@ -0,0 +1 @@
+std.manifestYamlDoc((import 'alerts.libsonnet').prometheusAlerts, indent_array_in_object=true, quote_keys=false)
diff --git a/monitoring/ceph-mixin/alerts.libsonnet b/monitoring/ceph-mixin/alerts.libsonnet
@@ -1,3 +1,3 @@
 {
-  prometheusAlerts+:: std.parseYaml(importstr 'prometheus_alerts.yml'),
+  prometheusAlerts+:: (import 'prometheus_alerts.libsonnet'),
 }
diff --git a/monitoring/ceph-mixin/prometheus_alerts.libsonnet b/monitoring/ceph-mixin/prometheus_alerts.libsonnet
diff --git a/monitoring/ceph-mixin/prometheus_alerts.yml b/monitoring/ceph-mixin/prometheus_alerts.yml
diff --git a/monitoring/ceph-mixin/test-jsonnet.sh b/monitoring/ceph-mixin/test-jsonnet.sh
@@ -19,6 +19,10 @@ do
     done
 done
 
+jsonnet -J vendor -S alerts.jsonnet -o ${TEMPDIR}/prometheus_alerts.yml
+jsondiff --indent 2 "prometheus_alerts.yml" "${TEMPDIR}/prometheus_alerts.yml" \
+    | tee -a ${TEMPDIR}/json_difference.log
+
 err=0
 if [ $(wc -l < ${TEMPDIR}/json_difference.log) -eq 0 ]
 then

diff --git a/monitoring/ceph-mixin/tests_alerts/test_alerts.yml b/monitoring/ceph-mixin/tests_alerts/test_alerts.yml
diff --git a/monitoring/ceph-mixin/tox.ini b/monitoring/ceph-mixin/tox.ini
@@ -22,7 +22,7 @@ whitelist_externals =
     sh
 description =
     check: Ensure that auto-generated files matches the current version
-    fix: Update generated files from jsonnet filse with latest changes
+    fix: Update generated files from jsonnet file with latest changes
     lint: Test if jsonnet files are linted (without any update)
 deps =
     -rrequirements-grafonnet.txt
@@ -56,14 +56,15 @@ whitelist_externals =
 commands =
     behave tests_dashboards/features 
 
-[testenv:alerts-{check,lint}]
+[testenv:alerts-{fix,check,lint}]
 deps =
     -rrequirements-alerts.txt
     pytest
 depends = grafonnet-check
 whitelist_externals =
   promtool
 commands =
+    fix: jsonnet -J vendor -S alerts.jsonnet -o prometheus_alerts.yml
     lint: promtool check rules prometheus_alerts.yml
     test: pytest -rA tests_alerts/test_syntax.py tests_alerts/test_unittests.py
           python3 ./tests_alerts/validate_rules.py