feat: map ems severity to prom sev

[7840vz]

Instead of using static 'critical', use label 'severity' coming from EMS and maps to recommended prometheus scale of info/warning/critical: https://monitoring.mixins.dev/#guidelines-for-alert-names-labels-and-annotations
Known severity from ems: emergency, alert, error, informational, and notice
Severities suggested for use in alertmanager/prom: critical/warning/info.

[cgrinds]

Thanks @7840vz!

promtool check rules passes

Do you know if Prom alerts.yaml supports anchors/aliases to help reduce duplication? I haven't tried, but thought you might know. I know support for anchors/aliases varies a lot by tool and it may not be worth it to reduce this duplication.

[7840vz]

hi! Regarding, anchors, unsure if it would work: prometheus/prometheus#6266

Turning your dashboard and alerts into monitoring-mixin might help though:
https://monitoring.mixins.dev/

some examples:
Kube: https://github.com/kubernetes-monitoring/kubernetes-mixin
Node: https://github.com/prometheus/node_exporter/tree/master/docs/node-mixin

Here , alerts can be redefined in jsonnet format and actual yamls can regenerated from Makefile or other tooling.

[cgrinds]

Thanks. Yes, we've discussed jsonnet in the past, but only for dashboards, where there were some bigger trade-offs. For alerts, jsonnet may be a better fit.