[FEATURE REQUEST] Alertmanager metrics for silenced alerts

[arctic-sharar]

Problem Statement

Prometheus supplies an ALERTS metric tracking the firing alerts. Alertmanager has the ability to silence metrics. But there is no metrics exposed by Alertmanager to track the alerts that are silenced.

Proposed Solution

Introduce an alertmanager metric for alerts that are being silenced. This can further be extended to other types of suppressions (inhibition rules)

Benefits

• Improved tracking of silence
• Better Auditing

[grobinson-grafana]

You might be looking for alertmanager_marked_alerts where state=suppressed? This includes other suppressions though, such as inhibitions. Did you want separate metrics for silences and inhibitions?

[adam-elementor]

@grobinson-grafana It would help us if there was a 'state=suppressed' in the ALERTS alertstate label, from what I see the alertmanager_marked_alerts only shows the number of suppressed, not data about them.

I wish to create a panel for 'silenced alerts' and show the exact alert that is silenced, is there a different approach to this?

[grobinson-grafana]

@grobinson-grafana It would help us if there was a 'state=suppressed' in the ALERTS alertstate label, from what I see the alertmanager_marked_alerts only shows the number of suppressed, not data about them.

I wish to create a panel for 'silenced alerts' and show the exact alert that is silenced, is there a different approach to this?

It won't include data about them such as the fingerprint or labels because the cardinality is too high. If you want fine-grained information about which alerts are silenced, you can get all silenced alerts from the API using /api/v2/alerts?silenced=true and show this on the panel.

[wbollock]

I have a use-case for a high cardinality metric that can be joined with ALERTS to show that specific alerts were silenced. A feature flag would be fine as most users wouldn't love the cardinality but this allows an operator to determine exactly how many of their ALERTS from Prometheus were actually silenced. alertname would be fine as the only high cardinality label in this metric. Note that this is different from other proposals wanting metrics summarizing the amount of suppressed alerts, I believe a time series on the exact alerts that were suppressed would be quite useful!