[FEATURE REQUEST] Alertmanager receiver for Suppressed Alerts

[arctic-sharar]

Problem Statement

Currently, Alertmanager does not expose metrics for silenced alerts, making it difficult to track them without manually navigating to the Alertmanager UI. Also, polling the /silences endpoint to know what alerts are silenced is inconvenient. The issue is better solved upstream on alertmanager side.

Proposed Solution

Add an optional suppressed receiver for an alert matcher: For an alert that matches an alertmanager route and has some sort of suppress configured, Alertmanager will hit the suppressed receiver instead when alert.state is suppressed. Otherwise, no-ops. This implementation is simple as there are existing egress mechanisms doing this for firing state.

Example Scenario

During a planned maintenance window, an alert is silenced. The new egress mechanism will notify a new receiver responsible for handling silences. This is very useful for critical services undergoing maintenance and require silencing to take further actions.

Benefits

• Automated Notifications
• Improved Visibility
• Efficient Monitoring
• Allows Auditing

[grobinson-grafana]

You can filter suppressed alerts via /api/v2/alerts. Would that be useful? I'm not sure a silenced receiver makes sense.

[arctic-sharar]

I did mention about using the silenced apis but a suppressed endpoint firing would allow the management of alerts that has been silenced more naturally.

We regularly undergo maintenance work and having to poll an endpoint and make a service just to poll and gather alerts from /api/v2/alerts adds complications + causes this management of alerts and silences to be done in separate places.

Whereas, having a way to route the suppressed alerts (similar to firing alerts) naturally solves the problem and keeps the configuration in a single spot.

I know that can cause a shift in how we think about alerts, but Alertmanager in my opinion should provide options to how to manage its alerts depending on its state.