feature request: cancel pushover notifications when the alert resolves itself

[stapelberg]

Currently, when an alert fires while my phone is in silent mode, pushover sends notification after notification, waiting for me to acknowledge them. Notably, even if the alert is resolved by the time I see the alert, I still need to acknowledge the notification. This seems unnecessarily aggressive.

https://pushover.net/api#receipt, section “Canceling Emergency-Priority Retries” describes how to cancel notifications.

I suggest alertmanager should cancel notifications when the corresponding alert is resolved.

[brian-brazil]

I'm not sure that that's the correct behaviour. Just because an alert is no longer firing doesn't mean the underlying issue is resolved.

[stapelberg]

If the underlying issue was transient in nature, there’s nothing one can do about it. Think network issues with an upstream provider. Over-alerting adds to alert fatigue, which I’d like to avoid.

I can see how different people and setups might have different approaches here. Maybe we could make the feature configurable?

[brian-brazil]

If the underlying issue was transient in nature, there’s nothing one can do about it. Think network issues with an upstream provider. Over-alerting adds to alert fatigue, which I’d like to avoid.

The correct solution then is to either adjust or delete the alert. A false positive alert that happens to resolve before you get to it isn't any better than one that resolves after you ack it.

[stapelberg]

I’ve been preaching the “adjust or delete the alert” mantra myself, so I’m familiar with the rationale. However, as soon as third-party systems (in my case: the internet) are involved, it’s often not quite as simple anymore. I don’t get any defined SLA from my ISPs, so I can’t design my alerts to only fire after their alerts have fired.

What I’m doing is best-effort oncall: I want alerts to fire soon-ish (my current threshold for most is 1 hour), but I will not get woken up by an alert that fires at 03:00 in the night. Ideally, my phone wouldn’t buzz and require acknowledgement when do-not-disturb ends at 08:00 sharp in the morning either, because starting the day in a mild panic for an issue that I can do nothing about isn’t a great experience :).

Increasing the alert hold-off time for alerts to ≥ 10 hours isn’t a good option either, as that would leave me flying blind for most of the day.

One alternative solution might be to silence all alerts in alertmanager for the duration of my night. I dislike that solution, however: suddenly I’d need to maintain the do-not-disturb status in both my phone and in alertmanager. This is a hassle when I’m traveling to different time zones, when I’m watching a movie, etc.

I hope this outlines the motivation for the feature.

[brian-brazil]

The feature you're requesting here is merely a workaround for the actual problem.

It sounds like the feature you're really requesting is time of day based alerting.

[stapelberg]

As I tried to explain, time-of-day based alerting seems like an inadequate way to solve the problem to me: I want time-based alerting to use the settings of my pager device. I consider duplicating the setup between my phone and alertmanager too much of a hassle to be practical.

If alertmanager implemented cancelling notifications, my pager setup would just do the right thing.

[brian-brazil]

What you're requesting is well beyond the scope of the alertmanager. Incident management with per-user notification configuration is the responsibility of something like Pagerduty.

[stapelberg]

Well, yes. What I’m saying is that in order for Pushover to implement my desired workflow, Pushover requires API users to not only create notifications upon alerts starting to fire, but also cancel the created notifications when alerts cease to fire.

This seems fairly simple to me from an actual implementation standpoint. Alertmanager just needs to store the Pushover notification identifier with the alert and send one additional HTTP request once the alert stops firing (it already sends a “resolved” notification in that case).

[brian-brazil]

Alertmanager just needs to store the Pushover notification identifier with the alert and send one additional HTTP request once the alert stops firing (it already sends a “resolved” notification in that case).

That's not going to work out technically. Such information wouldn't be synced across HA nodes, and would interact in a non-trivial way with grouping.

Neither the AM nor Pushover are really designed for this use case, and I don't think it's sane to try and make them cover it.

[stapelberg]

That’s a shame, but thanks for the details.

[discordianfish]

I came here to open the exact same issue. I fully agree with @stapelberg about the semantics and it matches what I'm used to and what I want to use pushover for. I think it's completely the right think to resolve the alert at pushover when it's resolved in the alertmanager.

[discordianfish]

It's also exactly what pagerduty is doing IIRC. I'm not sure how it works with grouped alerts, but I'm sure resolved prometheus alerts resolve the alerts in PD.

[SuperQ]

Looking over the other functionality, we do send resolved messages to PagerDuty. It seems like "Cancel" is the equivalent for Pushover. We should support this. Users can control the behavior with send_resolved.