Alerting - False alerts and Missing alerts #904
Description
What did you do?
Alert generated for an event(Label sets). Followed by this the event repeated a few times, However since prometheus generates the alert id based on the label sets it made a determination that this was a continuation alert vs new alert and due to the repeat interval no new alert was sent.
What did you expect to see?
Alert sent whenever the event repeats it self, not when it is a continuation.
What did you see instead? Under which circumstances?
New event did not generate a new alert because the label sets were the same.
Environment
Docker
System information:
Darwin ML07DTSG15KG8WL 15.6.0
insert output of uname -srm here
Prometheus version:
1.7.1
insert output of prometheus -version here
Alertmanager version:
0.7.1
insert output of alertmanager -version here (if relevant to the issue)
Prometheus configuration file:
insert configuration here
Alertmanager configuration file:
insert configuration here (if relevant to the issue)
Logs:
insert Prometheus and Alertmanager logs relevant to the issue here
Yes so for example in my case following is the situation:
Service A Working fine - No Alert
Service A Goes down - Alert issued (Taking into account the various intervals)
Service A Remains Down - No Alert Issued(Due to repeat Interval, Assuming this happened within the repeat interval)
Service A comes Back up - No Alert
Service A goes back down - No Alert Issued(Due to repeat Interval, Assuming this happened within the repeat interval) - This is where I was expecting an alert. Seems like the alert id is generated based on the label sets and alert manager thinks of this specific event as a continuation of the first instance of this event.
Basically I am trying to differentiate between an event continuing vs new event for the same service(Same label sets.)