Support Sub Alerts

[roidelapluie]

In order to have better alerting, I would like to have prometheus support for «sub alerts»:

Example(Pseudo Code):

• Alert when disk usage > 95%
  • Until: disk usage > 90%

It means that the alert fires when the main alert is firing, and keeps firing until all the subalerts and the main one are solved.

Sub alerts are only computed when the first alert is firing or all the sub alerts are not resolved.

[brian-brazil]

In Prometheus our philosophy is that alerts should be symptom based rather than cause based, and require intelligent human action (see https://prometheus.io/docs/practices/alerting/). If you're depending on hysteresis or other logic to indicate that a problem is "fixed", then that is an indication that you're probably doing cause-based alerting and trying to deal with the noisiness of that tactic rather than using the symptom-based strategy. If an alert fires it should be severe enough to call in a human, and what the alert does after that isn't too important - as the human is already there to determine the issue and when it's really resolved.

See also https://www.robustperception.io/running-into-burning-buildings-because-the-fire-alarm-stopped

[roidelapluie]

We have lots of alerts targetted at automation systems for healing ; not just people.