Prometheus turns alert from firing to pending every time value changes.

[joaquimroca]

Since prometheus version v2.4.0 when a firing alert's value increases, the alert turns to pending and start again to evaluate itself. This produces that your firing alert stops and starts again to evaluate itself, so everytime that it happens you receive a new notification.

All versions before v2.4.0, when your value increased, the alert was still in firing (ignoring value changes), so it didn't turns to pending, so you didn't receive new notifications. This is the correct performance if you have this example:

Situation: I want to receive an unique notification when a Pod is restarted more than 10 times (ONLY ONE, IGNORING IF THE VALUE INCREASES!!)

    - alert: PodRestarts-10
      expr: (changes(kube_pod_container_status_restarts_total[1h]) > 10
      for: 1m
      labels:
      annotations:
        description: 'Pod {{ $labels.pod }} restarts > 10, value: {{ $value }}'
        summary: Pod restarts > 10

Once PodRestarts is == 11 (for example), I'll be notified. But If after the value goes to 12, I'll not be notified because the value is still bigger than 11, so the alert doesn't turn to pending.
This is what doesn't happen in versions after 2.4.0 !! Why?? Could we fix it with some parameters??

Thank you very much! :)

[codesome]

Just for a pointer if anyone is looking it at, #4061 was introduced in 2.4.0
(I won't be able to take a look for now)

[joaquimroca]

Yes, I saw it, but i found that if you change old your variable value from labels to anotation the issue is fixed.

Before:

- alert: PodRestarts-10
  expr: (changes(kube_pod_container_status_restarts_total[1h]) > 10
  for: 1m
  labels:
  **Value: {{ $value }}**
  annotations:
    description: 'Pod {{ $labels.pod }} restarts > 10, value: {{ $value }}'
    summary: Pod restarts > 10

After:
- alert: PodRestarts-10
expr: (changes(kube_pod_container_status_restarts_total[1h]) > 10
for: 1m
labels:
annotations:
description: 'Pod {{ $labels.pod }} restarts > 10, value: {{ $value }}'
summary: Pod restarts > 10
Value: {{ $value }}

Thanks

[simonpasquier]

This is working as expected. If putting value in a label, it will resolve existing alerts and create new ones when the value change.