Active alerts don't take into account templated labels

[FreGalle]

Bug Report

What did you do?

I'm using the following alert, note the dynamic severity label:

- alert: DiskUsageHigh
    expr: (1 - node_filesystem_avail{fstype!="rootfs",mountpoint!="",mountpoint!~"/(run|var).*"} / node_filesystem_size) * 100 > 85
    labels:
      severity: '{{ if gt $value 90.0 }}critical{{ else }}warning{{ end }}'
    annotations:
      summary: 'Disk usage threshold reached'
      description: 'Disk usage of device {{ $labels.device }} on {{ $labels.instance }} (mounted at {{ $labels.mountpoint }}) is at {{ $value|humanize }}%'

What did you expect to see?

I would expect to see the severity label change as soon as this alert's value reaches 90. It doesn't, which I think can be attributed to the way in which active alerts are checked and updated.

In addition to the alert's value and annotations, wouldn't it make sense to also replace its labels to account for the possibly dynamic rule labels?

This used to work pre-v2.

What did you see instead? Under which circumstances?

The alert kept its warning severity label and never escalated to the proper receivers configured to act on critical alerts.

Environment

• Prometheus version:

prometheus, version 2.2.1 (branch: HEAD, revision: bc6058c81272a8d938c05e75607371284236aadc)
  build user:       root@149e5b3f0829
  build date:       20180314-14:15:45
  go version:       go1.10

• Alertmanager version:

alertmanager, version 0.14.0 (branch: HEAD, revision: 30af4d051b37ce817ea7e35b56c57a0e2ec9dbb0)
  build user:       root@37b6a49ebba9
  build date:       20180213-08:16:42
  go version:       go1.9.2

[brian-brazil]

Irrelevant of whether this is a bug, this is a very fragile alert. Alert labels should be constant relative to an alert, and if the value is hovering around the cutover point you may never get a notification. This should be written as two alerts.

…

On Thu, 17 May 2018, 10:21 Frederik, ***@***.***> wrote: Bug Report *What did you do?* I'm using the following alert, note the dynamic severity label: - alert: DiskUsageHigh expr: (1 - node_filesystem_avail{fstype!="rootfs",mountpoint!="",mountpoint!~"/(run|var).*"} / node_filesystem_size) * 100 > 85 labels: severity: '{{ if gt $value 90.0 }}critical{{ else }}warning{{ end }}' annotations: summary: 'Disk usage threshold reached' description: 'Disk usage of device {{ $labels.device }} on {{ $labels.instance }} (mounted at {{ $labels.mountpoint }}) is at {{ $value|humanize }}%' *What did you expect to see?* I would expect to see the severity label change as soon as this alert's value reaches 90. It doesn't, which I think can be attributed to the way in which active alerts are checked and updated <https://github.com/prometheus/prometheus/blob/986674a7902108acb5a863c20496aab03d4f147f/rules/alerting.go#L237-L246> . In addition to the alert's value and annotations, wouldn't it make sense to also replace its labels to account for the possibly dynamic rule labels? This used to work pre-v2. *What did you see instead? Under which circumstances?* The alert kept its warning severity label and never escalated to the proper receivers configured to act on critical alerts. *Environment* - Prometheus version: prometheus, version 2.2.1 (branch: HEAD, revision: bc6058c) build user: ***@***.*** build date: 20180314-14:15:45 go version: go1.10 - Alertmanager version: alertmanager, version 0.14.0 (branch: HEAD, revision: 30af4d051b37ce817ea7e35b56c57a0e2ec9dbb0) build user: ***@***.*** build date: 20180213-08:16:42 go version: go1.9.2 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4169>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGyTdj1XsCnrWOW_ZMFd2Drmf0RwrOXmks5tzTLugaJpZM4UCnIs> .

[FreGalle]

Ok, would it make sense to include this information in the documentation on alerting rules in addition to saying they can be templated. The latter seems to imply no guidelines.

That being said, I thought this was a clean way to keep the alert definitions concise without any duplication.

[brian-brazil]

That being said, I thought this was a clean way to keep the alert definitions concise without any duplication.

There is no safe way to write this alert without duplication. This is a bug in any case, we're using the pre-expansion labels in the hash.

[shubheksha]

I'm gonna take a stab at this!

[csmarchbanks]

@shubheksha Are you still planning to work on this? If not I would like to pick it up.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.