Alert template expansion is broken for global labels

[dan-cleinmark]

Per @juliusv on IRC: "the template expansion happens before the global labels are attached: https://github.com/prometheus/prometheus/blob/master/rules/manager.go#L221-L222". This results in references to global labels to not expand in alerts.

Relevant bits from an email alert:

Subject: [ALERT] Instance_prometheus_node_exporter_is_broken: <no value>/xxx_node is BROKEN!

<no value>: prometheus xxx_node exporter is broken

Alertmanager: http://xxx:9093/
Runbook entry: < link >

Grouping labels:

  alertname = "Instance_prometheus_node_exporter_is_broken"
  domain = "prometheus"
  env = "ci"
  instance = "xxx_node"
  job = "xxx"
  service = "xxx"
  severity = "email"

Payload labels:

  activeSince = "2015-09-28T23:59:40.37Z"
  alertingRule = "ALERT Instance_prometheus_node_exporter_is_broken
    IF up{job=~"^.+_node$"} == 0
    FOR 5m
    WITH {domain="prometheus", severity="email"}
    SUMMARY "{{$labels.env}}/{{$labels.job}} is BROKEN!"
    DESCRIPTION "{{$labels.env}}: prometheus {{$labels.job}} exporter is broken"
    RUNBOOK "< link >"
  generatorURL = "http://xxx:9090/graph#%5B%7B%22expr%22%3A%22up%7Bjob%3D~%5C%22%5E.%2B_node%24%5C%22%7D%20%3D%3D%200%22%2C%22tab%22%3A0%7D%5D"
  value = "0"

[brian-brazil]

For this usage, expecting every single alert to remember to put env in their summary and description is going to be very repetitive for you. This is a perfect use case for alert notification templates, once we have them as what you're really trying to do is format your notifications consistently.

[juliusv]

@brian-brazil Agreed that this use case would better be solved via notification templates on the Alertmanager side. However, is there harm in also setting the global labels here? In the worst case, someone uses them where they weren't able to use them before...

[brian-brazil]

In the worst case, someone uses them where they weren't able to use them before...

I'd generally consider it bad practice to use global labels in an alert definition, as that's not really maintainable as the structure of an organisation's global labels changes over time.

I think we should be encouraging that only the labels of the alert expression be used.

[juliusv]

Ok, closing this issue here then - AM notification templates will solve this use case better.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.