Added alert labels cannot be used in annotation

[cubranic]

What did you do?
When I add a label in an alert via LABELS I would like to be able to use it immediately in the annotation, however that doesn't seem to be possible.

ALERT InstanceDemo
  IF up == 1
  FOR 10s
  LABELS { 
    where = "{{ .Labels.instance | toUpper }}",
    severity = "demo" ,
  }
  ANNOTATIONS {
    summary = "Instance {{ $labels.where }} down",
    description = "{{ $labels.where }} of job {{ $labels.job }} has been down for more than 5 minutes.",
  }

What did you expect to see?
I would expect $labels.where in the annotation description to be replaced with the value I just set in the LABELS section.

What did you see instead? Under which circumstances?
It gets replaced with an empty string.

You can see "where" in the list of labels in the output of curl -sg '<prom-url>/api/v1/query?query=ALERTS, but the contents posted in the Slack channel is "Instance down".

Environment

• System information:

  Linux 4.4.0-83-generic x86_64

• Prometheus version:

prometheus, version 1.6.3 (branch: master, revision: c580b60c67f2c5f6b638c3322161bcdf6d68d7fc)
  build user:       root@a6410e65f5c7
  build date:       20170522-09:15:06
  go version:       go1.8.1

• Alertmanager version

alertmanager, version 0.6.2 (branch: master, revision: b011c0a32ce887c1a10f7d34d52fd8cce485c1cf)
  build user:       root@e3ca4de32142
  build date:       20170509-08:56:14
  go version:       go1.8.1

• Prometheus configuration file:

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  external_labels:
      monitor: 'demo-2454'

rule_files:
  - "alert-demo-2454.rules"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9091']

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9094

• Alertmanager configuration file:

global:
  slack_api_url: https://hooks.slack.com/services/XXXX/YYYY

route:
  receiver: demo-2454

  group_by: ['alertname', 'origin']

  group_wait: 10s

  group_interval: 30s

  # If an alert has successfully been sent, wait 'repeat_interval' to
  # resend them.
  repeat_interval: 1m


receivers:

  - name: demo-2454
    slack_configs:
      - channel: 'alert-testing'
        text: |
          {{ range .Alerts }}
            *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
            *Description:* {{ .Annotations.description }}
            *Details:*
            {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
            {{ end }}
          {{ end }}

[brian-brazil]

This is not supported, nor is it required in this case. You can use .Labels.instance | toUpper as needed, though I'd recommend sticking with the unmodified instance label.

[cubranic]

This is the minimal case to reproduce, in case that wasn't obvious. The real case is more complicated, but is beside the point here, which is that it would be very useful to use in ANNOTATIONS values that were set in LABELS. Based on the documentation, which says that both can be templated, it was surprising to me that they're totally independent.

[brian-brazil]

This has been discussed previously and there's no way to do it that won't cause confusion. The same argument can be made the other way around, and I don't think we want to try and build a partial ordering across annotations and labels.

If you share your actual use case we may be able to make suggestions/improvements.

[cubranic]

Now you say it's been discussed previously, but when I tried to add a point to #2454 you said there that the report is actually about a different use case.

Firstly, the semantics doesn't have to be complicated, or involve partial ordering. E.g., labels are fully defined before annotations are added, and all values defined in LABELS are added at once at the end, so that they are not available except in ANNOTATIONS. (Just like variables defined by let in Lisps.)

The use case in mind is having a phrase constructed from several labels of the original alert, and use it in summary and description annotation. Before you say that these should be added by Prometheus as part of the target label:

• sometimes it's really useful to have this alert-related labelling kept as part of the alert rule rather than a job
• the templating available to define alert labels is far richer than what can programmatically be done with target labels (where you only have regex rewriting)
• creating this more complicated phrase once and attaching it to LABEL means I can just reference it in every other annotation directly as $label.whatever, instead of having to duplicate the templating expression.

[brian-brazil]

labels are fully defined before annotations are added, and all values defined in LABELS are added at once at the end, so that they are not available except in ANNOTATIONS

If we added that we'd just get complaints that it didn't work the other way. And that it was all done at once. No matter what we do here, someone is going to be unhappy.

The use case in mind is having a phrase constructed from several labels of the original alert, and use it in summary and description annotation.

Can you be more specific about your use case? There's several directions something like this can be approached from, and I need more detail to know if none of them apply.

[brian-brazil]

Without further information about your use case I'll have to presume there's some other better way to resolve your use case.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.