Alert labels are not updated for annotation templates after rule label changes

[krasoffski]

Please see #4571 for more info for preconfiguration.

prometheus --version
prometheus, version 2.4.3 (branch: HEAD, revision: 167a4b4e73a8eca8df648d2d2043e21bdb9a7449)
  build user:       root@1e42b46043e9
  build date:       20181004-08:42:02
  go version:       go1.11.1

Only changes comparing with #4571 that labels within alert are changed but within alert manager template engine seem like not.

After changing original alert and reloading Prometheus I got following (pay attention on annotation section and $labels).

groups:
- name: virtual_group
  rules:
  - alert: alert_name
    expr: absent(piupiupiu)
    for: 66s
    labels:
      severity: High
      lkey1: lval1
      lkey2: lval2
      lkey3: lval3
    annotations:
      description: stuff's happening with '{{ . }}'
      olabel: olabel '{{ $labels }}'
      ovalue: ovalue '{{ $value }}'

After adding updating with rule file above alerts are following (old version and new one):

[
    {
        "labels": {
            "alertname": "alert_name",
            "role": "role",
            "rule_prefix": "prefix",
            "severity": "High",
            "text": "my_awesome_alert"
        },
        "annotations": {
            "description": "stuff's happening with '{map[role:role] 1}'"
        },
        "startsAt": "2018-11-01T09:46:28.665326066Z",
        "endsAt": "2018-11-01T10:31:28.665326066Z",
        "generatorURL": "..."
    },
    {
        "labels": {
            "alertname": "alert_name",
            "lkey1": "lval1",
            "lkey2": "lval2",
            "lkey3": "lval3",
            "role": "role",
            "severity": "High"
        },
        "annotations": {
            "description": "stuff's happening with '{map[role:role] 1}'",
            "olabel": "olabel 'map[role:role]'",
            "ovalue": "ovalue '1'"
        },
        "startsAt": "2018-11-01T10:33:28.665326066Z",
        "endsAt": "2018-11-01T10:46:28.665326066Z",
        "generatorURL": "..."
    }
]

[brian-brazil]

What do you think is wrong here? This looks plausible to me.

[krasoffski]

Hi @brian-brazil
After a few hour alert looks like:

[
    {
        "labels": {
            "alertname": "alert_name",
            "lkey1": "lval1",
            "lkey2": "lval2",
            "lkey3": "lval3",
            "role": "role",
            "severity": "High"
        },
        "annotations": {
            "description": "stuff's happening with '{map[role:role] 1}'",
            "olabel": "olabel 'map[role:role]'",
            "ovalue": "ovalue '1'"
        },
        "startsAt": "2018-11-01T10:33:28.665326066Z",
        "endsAt": "2018-11-01T11:48:28.665326066Z",
        "generatorURL": "..."
    }
]

And full alert rule

groups:
- name:virtual_group
  rules:
  - alert: alert_name
    expr: absent(piupiupiu)
    for: 66s
    labels:
      severity: High
      lkey1: lval1
      lkey2: lval2
      lkey3: lval3
    annotations:
      description: stuff's happening with '{{ . }}'
      olabel: olabel '{{ $labels }}'
      ovalue: ovalue '{{ $value }}'

If you take a look at alert rule annotation section and templates

    annotations:
      description: stuff's happening with '{{ . }}'
      olabel: olabel '{{ $labels }}'
      ovalue: ovalue '{{ $value }}'

I did not use specific label but print all existing, for example olabel: olabel '{{ $labels }}'.

I expect to get all labels as a map for olabel like map[alertname:_alert_name lkey1:lval1 lkey2:lval2 lkey3:lval3 role:role severity:High].

But instead $labels contains only old single label from previous version of alert rule "olabel 'map[role:role]'" when last version of alert rule does not have such label at all.

[krasoffski]

After creating new alert rule I got alert with empty $labels or .Labels maps.

Alert:

[
    {
        "labels": {
            "alertname": "ykrasovskiy_alert",
            "lkey1": "lval1",
            "lkey2": "lval2",
            "lkey3": "lval3",
            "rule_prefix": "rule_prefix",
            "severity": "High"
        },
        "annotations": {
            "description": "stuff's happening with '{map[] 1}'",
            "olabel": "olabel 'map[]'",
            "ovalue": "ovalue '1'"
        },
        "startsAt": "2018-11-01T12:32:36.524666141Z",
        "endsAt": "2018-11-01T12:37:36.524666141Z",
        "generatorURL": "..."
    }
]

Especially annotations with empty $labels map:

"annotations": {
    "description": "stuff's happening with '{map[] 1}'",
    "olabel": "olabel 'map[]'",
    "ovalue": "ovalue '1'"
}

I use following alert rule:

groups:
- name: ykrasovskiy_virtual_group
  rules:
  - alert: ykrasovskiy_alert
    expr: absent(piupiupiu)
    for: 66s
    labels:
      lkey1: lval1
      lkey2: lval2
      lkey3: lval3
      rule_prefix: rule_prefix
      severity: High
    annotations:
      description: stuff's happening with '{{ . }}'
      olabel: olabel '{{ $labels }}'
      ovalue: ovalue '{{ $value }}'

[brian-brazil]

You only have access to the labels from the alert expression, otherwise you'd have a circular dependency.

[krasoffski]

@brian-brazil
Thank you, got this.