Labels from alertmanager are empty

[craigwillis85]

What did you do?
I created an alert in my prometheus config to post some basic information about node CPU

What did you expect to see?
mynode.com has high cpu activity where mynode.com is the hostname of my node

What did you see instead? Under which circumstances?
has high cpu activity the host part of the $labels.host is blank

• Prometheus version:

  I'm using the latest (prom/prometheus:latest)

• Alertmanager version:

  I'm using the latest (prom/alertmanager:latest)

An example of the alert is:

ALERT InstanceHighCpu
  IF (100 - avg by(host)(irate(node_cpu{mode='idle'}[5m])) * 100) > 70
  LABELS { severity="info" }
  ANNOTATIONS {
    summary = "Instance {{$labels.host}} cpu high",
    description = "{{$labels.host}} has high cpu activity",
  }

I get the alert in slack, but the host is blank

[brian-brazil]

Do you have a host label on the data?

See also https://www.robustperception.io/controlling-the-instance-label/

[craigwillis85]

@brian-brazil I have the following in my kubernetes configmap

https://gist.github.com/craigwillis85/af3411ad393580bad1cf68e28bb03c41

Where would I add the instance label in my configuration?

Thanks

[brian-brazil]

That doesn't appear to be creating a host label. Try using the instance label in your alert.

[craigwillis85]

I've used the instance label, which was also blank.

I see in alert manager the following:

description xx.xx.xx.xx has high cpu activity
summary Instance xx.xx.xx.xx cpu high

So, it seems that I'm getting the right description/summary in alertmanager
But in my slack alert, i only see:

[FIRING:1] InstanceHighCpu (info)

I added the following to my node

  - source_labels: [__address__]
    target_label: host
    regex: '(.*):.*'

[brian-brazil]

It's not possible for the instance label to be blank with your setup. Can you share your full configs?

[craigwillis85]

https://gist.github.com/craigwillis85/310e50fbea652bb2bdf0def31fb17701

These are my alertmanager and prometheus configs

[brian-brazil]

You're using CommonAnnotations, so you'll only see the description is only one instance is affected.

[craigwillis85]

Should I remove that part from the receivers section?

[brian-brazil]

You could, what you probably want is to range over the Alerts and pull out the annotations from there.

[craigwillis85]

How do i range over the alerts?

[brian-brazil]

There's some examples in https://github.com/prometheus/alertmanager/blob/master/template/default.tmpl

[craigwillis85]

Ok, i managed to get something working:

text: "Nodes: {{ range .Alerts }}{{ .Labels.instance }} {{ end }} \nDescription: {{ .CommonAnnotations.description }} \nSummary: {{ .CommonAnnotations.summary }}"

This displays what i want in slack

@brian-brazil thanks for your help

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.