Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alerting rules docs reference $labels, but only .Labels appear to work #4859

Closed
mkadin opened this Issue Nov 12, 2018 · 3 comments

Comments

Projects
None yet
2 participants
@mkadin
Copy link

mkadin commented Nov 12, 2018

Bug Report

What did you do?
I set up a rule that looks like this:

  - alert: Server memory utilization
    expr: (node_memory_MemTotal{instance=~"(job|app|web).*"} -node_memory_MemAvailable{instance=~"(job|app|web).*"})/ node_memory_MemTotal{instance=~"(job|app|web).*"} * 100 > 80
    for: 5m
    labels:
      service: server
      severity: high
    annotations:
      summary: {{$labels.instance}} memory usage > 80%

What did you expect to see?
For this configuration to be valid, based on the docs at https://github.com/prometheus/prometheus/blob/master/docs/configuration/alerting_rules.md

What did you see instead? Under which circumstances?
Prometheus crashed on startup:

level=info ts=2018-11-12T18:02:26.966132819Z caller=targetmanager.go:71 component="target manager" msg="Starting target manager..."
level=info ts=2018-11-12T18:02:27.553405224Z caller=main.go:326 msg="TSDB started"
level=info ts=2018-11-12T18:02:27.553995989Z caller=main.go:394 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=error ts=2018-11-12T18:02:27.557580751Z caller=manager.go:485 component="rule manager" msg="loading groups failed" err="yaml: line 10: did not find expected key"
level=error ts=2018-11-12T18:02:27.557859605Z caller=main.go:413 msg="Failed to apply configuration" err="error loading rules, previous rule set restored"
level=error ts=2018-11-12T18:02:27.558245459Z caller=main.go:356 msg="Error loading config" err="one or more errors occurred while applying the new configuration (--config.file=/etc/prometheus/prometheus.yml)"

Environment
ubuntu 16.04

  • System information:
ubuntu@infra1-us-east-2:~$ uname -srm
Linux 4.4.0-1038-aws x86_64
  • Prometheus version:
ubuntu@infra1-us-east-2:~$ /src/prometheus/prometheus-server/prometheus --version
prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98628a0463dddc90528220c94de5032d1a0)
  build user:       root@615b82cb36b6
  build date:       20171108-07:11:59
  go version:       go1.9.2
  • Alertmanager version:
ubuntu@infra1-us-east-2:~$ /src/prometheus/prometheus-server/prometheus --version
prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98628a0463dddc90528220c94de5032d1a0)
  build user:       root@615b82cb36b6
  build date:       20171108-07:11:59
  go version:       go1.9.2
  • Prometheus configuration file:
    see above

  • Alertmanager configuration file:
    see above

@mkadin

This comment has been minimized.

Copy link
Author

mkadin commented Nov 12, 2018

I should have added, once I changes $labels to .Labels, it worked properly! Did this variable change names and the docs need updating?

@brian-brazil

This comment has been minimized.

Copy link
Member

brian-brazil commented Nov 12, 2018

That's a YAML issue, try quoting the template string.

@mkadin

This comment has been minimized.

Copy link
Author

mkadin commented Nov 13, 2018

You're right...sorry for the bad report!

@mkadin mkadin closed this Nov 13, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.