send alert on reload error

[leo-baltus]

What did you do?
We made a error in the alerting rules configuration.

What did you expect to see?
I want to have an alert triggered, since prometheus can not be restarted.

What did you see instead? Under which circumstances?
I think this is a missing feature.

Environment
CentOS7, Docker-1.10.3

• System information:
  Linux 3.10.0-327.36.3.el7.x86_64 x86_64

• Prometheus version:
  prometheus, version 1.2.1 (branch: master, revision: dd66f2e)
  build user: root@fd9b0daff6bd
  build date: 20161010-15:58:23
  go version: go1.7.1

• Alertmanager version:
  alertmanager, version 0.4.2 (branch: master, revision: 9a5ab2fa63dd7951f4f202b0846d4f4d8e9615b0)
  build user: root@2811d2f42616
  build date: 20160902-15:33:13
  go version: go1.6.3

• Prometheus configuration file:
  n/a

• Alertmanager configuration file:
  n/a

• Logs:
  Failed to apply configuration: error loading rules, previous rule set restored

[brancz]

Prometheus has a config_last_reload_successful metric. You can alert on that just like any other metric.

[brancz]

However, I would additionally recommend checking your rule files with promtool before applying them. This should typically be done within your CI pipeline. Rule files are really also just code :)

[leo-baltus]

Thanks for the pointers.

[grobie]

It's already possible to create an alert based on config_last_reload_successful.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.