Prometheus doesn't reload rule files obtained with glob

[michael-doubez]

Bug Report

Rule files are not reloaded when modifying them or sending the HUP signal.

What did you do?

I configured multiple rule files in a directory:

The rules are correctly loaded at startup but when modifying them, they are not reloaded:

• when modifying them
• when sending HUP signal to Prometheus

What did you expect to see?

Prometheus reloading the rule files and updating internal rules (as seen in GUI).

What did you see instead? Under which circumstances?

No trace of rule reload in logs and rules not updated in GUI

Environment

• System information:

  Linux 2.6.32-696.1.1.el6.x86_64 x86_64

• Prometheus version:

  prometheus, version 2.5.0 (branch: HEAD, revision: 67dc912)
  build user: root@578ab108d0b9
  build date: 20181106-11:40:44
  go version: go1.11.1

• Prometheus configuration file:

rule_files:
  - alerts/*.rule.yml

• Logs:

...
level=info ts=2019-01-21T23:05:10.169104815Z caller=main.go:632 msg="Loading configuration file" filename=XXXX
level=info ts=2019-01-21T23:05:10.172320287Z caller=main.go:658 msg="Completed loading of configuration file" filename=XXXX
...
level=info ts=2019-01-22T11:00:30.210078663Z caller=main.go:632 msg="Loading configuration file" filename=XXXX
level=info ts=2019-01-22T11:00:39.844110776Z caller=main.go:658 msg="Completed loading of configuration file" filename=XXXX

[simonpasquier]

Hmm, I can't reproduce the issue. Can you check the following metrics?

• prometheus_config_last_reload_successful
• prometheus_config_last_reload_success_timestamp_seconds
• prometheus_rule_group_rules

[michael-doubez]

In the meantime, I restarted Prometheus to reload the rules.
Retrying, I get:

Logs

level=info ts=2019-01-22T11:16:10.812077822Z caller=main.go:632 msg="Loading configuration file" filename=XXXX
level=info ts=2019-01-22T11:16:10.815724422Z caller=main.go:658 msg="Completed loading of configuration file" filename=XXXX
level=info ts=2019-01-22T11:16:10.81578853Z caller=main.go:531 msg="Server is ready to receive web requests."
level=info ts=2019-01-22T13:00:01.948982038Z caller=compact.go:398 component=tsdb msg="write block" mint=1548151200000 maxt=1548158400000 ulid=01D1TTASBABJ2305NQS5BP2DEJ
level=info ts=2019-01-22T13:00:02.150514894Z caller=head.go:488 component=tsdb msg="head GC completed" duration=103.668551ms
level=info ts=2019-01-22T13:00:03.628165541Z caller=head.go:535 component=tsdb msg="WAL checkpoint complete" low=504 high=505 duration=1.47757593s
level=info ts=2019-01-22T14:11:34.454150345Z caller=main.go:632 msg="Loading configuration file" filename=XXX
level=info ts=2019-01-22T14:11:42.77715041Z caller=main.go:658 msg="Completed loading of configuration file" filename=XXXX

Metrics

prometheus_config_last_reload_successful{...,job="prometheus"} | 1
prometheus_config_last_reload_success_timestamp_seconds{...,job="prometheus"} | 1548166302.7771971
prometheus_rule_group_rules | <no data>

[michael-doubez]

Side information that may be relevant: the config file is passed in parameters with a full path (not local path) and the rules are placed relatively to the config file in <dir to config file>/alerts/<something>.rule.yml

[simonpasquier]

Can you check the configuration output from the /config endpoint?

[michael-doubez]

It is consistent with the config file:

rule_files:
- /XXXX/prometheus/config/alerts/*.rule.yml

[juliusv]

And Docker with per-rule-file volumes isn't involved? Because if you mount a file (instead of a directory) as a volume into a container, changes to those files on the host do not become visible in the container until the container is restarted.

[michael-doubez]

It seems that sometimes it works and sometime it doesn't. 💀
I think it is rather related to some slowness.

Looking into the grafana dashboard, I see bumps in fds_count (x2) and query duration (x500).
But it may be due to the reload.

I close the issue since it is not related to the wildcards.

[simonpasquier]

Thanks for the follow-up @michael-doubez!