Implement Configuration and Target Reloading via Disk Watches

[matttproud]

http://golang.org/pkg/syscall/#InotifyAddWatch

[mwitkow]

+1
Would be awesome if we didn't need to restart Prometheus to re-read configs and consoles

[brian-brazil]

I'm likely to get to this in the next few weeks.

[discordianfish]

@brian-brazil That'd be awesome. Now restart prometheus automatically by using https://github.com/docker-infra/prometheus-consul and I hope I won't run into issues for now.

[brian-brazil]

I haven't touched this since then unfortunately.

[beorn7]

@fabxc will likely work on this as part of his internship.

[fabxc]

This will probably be on hold until service discovery is done, though - unless it's considered super urgent.

[mwitkow]

How do you guys want to do service discovery? :)

[beorn7]

http://prometheus.io/docs/introduction/roadmap/#support-for-more-types-of-service-discovery is the roadmap entry. We are in the design state right now.

[fabxc]

We'll also want to reload the config on SIGHUP. Not all file systems (natively) support watches.

[matthiasr]

I would even prefer to have disk watches be optional. I'd much prefer explicit SIGHUPs in many cases (e.g. when I can't guarantee that the config file is replaced atomically).

[discordianfish]

👍 on having both
@fabxc: If SD will integrate easily with consul and have a way to use the consul tag as label value, I don't really need the config reload anymore. Otherwise it would be more important to me to get the config reload first.

[levilovelock]

I'm in the same position as @discordianfish - would prefer just having SD first, but config refresh would let me brew my own workaround at least.

[juliusv]

@levilovelock Just work-wise, it makes much more sense to get SD in order first as a base for implementing sane reloadable configs. Expect both soon[tm].

[fabxc]

We'll also want to reload changed rule files.

[beorn7]

We had a discussion that it might be better to reload only upon SIGHUP (which is implemented by now).

Does anybody feel a strong need for a file watch? If yes, I'd be willing to make the point against it.
Otherwise, I'll just close this.

[brian-brazil]

For configuration, a HUP is safest as it lets you do atomic changes.

[dylanmei]

This work is great. I can use console-template or confd to handle config changes and fire the SIGHUP.

[fabxc]

As we don't apply bad config/rules it would work but so far we all agreed
that SIGHUP is the way to go.

On Thu, May 21, 2015 at 6:44 PM Dylan Meissner notifications@github.com
wrote:

This work is great. I can use console-template or confd to handle config
changes and fire the SIGHUP.

—
Reply to this email directly or view it on GitHub
#108 (comment)
.

[discordianfish]

👍 for SIGHUP only. If someone needs disk watches they can use a simple shell fragment like:
while inotifywait -e close_write prometheus.conf; do service reload prometheus; done

[beorn7]

Thanks for your feedback. Closing.

[mingfang]

I like the way nginx works.
nginx -t to test a config.
If the test succeeds then nginx -s reload to reload the config without restart.
This worths perfectly with confd and etcd combination.

[discordianfish]

@mingfang That's pretty much how prometheus works: Use the promtool to check the config, then send prometheus a HUP signal.

[mvmn]

How often can I do reload?
What if I implement a custom server to manage rules, and it will send POST to /-/reload every time I edit and save a rule - can that impair Prometheus performance significantly if reloads will happen often (say, every minute)?

[brian-brazil]

Once a minute should be fine

[mvmn]

Good. Thanks!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.