Providing an API to configure configuration

[fatih]

Hi,

I was recently looking how to modify/add/delete rules to the configuration file via an HTTP endpoint. I've searched the issues a little bit and found these two comments:

#1508 (comment)
#1505 (comment)

I just wanted to ask if this still applies and if we could rethink the decision. We already have the /-/reload endpoint to reload the configuration. I think we should have some sort of REST API that allows us to modify the configuration.

I'm ok if you decided against it, but I think it would be a valuable addition and very useful for companies that have a very dynamic set of rules, that are deleted/added very often. I would be very happy if we can move this forward. Thanks.

[beorn7]

I'm pretty sure there is a very fundamental and very strong consensus among the core developers to keep the config statically in a file. Any interactive tool to change it should be done outside of the Prometheus server itself (sending a SIGHUP or hit the reload endpoint once it is done changing the config).

@brian-brazil @fabxc @juliusv Feel free to re-open if my assumption above is wrong.

[juliusv]

Yes, I think so too. More reasoning: when you need to apply the config via HTTP, a Prometheus server is not ready for operation after startup (or crash restart) until some process applies the config. Persisting the applied config would also be hairy, because now the authoritative version of the config can come from multiple places (local config management or HTTP). All in all, we want configuration to flow along one clear path, from config management into a file, and from there into Prometheus.

[fatih]

I respect your decisions to keep it simple and let the consumers of prometheus do their own implementation on top of it. Thanks for the feedback, there is always hope for the future :)

[juliusv]

What IMO is an option (but not sure who'd agree): instead of loading the config from a local file, allow watching a config file in a store like Etcd and automatically reload the configuration when it changes.

[grobie]

This is already possible using confd, not sure this should be re-implemented in Prometheus.

[juliusv]

@grobie Good point, and it supports a bunch of stores already. Even if it comes at the cost of running confd alongside each Prometheus server. Btw., can it send a SIGHUP or do you need an extra thing for that?

[juliusv]

Ah, it can. Cool.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.