rule_files does not seem to detect file changes

[avonwyss]

Proposal

Use case. Why is this important?
Changes to rule files specified in rule_files should ideally take effect automatically, without the need to trigger a configuration reload manually. This would allow external tools/scripts to generate of modify these files without requiring direct interaction with Prometheus.

I would expect it to behave the same as scrape_configs.*.file_sd_configs.files which detects file changes as well as new files when using a wildcard.

[brian-brazil]

You need to reload Prometheus's configuration for changes to rules to take effect, just as for prometheus.yml changes.

[avonwyss]

I'm aware of that, which is why I have made a proposal to add file change detection.

[brian-brazil]

The problem with that is that it wouldn't allow for atomic changes of multiple files, which is why it's currently tied into the general reload logic.

[avonwyss]

I don't mind the general reload. Would it be a problem if changes to any of the change monitored files/patterns caused a general reload? That would keep it atomic...

[brian-brazil]

Yes, that'd also prevent users from doing a multi-file atomic change as the first file changed would trigger a reload before the rest were done.

[avonwyss]

I understand what you are saying, but I'm not convinced that this is actually a problem. If I remember correctly the docs say that, whenever the config is not valid on a reload, it will not change the running configuration. Therefore an inconsistent multi-file change should not break anything. Also, usually file monitors triggers use some debouncing/quiet wait time which also mitigates this kind of issues.

[brian-brazil]

It is a problem, say I want to replace file a.rules with b.rules. That's not possible atomically with your suggestion.

Also, usually file monitors triggers use some debouncing/quiet wait time which also mitigates this kind of issues.

That makes it into a race condition, which isn't much better. In addition you'd be breaking all the users who depend on the current documented behaviour.

[krasi-georgiev]

@avonwyss if you use tools that generate these, can't it send a reload request to Prom as well?

[avonwyss]

@krasi-georgiev Of course I can create some mechanism to trigger the reload externally, but I try to keep the tools and systems as decoupled as possible. Since file monitoring actually exists in other places of the configuration it did not seem to be such a far-fetched idea.

[krasi-georgiev]

try to keep the tools and systems as decoupled

To me it seems logical that the same tool that generates the config should be responsible for actually enabling the new config.

And @brian-brazil's argument about the race condition is a rather big blocker against changing the current behaviour.

[roidelapluie]

PSA: We need that atomic change thing. To us changing that would cause troubles.

[avonwyss]

Since this proposal has apparently significant problem potential I'm closing it.
I have now added the --web.enable-lifecycle flag and trigger the reload externally via web request.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.