Expose full support for file globs for rules

[dacameron]

What did you do?
Tried putting a file glob for rule_files:

/etc/prometheus/rules/*/*.rule

What did you expect to see?
Rules loaded like:

/etc/prometheus/rules/service_A/some.rule
/etc/prometheus/rules/service_B/another.rule

What did you see instead? Under which circumstances?

Invalid rule file path "/etc/prometheus/rules/*/*.rule"

and prometheus doesn't load.

To work around this I padded the service names so they are always 16 chars and added a rule:
/etc/prometheus/rules/????????????????/*.rule

a total hack but it works for now.

I am happy to send a pull request to fix the regex in config.go. But I am not sure if this is by design? or can the rule file path regex in config.go be improved?

The main use case is to have rules for different services in different directories. When deploying those services we can blow away the old rules by blowing away the directory with the rules, adding the new rules and then reloading prometheus.

[brancz]

I think this is a valid use case. I would be interested as well why this decision was made. But it was before my time, so I would ask @beorn7 @brian-brazil @fabxc @juliusv as to why this decision was made. What would be the problem with a standard file glob?

[brian-brazil]

Would that work with inotify?

[brancz]

I wasn't aware that rule files are watched and reloaded. I think in that case a watch per subdirectory has to be instantiated, and gets more complicated when you want to dynamically add new directories to watch when they're added.

[brancz]

Also the documentation makes it sound like file globs work:

# Rule files specifies a list of globs. Rules and alerts are read from
# all matching files.
rule_files:
  [ - <filepath_glob> ... ]

[dacameron]

I can see how adding watched rule support would be useful, and I think doing this with full glob support would be a world of pain.

But you need to be more limiting in your allowed file globs. Someone could be using ? between separators now which means inotify won't work as is. So in any case the regex needs to be patched to either be more permissive or more restrictive.

I'll stop doing my ? between separators hack and I will just prefix rules in one directory when I deploy them. This achieves the same purpose and is future proof for whatever you decide to do.

[jwhitlark]

Can we please have updated documentation on what file path support is available? I'd do a PR myself, but as it isn't actually a glob, I don't know what it actually supports.

[brian-brazil]

A glob is supported in the filename, looks like both rule_files and file_sd_configs are missing that detail.

[brian-brazil]

I don't think it would make sense to have this in one place and not the other, as that'd just cause confusion. Let's stick with just a glob in the file name.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.