Discovery for scrape intervals, scheme & paths

[jimmidyson]

While adding in Kubernetes discovery in a separate PR, I would like to be able to discover scrape intervals & paths from metadata in Kubernetes & update targets appropriately.

The job config contains details of the intervals/scheme/path & this is configured statically in the Prometheus config file, a restart is required to pick up changes. The only dynamic changes allowed are in target groups, which I can update labels & addresses per target but nothing more.

[beorn7]

The best person to discuss this issue is probably @fabxc , but he is on vacation.

[jimmidyson]

@beorn7 Thanks - will wait for his return.

[beorn7]

BTW: sending a SIGHUP reloads the config. If you have some mechanism to rewrite the config, you don't need a restart, just a signal. Might be sufficient for a work-around...

[jimmidyson]

@beorn7 Ah good tip thanks.

[fabxc]

Sorry for the delay.

The path is already available via __metrics_path__ and can thus be relabeled, i.e. populated from meta labels coming from a service discovery.
I agree that exposing a __scheme__ would be consistent with that.

Now, scrape interval is another beast. It certainly is possible, as many things. But I'm wondering whether scrape intervals are something that should come form service discovery and in which direction this puts us. If people can configure the scrape interval through their SD, naturally requests for the scrape timeout, auth credentials etc. would follow.
We should avoid ending up with a system that is so rich (bloated) in possibilities to operate that it becomes a challenge to figure out a proper setup.

There is no one true answer but unless there is a strong need I'd rather slow down the "everything-relabelable" movement.

[jimmidyson]

No worries - hope you had a good break.

So where would I relabel __metrics_path? On the target group or target? Sorry I'm not so up on these prometheus concepts, but getting there :)

Agreed adding __scheme__ in the same way makes sense. I can look at a PR for this if I get a moment.

As for scrape intervals, totally understand your reasoning to resist everything-relabelable. Scheme would definitely be enough for us (for now ;)).

[jimmidyson]

Closing as scheme is now discoverable.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.