Extend JSON file-based service discovery to http

[quolix]

Prometheus supports reading service lists from a JSON file.
https://prometheus.io/docs/operating/configuration/#file_sd_config

It would be great to extend this mechanism to fetch the JSON file from an http endpoint. Ideally it would support a list of http endpoints which it tries round-robin, and http endponts in which the host is a SRV record from which it gets the list of host:ports.

Background:
We are working on a distributed file system (Quobyte) that has its own service discovery mechanism. All Quobyte services and clients are able to export metrics, but as the system structure is dynamic, it is a hassle to list targets statically.

[brian-brazil]

This has already been discussed at length and rejected. Wget in a cronjob will do what you want.

[quolix]

Thanks for the empathy! I couldn't find the discussion on prometheus-developers, but I'll just assume that there were good technical reasons beyond "wget and crontab can do it".

[juliusv]

@quolix Sorry, I agree this could have been answered with more empathy.

On a factual side, this request has come up sometimes, like in the recent issue #1675. There might have been a longer discussion about this, though I'm not aware. But in general, we've usually decided to have one generic way of doing things vs. two. That is, either a file-based interface or a network-based one. That general topic has come up for other things, like for example the Node Exporter having a textfile collector for generic node metrics vs. having it execute binaries, etc. I can see the desire to have only one interface per purpose, but I also am wondering whether in this cloudy dynamic day and age it wouldn't be better to also support a generic network-based way of discovering targets. Then again, we already support a multitude of network-based SD mechanisms, including ones that are not tied to a specific cloud provider or container engine (like DNS SRV, Consul, or Zookeeper). I'm wondering if putting your targets into one of those would be an option?

[quolix]

Thanks for the background, @juliusv!

As an alternative I already contemplated adding a DNS port to our service registry for this purpose. Not very elegant as we were able to keep the number of ports of our system down to two, one being http. I couldn't check yet if zk or consul are http-based APIs but if that's the case that would be a good solution.

I think a http interface for discovery would make sense for Prometheus, as http has become the language of service integration and you guys do outcalls anyway. Generally I think a container environment should always be considered which makes the crontab approach unwieldy. But you know that I am a big proponent of architectural principles so I fully understand your argument.

[juliusv]

Yeah, Consul is HTTP-based (although it also has a DNS interface), so that should hopefully work fine.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.