Question: n:m target probing

[kim]

I have a cluster of services which are not themselves aware of the fact that they form a cluster, yet must be able to communicate with each other over TCP. The service instances are discovered using DNS SRV.

I am wondering if I could utilise prometheus in conjunction with blackbox exporters deployed alongside service instances to monitor inter-service connectivity. For this to work, I'd need to be able to apply relabelling on the set of targets returned by SD, for each target. However, it is my understanding that relabelling is applied pointwise.

An alternative approach would be to perform SD and probing of peers on each service instance separately and expose the results as a regular scrape endpoint to prometheus. This seems to violate a philosophical stance taken by prometheus maintainers, though (cf. eg. prometheus/blackbox_exporter#82).

Hence, I'd like to know if similar use cases exist for others and whether there is another solution apart from what I outlined above.

(apologies if this kind of inquiries are better served by some mailing list; I am not subscribed there)

[brian-brazil]

This is not possible. I'd suggest looking at file-based service discovery.

[kim]

I don't think file-based service discovery helps in any way here. But thanks anyways for your insightful response.

[brian-brazil]

You could generate all the n:m targets and then use file_sd to provide that to prometheus.

[kim]

Thanks. I actually interpolated that from your first answer, yet I can't find this operationally preferable to the second approach I gave. Perhaps I was just looking for a convenient solution, but it's clear that it doesn't fit the prometheus model very well in the general case. So thanks again, and nevermind.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.