Annotate kube objects monitored via prometheus

[tamalsaha]

In our kube dashboard, we would like to know whether metrics for a particular Kube object is collected by Prometheus or not. We think that one easy way to do that will be the operator to apply "monitored-by:" annotation to Kube objects. We want to annotate any workload object type that matches from service labels.

What do you think about this idea? Do you think this can be done from Prometheus operator?

cc: @aerokite

[mxinden]

The Prometheus Operator does not know of all Kube objects that are monitored by a Prometheus. They might not all be specified via the ServiceMonitor ThirdPartyRessource but via the Prometheus Config itself.

Maybe this could live in a separate tool which scrapes the Prometheus Target API and applies the Annotation to the corresponding objects.

[brancz]

The difficulty here is that Prometheus would need to know about all Pods and all monitored Pods and constantly add the annotation for the union of those and remove them for the inverted union, as when Prometheus doesn't monitor them anymore, then the annotation would need to be removed.

The Prometheus Operator doesn't know anything about the targets that the actual Prometheus instance scrapes. So I agree with @mxinden that this should probably live in a separate tool, I'm happy to give it a home under contrib/ if you want, but I'm unsure whether it's suitable to be part of the main Operator code base and binary.

[tamalsaha]

Thanks folks! I am going to look into the Targets API.

[brancz]

For now I think we can close this as it won't go into the main Operator code base, feel free to open a PR if you would like to add such a tool to contrib/. And we're still around if you have any further questions 🙂