Kubernetes discovery

[jimmidyson]

Bit of a brain dump while I think about it.... Kubernetes components expose Prometheus metrics for the master, nodes, etcd, etc. Autodiscovery of nodes would mean all nodes & running containers are automatically discovered. Nodes expose Prometheus metrics for all Docker containers that Kubernetes manages. As every resource can be labelled in Kubernetes these labels could be reused as metric labels in Prometheus.

Following on from this, Kubernetes annotations could be used on Kubernetes services to monitor service endpoints.

If you're interested in this I'll try to put in a PR for review as soon as I have something basic working to get feedback?

[grobie]

This sounds very interesting to me! I'd be interested to hear how you envision the integration and necessary changes/additions in Prometheus. Happy to take a look at a PR, will read up on interfaces Kubernetes provides so far.

[fabxc]

We are absolutely interested in this!

[jimmidyson]

To begin with I'm thinking of just adding kubernetes node discovery. This will be enough to collect container (cgroup) metrics for all containers in the cluster as well as cluster level components. Each resource on kubernetes can be labeled (key/value pairs). Each node can be labeled independently & reusing these node labels as Prometheus target labels makes sense so I'm thinking of adding each node as a separate target group. This will mean each target group will only ever contain 1 target. Is there any performance impact of that compared to using a target group with multiple targets? I don't think targets in the same target group can have different labels can they? Well without relabelling which isn't part of discovery.

[matthiasr]

This sounds great! For pods, it's already kind of possible using DNS and SRV records but from my limited playing with it this only works if the pod is covered by a service.

Once all the bits for monitoring cluster, nodes and pods I think it would be great to have a blessed example configuration/replication controller/prometheus service definition that new kubernetes/Prometheus users can drop in to get started quickly.

[jimmidyson]

@matthiasr while using DNS is possible for pods that are service endpoints, you won't be able to use the kubernetes labels as metric labels. Once I've added in node metrics, I hope to expand this initial node only functionality to cover kubernetes service endpoints via the kubernetes api. This will enable reusing kubernetes labels in metrics labels. Haven't thought about that too much yet but any ideas most welcome.

Your idea for a blessed config sounds great. Once I have something basically working I'll let you know & perhaps you could help me work out a generic config that will work out of the box for most people. I know kubernetes better than Prometheus so I have a lot to learn here.

[jimmidyson]

@matthiasr BTW if you wanted to try out a drop in config for kubernetes nodes that uses file config targets & a separate watcher utility to update the targets as changes happen we provide one in fabric8. The kubernetes manifest is available at http://repo1.maven.org/maven2/io/fabric8/jube/images/fabric8/prometheus/2.2.9.1/prometheus-2.2.9.1-kubernetes.json It's this work that I'm trying to convert over to built in Prometheus discovery.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.