kubernetes sd config support for controllermanager and scheduler

[andyxning]

Currently, kubernetes sd config can be used to fetch the knode ip. With this info and relabel, we can query both kubelet and kubeproxy metrics.

It seems that there is no support for querying controllermanager and scheduler instance address.
BTW, how to scrape apiserver metrics.

Any thoughts on this?

[andyxning]

/ping @fabxc @keegancsmith @brancz

[fabxc]

Unlike the other things you can discover, controller manager and scheduler are no first class objects you can retrieve from the Kubernetes API.

You can easily add a headless services with endpoints objects though, that point to these components. Than you can simply discover them as regular services.

How exactly that looks like depends on your environment a bit. See here for an example using minikube: https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus/manifests/k8s/minikube

[andyxning]

@fabxc Sounds reasonable.

K8s supports an endpoint called api/v1/componentstatuses with this we can find the status about scheduler and controllermanager. However, we can not get the ip address from this endpoint for now. In order to support this, we need to support list and watch verbs for scheduler and controllermanager just like what we have done for other first class resources.

BTW, could you explain how to setup an apiserver metric monitor along with knode. I mean whether there is a way to set up a single kubernetes_sd_configs to both query apiserver and knode. It seems currently kubernetes_sd_configs only supports node, pod, service and endpoint. Examples for adding both apiserver and knode with one kubernetes_sd_configs in this coreos blog about prometheus and kubernetes seems can not work.

[brancz]

Treating Kubernetes components in a special way is not required at all or good. For the apiserver there is already a Service being maintained by default in the default namespaces. So you can just use the endpoints discovery to discover your apiservers through that (because every Service in Kubernetes maintains an Endpoints object).

For kubelets you have different options:

1. use node discovery with insecure port
2. use node discovery with secure port (however for this you will have to provision the kubelets ssl client cert with your Prometheus instance)
3. use node discovery and proxy the requests through the apiserver so you don't have to provision the client cert to your Prometheus instances and manage the permissions via RBAC.
4. use any of the above method with endpoints discovery by syncing the kubelet IP adresses into an Endpoints object. This is a feature provided by the Prometheus Operator.

[andyxning]

@brancz @fabxc Prometheus Operator can only works with kubernetes clusters running on a k8s cluster, Right?

[brancz]

If you are talking about self-hosted Kubernetes clusters, then no the Prometheus Operator will work on any cluster, however, if you use the scripts in the contrib/kube-prometheus directory, those are optimized for self-hosted Kubernetes clusters, but work just as well with kops (I know that because a member of the community contributed a kops guide to it).