Kubernetes TokenRequest support

[brancz]

Currently Prometheus uses the ServiceAccount token given to it in order to perform service discovery. The same token is used when retrieving metrics from the Kubernetes apiserver, kubelet, and other cluster components. This, however, has a security concern, as these components receive the token that Prometheus sends, therefore they can impersonate it.

Kubernetes 1.10 introduced the TokenRequest API, which allows entities to request tokens for a specific audience, as well as other optional constraints. This would allow Prometheus to request tokens in order to retrieve metrics from these cluster components without their possibility to impersonate Prometheus, by using it's token to talk to the Kubernetes API.

However, I'm struggling to decide whether this is functionality that should belong in Prometheus itself, or whether it should go into something like the Prometheus Operator, as this may open up the door of authentication mechanisms that may deeply dependent on the environment Prometheus runs in. On the other hand we demonstrate the above security in our example configuration for Kubernetes (or maybe worse as we currently default to telling people to give their Prometheus server full access to the kubelet API).

/cc @fabxc @mxinden @brian-brazil @grobie

[brian-brazil]

From a security standpoint even if we were to fully support this, it'd still be down to the user as to whether they used it correctly or not which is (roughly) where we are today. It sounds to me like the real issue here is the lack of a challenge-response mechanism for authentication.

I'm against adding any features, particularly security features, that only work for one environment. I'd like to stick to standard auth mechanisms, of which we already support three.

On the other hand we demonstrate the above security in our example configuration for Kubernetes (or maybe worse as we currently default to telling people to give their Prometheus server full access to the kubelet API).

I think there's a few things that are suboptimal in our current k8 example, we should look to improve them.

[brancz]

Closing as this should not belong in Prometheus itself.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.