K8s SD: Namespace discovery should use namespace from serviceaccount folder

[ramato-procon]

Proposal

Currently the ca-cert and token for Kubernetes API is read from mounted container serviceaccount in folder: /var/run/secrets/kubernetes.io/serviceaccount
Unfortunately the namespace /var/run/secrets/kubernetes.io/serviceaccount/namespace is not used. Therefore all queries run in cluster context, not in the namepsace. If the service account is restricted to the namespace an error will occur.
Setting the namespace in the prometheus.yml for each namespace is not handy.

[brancz]

While I agree with this from a security perspective, this would be a breaking change to the Kubernetes service discovery, that would break the majority of configurations. The majority of people want to start by monitoring their entire cluster, and the current way is the expected behavior by most users.

In your case, I agree that this takes some effort, but it's just a matter of configuring the namespace in one place in your Prometheus Kubernetes SD configuration. This shouldn't be much of an issue, as you must be specifying the namespace you're deploying Prometheus to in some way as well, so you should be able to template the configuration in the same way.

[ramato-procon]

I don't know how most users monitor their application in Kubernetes cluster, but we have a large cluster with hundreds of projects. Each project is responsible to collect their own metrics with an own prometheus pod (it can be aggregated to a central prometheus server).
And of course each project has access to their namespace only.
I assume this is the basic approach in large clusters.

Currently I'm replacing the namespace in my docker run script:

KUBERNETES_NAMESPACE=$( cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
sed -i s/KUBERNETES_NAMESPACE/${KUBERNETES_NAMESPACE}/g /var/prometheus/config/prometheus.yml

prometheus.yml:

kubernetes_sd_configs:
     - role: endpoints
       namespaces:
         names:
           - KUBERNETES_NAMESPACE

I can live with that, but in my oppinion it would be better to have on option like "use_project_namespace:true" which reads the namespace from the serviceaccount folder.

[brian-brazil]

I agree with @brancz here. This is something to be handled via configuration management.

[brancz]

As a side note, the Prometheus Operator does exactly what you're asking for here by default.

[ramato-procon]

@brancz Thank, you. I didn't know that. Looks promising.