Prometheus cannot discover services, pods, etc. from Kubernetes user namespaces

[sudhi-vm]

The current Kubernetes discovery module assumes the services, pods, etc. are always available on "/api/v1/services", "/api/v1/pods", etc. paths. Kubernetes supports having services, pods, etc. hosted in different namespaces. If prometheus server itself is deployed in a pod running within a namespace, then it has access only to resources in that namespace. Trying to access "/api/v1/pods" or "/api/v1/services" without specifying namespace results in "403 Forbidden" from the API server.

[sudhi-vm]

CC: @fabxc

[grobie]

The Prometheus Kubernetes discovery already supports Kubernetes namespaces. By default all namespaces get scraped, this can be changed by using relabeling and the __meta_kubernetes_service_namespace etc.

"/api/v1/services", "/api/v1/pods", etc. paths. This points to the default namespace in Kubernetes.

This is not true, these endpoints are namespace agnostic.

The right discovery API prefix should be namespaced, for e.g., "/api/v1/namespaces/default/services" or "/api/v1/namespaces/tenant1/services".

No, we want to be able to scrape all namespaces at once.

[sudhi-vm]

@grobie I agree that "/api/v1/services" are namespace agnostic. My comment about This points to the default namespace in Kubernetes. was incorrect.

Consider the use case where the prometheus server is deployed in a user namespace (which is what we are doing) in K8. In this case, the prometheus server has access only to resources in that namespace. With RBAC/ABAC enabled on k8, this would result in a "403 Forbidden" response from k8 API server during discovery.

One solution could be to use "/api/v1/namespaces/{{namespace}}/services" if namespace is provided in the config, otherwise use "/api/v1/services". I updated the issue description & commit accordingly.

[grobie]

If prometheus server itself is deployed in a pod running within a namespace, then it has access only to resources in that namespace.

That also depends on the respective authorization settings I think. We actually deploy our Prometheus server in a custom namespace and gave that special namespace the necessary rights (via a ABAC authorization policy file) to scrape all other namespaces.

I'd like to avoid introducing another configuration option if possible. Could you solve your problem by changing your authorization policy?

[sudhi-vm]

The specific use case is we have our application and prometheus deployed in a custom namespace in Kubernetes. We want the prometheus server to scrape metrics only from our application pods & services running within the custom namespace.

Having authorization policy to allow access to other namespaces does not seem right from security point of view. Another option that may avoid extra config param is to automatically read the namespace from /var/run/secrets/kubernetes.io/serviceaccount/namespace (http://kubernetes.io/docs/user-guide/accessing-the-cluster/#accessing-the-api-from-a-pod) and this can be tied with the in_cluster configuration. What do you think ?

[jimmidyson]

While I'm not keen on it personally, I can see the valid use case that we can't cover without a loose auth policy. If we were to do this then I would prefer to support optional multiple namespaces via config, although that increases the number of watches required to be set up (they are cheap I guess). We can retain backwards compatibility easily here too by keeping existing functionality if no namespaces are specified.

[fabxc]

Sorry, I overlooked that issue. Seems like the Kubernetes SD will simply not work with auth policies setup, which is not really acceptable.

I'm not familiar with the authz behavior. Do we have a way to determine namespaces we have access to so that we could do this automatically behind the scenes?
I'd argue for leaving this issue open and investigate this after the main changes are done.

[nsams]

In OpenShift there is the additional endpoint /oapi/v1/projects which will return namespaces. But that would require a openshift SD...

[grobie]

Implemented in #2642.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.