Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What is required api rights for prometheus plugin? #39

Closed
feym78 opened this issue Apr 27, 2024 · 6 comments
Closed

What is required api rights for prometheus plugin? #39

feym78 opened this issue Apr 27, 2024 · 6 comments

Comments

@feym78
Copy link

feym78 commented Apr 27, 2024

Hi. I'm running in-cluster installation of headlamp 0.23.1.
When I use cluster-admin or edit cluster role then prometheus chart is working fine for pods, but if I use view role or custom set of privileges, then chart is not showing with error "Error fetching prometheus Info"
I managed to get understanding that plugin is trying to search all namespaces to see if there is prometheus pod, so I added previleges to get pods in all namespaces, error message disappear, but chart still not showing, in browser console I see error "GET generated url with prometheus pod 403 (Forbidden)".

I can't understand which api right is missing, I tried to grant all api resources in "" group, "apps" group and etc with full scope ("get", "list", "watch", "create", "update", "patch", "delete") clusterwide, but still no result.

My idea is that I don't want to give full access to the cluster for team or even edit role clusterwide and give as few as possible api scope for token, but still provide necessary metrics and tools.

@joaquimrocha
Copy link
Contributor

@yolossn Can you reply this one?

@yolossn
Copy link
Contributor

yolossn commented Jul 16, 2024

Hey @feym78 The plugin identifies the prometheus pod using the app.kubernetes.io/name=prometheus label, can you check if your prometheus pod has this label?

kubectl get pods -l app.kubernetes.io/name=prometheus --all-namespace

We are working on making the prometheus url configurable so that the user can configure the prometheus endpoint to fetch the metrics from.

@feym78
Copy link
Author

feym78 commented Jul 16, 2024

@yolossn Hi, thx for reply
Yes, in my installation prometheus pod has this label (your query works, but with -A instead of "--all-namespace").
Thing is that plugin works with admin or edit cluster role, but I can't figure out which api rights necessary with custom set of rights.
I'm trying to setup "read-only" role in k8s or with limited edit rights in cluster. For testing purposes I tried to grant all api resources in "" group, "apps" group and etc with full scope ("get", "list", "watch", "create", "update", "patch", "delete") clusterwide, but still no result.

@feym78
Copy link
Author

feym78 commented Aug 27, 2024

@yolossn Hi! Any thoughts on this?

@yolossn
Copy link
Contributor

yolossn commented Aug 29, 2024

Hey sorry, I missed this message. I think the reason why the read-only setup is not working is that the plugin creates a proxy to the prometheus plugin for fetching the metrics data. Can you share the output of this command with the "read-only" grant.

kubectl auth can-i create pods/proxy
kubectl auth can-i create services/proxy

@feym78
Copy link
Author

feym78 commented Sep 2, 2024

Hey sorry, I missed this message. I think the reason why the read-only setup is not working is that the plugin creates a proxy to the prometheus plugin for fetching the metrics data. Can you share the output of this command with the "read-only" grant.

kubectl auth can-i create pods/proxy kubectl auth can-i create services/proxy

Strange thing - with admin role I get "yes" to this commands, but if I try them even with full scope for cluster role as:

  • apiGroups: [""]
    resources: ["pods/proxy", "services/proxy"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
    I still getting "no".
    BUT prometheus chart starts to work with this scope:
  • apiGroups: [""]
    resources: ["pods/proxy", "services/proxy"]
    verbs: ["get"]
    even more, it still works with this:
  • apiGroups: [""]
    resources: ["pods/proxy"]
    verbs: ["get"]

So in order to get prometheus plugin working you need to grant SA get/list to all namespaces and pods (so plugin could find prometheus pod) and scope above for pods/proxy.
Thanks a lot for your help!

As offtop: now it would be nice to make chart time range customizable, not hardcoded 10 minutes :)

@feym78 feym78 closed this as completed Sep 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants