Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prometheus cannot find all k8s resouces, such as pod/services etc. #2947

Closed
ghost opened this Issue Jul 14, 2017 · 2 comments

Comments

Projects
None yet
1 participant
@ghost
Copy link

ghost commented Jul 14, 2017

Prometheus reported the error information below after I deployed EFK:
"
time="2017-07-14T06:43:21Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:180: Failed to list *v1.Service: User "system:serviceaccount:monitoring:default" cannot list all services in the cluster" component="kube_client_runtime" source="kubernetes.go:75"
time="2017-07-14T06:43:21Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:245: Failed to list *v1.Node: User "system:serviceaccount:monitoring:default" cannot list all nodes in the cluster" component="kube_client_runtime" source="kubernetes.go:75"
time="2017-07-14T06:43:21Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to list *v1.Endpoints: User "system:serviceaccount:monitoring:default" cannot list all endpoints in the cluster" component="kube_client_runtime" source="kubernetes.go:75"
time="2017-07-14T06:43:21Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:207: Failed to list *v1.Pod: User "system:serviceaccount:monitoring:default" cannot list all pods in the cluster" component="kube_client_runtime" source="kubernetes.go:75"
time="2017-07-14T06:43:21Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:227: Failed to list *v1.Service: User "system:serviceaccount:monitoring:default" cannot list all services in the cluster" component="kube_client_runtime" source="kubernetes.go:75"
time="2017-07-14T06:43:22Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:180: Failed to list *v1.Service: User "system:serviceaccount:monitoring:default" cannot list all services in the cluster" component="kube_client_runtime" source="kubernetes.go:75"
time="2017-07-14T06:43:22Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:181: Failed to list *v1.Pod: User "system:serviceaccount:monitoring:default" cannot list all pods in the cluster" component="kube_client_runtime" source="kubernetes.go:75"
time="2017-07-14T06:43:22Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:207: Failed to list *v1.Pod: User "system:serviceaccount:monitoring:default" cannot list all pods in the cluster" component="kube_client_runtime" source="kubernetes.go:75"
time="2017-07-14T06:43:22Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:227: Failed to list *v1.Service: User "system:serviceaccount:monitoring:default" cannot list all services in the cluster" component="kube_client_runtime" source="kubernetes.go:75"
time="2017-07-14T06:43:22Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to list *v1.Endpoints: User "system:serviceaccount:monitoring:default" cannot list all endpoints in the cluster" component="kube_client_runtime" source="kubernetes.go:75"
"

What shoud I do now? @Conorbro

@matthiasr

This comment has been minimized.

Copy link
Contributor

matthiasr commented Jul 14, 2017

this looks like a permissions / RBAC issue – Kubernetes is denying the Prometheus service account the access it needs. Check the roles and role bindings.

Please post usage questions to the prometheus-users mailing list, not issues.

@matthiasr matthiasr closed this Jul 14, 2017

@lock

This comment has been minimized.

Copy link

lock bot commented Mar 23, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 23, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.