kubernetes-service-endpoints is down

[hkailantzis]

kubernetes-service-endpoints is down. Connection refused. Using Exposing Prometheus As A Service, e.g. NodePort with the following annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /
prometheus.io/port: '8080'

Any hint of what might be wrong ? Also pods are not showing in Prometheus. Using 2.2.1 and 2.3.2 version of the Prometheus image shows same results.

Thanks for repo.

[bibinwilson]

@hkailantzis the cluster roles are probably not applied properly. The Prometheus deployment needs read access to all cluster components to get the metrics.

[hkailantzis]

thanks for your reply. Well, I used the cluserRole.yaml file from this repo, but not sure what might be the problem. yaml file is applied.[UPDATE]. I removed the blocks with "annotation_prometheus" from the config map yaml file, as shown in the official example page:
https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml,
and now I have some pods showing up (7/31 up) and some service endpoints (8/27 up). Most errors are "404 page not found" and only two "connection refused".

[devopscube]

Closing this as the configs are working..

[phong-ho]

@hkailantzis the cluster roles are probably not applied properly. The Prometheus deployment needs read access to all cluster components to get the metrics.

Hi
How can we check it ?

[bibinwilson]

@phong-ho is the issue resolved?

[phong-ho]

@bibinwilson not yet, Can you give me a hint to troubleshoot the configuration ?
annotation_prometheus is used in Prometheus and Alertmanager which they cause the problem of Connection refused.
Have you checked it on your system yet ?

[jbmcfarlin31]

Hi, I just cloned this repo and deployed it into my kubernetes repo and I am also seeing the connect: connection refused error on my kubernetes-service-endpoints target.

[bibinwilson]

Hi, I just cloned this repo and deployed it into my kubernetes repo and I am also seeing the connect: connection refused error on my kubernetes-service-endpoints target.

@jbmcfarlin31 please refer #3 (comment)

[jbmcfarlin31]

@bibinwilson I actually got this to work by changing the image of prometheus to their latest stable one. I went to version 2.11.1 and the problem went away. Not sure if that helps at all.