how to access prometheus from custom service

[ryandawsonuk]

I am trying to query the prometheus-operated service in the openshift-monitoring namespace. When I port-forward and call from postman I am able to get data:

But when I make the same call from inside the cluster I get a connection refused error:

Get http://prometheus-operated.openshift-monitoring:9090/api/v1/query_range?end=1589793390&query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%22seldon%22%2Cpod%3D~%22.%2Airis-container-.%2A%22%7D%5B1m%5D%29%29&query_type=query_range&start=1589792490&step=30: dial tcp 10.130.0.11:9090: connect: connection refused

Here I am calling from a namespace in an istio mesh but I can't see any NetworkPolicy blocking this access and from what I've read it should be accessible by default.

I've exported a curl from postman and that also works locally and gives connection refused when run from inside the Pod (I've installed curl and done an exec into the Pod) to prometheus-operated.openshift-monitoring. So it's an access problem and not a problem with the call being run. This Pod is able to make calls to other namespaces, seemingly just not this particular service.

I've also deployed a curler Pod to the openshift-monitoring namespace and the curl from inside that gives connection refused too. From there I see

curl -X GET \
>   'http://prometheus-operated.openshift-monitoring:9090/api/v1/query_range?end=1589557920&query=sum%28rate%28container_memory_usage_bytes%7Bnamespace%3D%22seldon%22%2Cpod%3D~%22.%2Airis-container-.%2A%22%7D%5B1m%5D%29%29&query_type=query_range&start=1589557020&step=30' \
>   -H 'Accept: */*' \
>   -H 'Content-Type: application/x-www-form-urlencoded' \
>   -H 'Host: localhost:9090' \
>   -d 'query=sum(rate(container_cpu_usage_seconds_total%7Bnamespace%3D%22default%22%2Cpod%3D~%22income-default-.*%22%7D%5B1m%5D))&query_type=query_range' -v
* Hostname was NOT found in DNS cache
*   Trying 10.130.0.11...
* connect to 10.130.0.11 port 9090 failed: Connection refused
*   Trying 10.131.0.2...
* connect to 10.131.0.2 port 9090 failed: Connection refused
* Failed to connect to prometheus-operated.openshift-monitoring port 9090: Connection refused
* Closing connection 0
curl: (7) Failed to connect to prometheus-operated.openshift-monitoring port 9090: Connection refused

Any idea what could be blocking this?

[lilic]

What are you trying to achieve?

from a namespace in an istio mesh

What ns is this?

[ryandawsonuk]

Sorry I've now updated the issue a bit more since I first posted it, adding details on the curl. I am trying to run an app with integrated visualisations based on the prometheus data. But the problem I'm hitting seems to be with any call to this service from within the cluster as I'm also unable to run the curl from within openshift-monitoring.

That 10.130.0.11 is also what a ping resolves to, though in the failed curls it then moves on to 10.131.0.2.

I notice there's a way to hook a grafana into this prometheus. In my case the tool isn't grafana but I'm wondering if I should be able to connect in a similar way. But I'm hoping I don't have to deal with auth as I'm just calling from the backend inside the cluster. I was expecting to be able to call the service in the same way in the cluster as when port-forwarding. Quite confused about what I could be missing.

[ryandawsonuk]

Oh just noticed the prometheus-operated services doesn't have a ClusterIP. I guess that's why.

I guess I should be using the prometheus-k8s service instead. That one requires auth but I can curl it from in the cluster when I add a token header. So I should be able to go that route and add the token. Will try that and re-open if I hit problems with that.