Route Prometheus endopint shows "Application is not available"

[schau87]

Current version
4.9.0-0.okd-2022-02-12-140851

I have followed this guide to create a cluster with 1 worker and 1 master node. So, far so good.

But, when I restart the cluster using the steps like this then Promtheus API endpoint or the Prometheus route shows the error like in the screenshot. However, the Thanos querier is available.

How can I troubleshoot it ?

[jacksgt]

Hello,

which URL are you trying to address? Recent versions of OKD / OpenShift don't expose the Prometheus UI anymore (https://prometheus-k8s-openshift-monitoring.<BASE_DOMAIN>/), only the API (https://prometheus-k8s-openshift-monitoring.<BASE_DOMAIN>/api).

Could you post the output of this command?

oc -n openshift-monitoring get svc,ep,route -o yaml

[easternnl]

Is there a way to re-expose the Prometheus UI again?

[imdmahajankanika]

@jacksgt Is it mentioned somewhere on any documentation that the recent versions of OKD don't expose Prometheus UI anymore?
I am also getting "404 page not found" for prometheus after upgrading my OKD cluster

[easternnl]

I tried that one, but it will result in: ``` {"error":"invalid_request","error_description":"The request is missing a required parameter, includes an invalid parameter value, includes a parameter more than once, or is otherwise malformed.","state":"3fc86d63140072273fd3e959c8e7c409:/"} ``` When login on with oauth / openshift.

…

On Fri, 3 Feb 2023 at 17:34, Jack wrote: Sure, try this: apiVersion: route.openshift.io/v1kind: Routemetadata: name: prometheus-ui namespace: openshift-monitoringspec: path: / to: kind: Service name: prometheus-k8s port: targetPort: web tls: insecureEdgeTerminationPolicy: Redirect termination: reencrypt

[jacksgt]

You are correct, the approach I mentioned before doesn't work (I believe it used to work at some point, but I can't reproduce it now on a OKD 4.11 cluster).

However, you can use this recipe to access Prometheus directly (if you are cluster admin):
https://blog.cubieserver.de/2023/debugging-prometheus-on-openshift-the-hard-way/

It boils down to:

oc -n openshift-monitoring rsh pod/prometheus-0  socat tcp-l:5050,fork,reuseaddr tcp:127.0.0.1:9090 &
oc port-forward -n openshift-monitoring pod/prometheus-k8s-0 5050:5050

I verified this works on a 4.11 cluster - hope this helps!

[Ariaeimehr]

Hello buddy

I am using openshift 4.13.when I run socat tcp-l:5050,fork,reuseaddr tcp:127.0.0.1:9090 & I get the error that socat is not in the $PATH.I checked the pod and there is not any socat package named socat in the pod prometheus-0. how can I handle this?
Thanks

[nate-duke]

Depending on what you're after you can also just make a service account and give it the cluster-monitoring-viewer role and then use that to connect to the thanos-querier endpoint. That's what we do for peeking into the in-cluster prometheus with an external grafana instance.

[imdmahajankanika]

Thanos-querier is using 'provider=openshift', so it is redirecting to login page