Failed to fetch Prometheus configuration: i/o timeout

[heron-bitsensing]

Hi, I'm totally newbie of kubernetes.

I installed kubeflow on eks, and prometheus is already included in the kubeflow.

And I installed kiali using this page.

https://istio.io/latest/docs/ops/integrations/kiali/#option-1-quick-start

And I did port-forward kiali service to 127.0.0.1:20001.

But when I push login button, nothing shows up.

And this is pod/kiali log.

kubectl logs -f pod/kiali-d4fdb9cdb-9lmx9 -n istio-system
2021-04-24T06:38:10Z INF Kiali: Version: v1.29.2, Commit: 953d767791a95dfad533f87cc032d4834dc888e7

2021-04-24T06:38:10Z INF Using authentication strategy [anonymous]
2021-04-24T06:38:10Z WRN Kiali auth strategy is configured for anonymous access - users will not be authenticated.
2021-04-24T06:38:10Z INF Kiali: Console version: 1.29.1
2021-04-24T06:38:10Z INF Updating base URL in index.html with [/kiali]
2021-04-24T06:38:10Z INF Generating env.js from config
2021-04-24T06:38:10Z INF Server endpoint will start at [:20001/kiali]
2021-04-24T06:38:10Z INF Server endpoint will serve static content from [/opt/kiali/console]
2021-04-24T06:38:10Z INF Starting Metrics Server on [:9090]
2021-04-24T06:38:56Z INF [Prom Cache] Enabled
2021-04-24T06:39:26Z ERR Failed to fetch Prometheus configuration: Get "http://prometheus.istio-system:9090/api/v1/status/config": dial tcp 10.100.131.32:9090: i/o timeout
2021-04-24T06:39:56Z ERR Failed to fetch Prometheus flags: Get "http://prometheus.istio-system:9090/api/v1/status/flags": dial tcp 10.100.131.32:9090: i/o timeout
2021-04-24T06:39:56Z INF Kiali Cache is active for namespaces [.*]
2021-04-24T06:45:37Z ERR Failed to fetch Prometheus configuration: Get "http://prometheus.istio-system:9090/api/v1/status/config": dial tcp 10.100.131.32:9090: i/o timeout
2021-04-24T06:46:07Z ERR Failed to fetch Prometheus flags: Get "http://prometheus.istio-system:9090/api/v1/status/flags": dial tcp 10.100.131.32:9090: i/o timeout

kubectl get pods -n istio-system
NAME                                      READY   STATUS    RESTARTS   AGE
authservice-0                             1/1     Running   0          11d
cluster-local-gateway-84bb595449-t2rf8    1/1     Running   0          11d
istio-citadel-7f66ddfcfb-4jktz            1/1     Running   0          11d
istio-galley-7976dd55cd-2djkz             1/1     Running   0          11d
istio-ingressgateway-c79f9f6f-56z9t       1/1     Running   0          11d
istio-nodeagent-5n52p                     1/1     Running   0          2d21h
istio-nodeagent-9jzrh                     1/1     Running   0          11d
istio-pilot-7bd96d69d9-c87lv              2/2     Running   0          11d
istio-policy-66b5d9887c-hg4tn             2/2     Running   0          11d
istio-sidecar-injector-56b6997f7d-zwtsg   1/1     Running   0          11d
istio-telemetry-856f7bcff4-r2plt          2/2     Running   0          11d
kiali-d4fdb9cdb-9lmx9                     1/1     Running   0          26m
prometheus-7bfc895484-wmr82               1/1     Running   0          13m

I thought Prometheus works fine.

kubectl port-forward pod/prometheus-7bfc895484-wmr82 -n istio-system 9090

But I tried to call through dns name in kubeflow jupyter lab instance, kiali success but prometheus failed.

jovyan@work-0:~$ curl http://kiali.istio-system:20001
<a href="/kiali/">Found</a>.

jovyan@work-0:~$ curl http://prometheus.istio-system:9090
upstream connect error or disconnect/reset before headers. reset reason: connection failurejovyan@work-0:~$ 

Versions used
Kiali: 1.29.1
Istio: 1.3.1
Kubernetes flavour and version: v1.18.9-eks-d1db3c

[jmazzitelli]

This is not a bug - it is just a Q&A Discussion - so moving it over there.

[carlosbmg]

Did someone find a solution for this? I'm facing the same issue and i haven't been able to make it work

[israel-hdez]

My only idea is that there may be a Network Policy blocking traffic to prometheus.

But I've never used kubeflow. So, I cannot be sure.

[carlosbmg]

Thanks a lot. I found out yesterday that a colleague made a networkpolicy as you said! It's fixed now :). Cheers mate!