Could not fetch Istio objects list: unable to proxy Istiod pods. Make sure your Kubernetes API server has access to the Istio control plane through 8080 port

[pk185235]

Does not load graphs with the connections.

[lucasponce]

Please, these bugs can't be reported without a reproducer or more details about the environment and the failing version.

We'd like and need to learn more about why is failing, it can be a bad combination of Istio + Kiali version, an environment problem (not connectivity between Kiali and Istiod pod), or a bug.

Kiali is an open-source project with an active community but we need more info to progress, the "issues" require more data.

[jshaughn]

Not enough information to proceed.

[partnerSun]

check your istio‘s pod log,if have content :"uid : unable to do port forwarding: socat not found", you can try this command:
yum install socat -y

[karan-arora-netapp]

check your istio‘s pod log,if have content :"uid : unable to do port forwarding: socat not found", you can try this command:
yum install socat -y

This fixed the issue i was having . I was trying to follow https://istio.io/latest/docs/setup/getting-started/ and hit this issue on ubuntu and installing socat fixed the error for me

[subin123123]

For my deployment process, refer to the Getting Started documentation, i have tried :yum install socat -y,hava no effect.

[naiame-neltheri]

I'm having also same issue except that one of the service is able to be detected.
I have Grafana, Prometheus, Grafana loki in same namespace. When I access to kiali dashboard on the graph, it shows only prometheus traffic. And in the log of kiali the following logs are reported:

2022-10-10T02:52:54Z WRN GetPodProxyStatus is failing for [namespace: monitoring] [pod: loki-ingester-2]: unable to proxy Istiod pods. Make sure your Kubernetes API server has access to the Istio control plane through 8080 port 
2022-10-10T02:52:54Z WRN GetPodProxyStatus is failing for [namespace: monitoring] [pod: loki-query-frontend-67b6c5f5f5-xjkvj]: unable to proxy Istiod pods. Make sure your Kubernetes API server has access to the Istio control plane through 8080 port
2022-10-10T02:52:54Z WRN GetPodProxyStatus is failing for [namespace: monitoring] [pod: loki-querier-1]: unable to proxy Istiod pods. Make sure your Kubernetes API server has access to the Istio control plane through 8080 port 

[israel-hdez]

@naiame-neltheri Is the monitoring namespace in the list of accessible namespaces for Kiali?

[nicoche]

I had the same issue in my cluster. Turns out that I had some pods in istio-system with status Terminated. Garbage collecting the Terminated pods fixed it for me.

I run latest (1.15.x) istio and latest (v1.57.1) kiali

[israel-hdez]

@nrfox I wonder if pods in Terminated status may not be correctly removed from the cache, unless they fully go away?
It's the only reason I can think of for those warnings.

[naiame-neltheri]

@israel-hdez yes, kiali deployment is associated with service account which has clusterrolebinding with clusterrole that allows list namespaces and pods etc
To verify it:
kubectl auth can-i list namespace --as=system:serviceaccount:istio-system:kiali -n monitoring this outputs:

Warning: resource 'namespaces' is not namespace scoped
yes

PS: Still wondering about the warning. Also does istiod service needs to be forwarding 8080 traffic to 8080 of istiod pod?

[israel-hdez]

@naiame-neltheri Just to double-check, when you say that you have the same issue, do you mean that the "Istio Config" page show the error mentioned in the main comment?

BTW, the logs you posted about the WRN GetPodProxyStatus is failing probably make sense. I see it is for loki-* pods. My guess is that those deployments don't have an Istio sidecar, so I think the warnings are expected. However, I'm not familiar with that part of Kiali. I think @hhovsepy or @nrfox could know better.

[naiame-neltheri]

@israel-hdez yes the config page also reports same error. And when I access to graph page there is some traffic but still seeing WRN GetPodProxyStatus is failing error. And the monitoring namespace is istio-injection enabled so all pods on the namespace already have the sidecar proxy container

[jshaughn]

I'll re-open this issue given the recent activity.

[nrfox]

@naiame-neltheri do you have an app=istiod label on any pods other than istiod in your istiod's namespace? Kiali tries to select the istiod pods based on that label.

Also does istiod service needs to be forwarding 8080 traffic to 8080 of istiod pod?

Kiali attempts to connect to 8080 just to ensure it can actually reach the istiod pod. By default 8080 is the istiod health probe. The actual port that kiali uses when getting the registry info is the istiod monitoring port: 15014.

[nrfox]

Did some searching on the socat issue talked about further up. It seems that the kubelet relies on socat to do port forwarding: kubernetes/kubernetes#26093 and kiali relies on port forwarding to talk to istiod so the nodes running your istiod pods need to have socat installed on them.

[naiame-neltheri]

@nrfox no, I have only one pod labeled as app=istiod under istio-system namespace which is istiod pod

[yueyanglouji]

I had the same issue in my cluster, when I use proxy transport the kiali svc port 20001,
but use kubectl forward is ok.

follow command
istioctl dashboard kiali --address 0.0.0.0

[naiame-neltheri]

I have fixed the issue by reinstalling istio

[jshaughn]

@naiame-neltheri , glad the problem is resolved. If you have any details as to the root cause please add a note. I'll close this issue, thanks.

[jasonhao518]

Hi, I have the same issue. I found the issue is because there are istiod pods with completed status or any other status than running. delete these pods will fix the issue.

[jshaughn]

@nrfox I wonder if pods in Terminated status may not be correctly removed from the cache, unless they fully go away?
It's the only reason I can think of for those warnings.

@nrfox , it sounds like we do have some sort of issue with non-running pods. Any ideas?

[naiame-neltheri]

I am not sure what was the root cause, there was nothing abnormal. And now because I reinstalled, I cannot reproduce the issue