[Multicluster]the istioctl debug tool in Primary-Remote model behavior improperly

[morvencao]

Bug description

Install istio in Primary-Remote mode, and follow the verify steps to deploy helloworld and sleep applications, then I try to use the istioctl proxy-status and proxy-config commands to debug the Envoy and Istiod configurations, I found that:

1. In the primary cluster, the istioctl proxy-status also shows sidecars from remote cluster:

# ./bin/istioctl --context="${CTX_CLUSTER1}" proxy-status
NAME                                                   CDS        LDS        EDS        RDS          ISTIOD                      VERSION
helloworld-v1-776f57d5f6-8lztd.sample                  SYNCED     SYNCED     SYNCED     SYNCED       istiod-5d46c5c495-n8lsv     1.8.1
helloworld-v2-54df5f84b-mc8f4.sample                   SYNCED     SYNCED     SYNCED     SYNCED       istiod-5d46c5c495-n8lsv     1.8.1
istio-eastwestgateway-dcdcbcb5d-g7bkb.istio-system     SYNCED     SYNCED     SYNCED     NOT SENT     istiod-5d46c5c495-n8lsv     1.8.1
istio-ingressgateway-5f89d44d4f-bxnrh.istio-system     SYNCED     SYNCED     SYNCED     NOT SENT     istiod-5d46c5c495-n8lsv     1.8.1
istio-ingressgateway-d99655c7b-rbxvv.istio-system      SYNCED     SYNCED     SYNCED     NOT SENT     istiod-5d46c5c495-n8lsv     1.8.1
sleep-854565cb79-dgmgc.sample                          SYNCED     SYNCED     SYNCED     SYNCED       istiod-5d46c5c495-n8lsv     1.8.1
sleep-854565cb79-h94bv.sample                          SYNCED     SYNCED     SYNCED     SYNCED       istiod-5d46c5c495-n8lsv     1.8.1

2. But when I use proxy-config to check the XDS configuration on primary cluster from the sidecar in the remote cluster, I get pod not found error.

# ./bin/istioctl --context="${CTX_CLUSTER1}" proxy-config cluster sleep-854565cb79-h94bv.sample
Error: failed to execute command on sleep-854565cb79-h94bv.sample sidecar: failed retrieving pod: pods "sleep-854565cb79-h94bv" not found

3. If I use the istioctl proxy-status to check all the sidecars on the remote cluster, there will no items:

# ./bin/istioctl --context="${CTX_CLUSTER2}" proxy-status
NAME     CDS     LDS     EDS     RDS     ISTIOD     VERSION

This is not really a bug, but an UX issue, the istioctl current can only retrieve information for sidecars from current cluster(primary cluster), we may need some get sidecars from remote cluster through apiserver, because the apiserver is available from primary cluster.

[ X ] User Experience
[ X ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane

[shamsher31]

I tried it today and I can see that it only shows pod from primary cluster.

$ istioctl --context="${CTX_CLUSTER1}" proxy-status
NAME                                                    CDS        LDS        EDS        RDS          ISTIOD                      VERSION
helloworld-v1-776f57d5f6-ksrn4.sample                   SYNCED     SYNCED     SYNCED     SYNCED       istiod-679748f6d8-rlvpf     1.8.1
istio-eastwestgateway-854d659967-dlhhn.istio-system     SYNCED     SYNCED     SYNCED     NOT SENT     istiod-679748f6d8-rlvpf     1.8.1
istio-ingressgateway-7f6859b88-5snbt.istio-system       SYNCED     SYNCED     SYNCED     NOT SENT     istiod-679748f6d8-rlvpf     1.8.1
sleep-557747455f-fgvr7.sample                           SYNCED     SYNCED     SYNCED     SYNCED       istiod-679748f6d8-rlvpf     1.8.1

Also it's able to read proxy config from sleep pod.

$ istioctl --context="${CTX_CLUSTER1}" proxy-config cluster sleep-557747455f-fgvr7.sample
SERVICE FQDN                                             PORT      SUBSET     DIRECTION     TYPE             DESTINATION RULE
                                                         80        -          inbound       STATIC           
BlackHoleCluster                                         -         -          -             STATIC           
InboundPassthroughClusterIpv4                            -         -          -             ORIGINAL_DST     
PassthroughCluster                                       -         -          -             ORIGINAL_DST     
agent                                                    -         -          -             STATIC           
helloworld.sample.svc.cluster.local                      5000      -          outbound      EDS              
istio-eastwestgateway.istio-system.svc.cluster.local     15012     -          outbound      EDS                   
istio-ingressgateway.istio-system.svc.cluster.local      80        -          outbound      EDS

But as you mentioned it's not able to read proxy-status from remote cluster.

$ istioctl --context="${CTX_CLUSTER2}" proxy-status
NAME     CDS     LDS     EDS     RDS     ISTIOD     VERSION

[jeremybramwell]

I have been working on a POC based on the "Primary-Remote on the same network" example using istio 1.9.0 on two GKE clusters. I noticed the same awkward behavior described in this issue. I also found that trying to diff the config of an envoy proxy in the remote cluster against the istiod config doesn't work.

As described above calling istioctl proxy-status on the primary cluster returns all the proxies in the mesh:

$ istioctl proxy-status --context $PRIMARY_CLUSTER_CTX
NAME                                                   CDS        LDS        EDS        RDS          ISTIOD                      VERSION
api-worker-5b5d9f6d4f-crvzv.tenant-a                   SYNCED     SYNCED     SYNCED     SYNCED       istiod-79dfb4f5bd-bdlm2     1.9.0
api-worker-66ffd6599d-7jndf.tenant-b                   SYNCED     SYNCED     SYNCED     SYNCED       istiod-79dfb4f5bd-bdlm2     1.9.0
api-worker-685d4c6954-q5qxb.global                     SYNCED     SYNCED     SYNCED     SYNCED       istiod-79dfb4f5bd-bdlm2     1.9.0
istio-eastwestgateway-8fbfc8998-vsdk9.istio-system     SYNCED     SYNCED     SYNCED     NOT SENT     istiod-79dfb4f5bd-bdlm2     1.9.0
istio-ingressgateway-659fbbd8bd-96w6h.istio-system     SYNCED     SYNCED     SYNCED     SYNCED       istiod-79dfb4f5bd-bdlm2     1.9.0 

the first two pods, api-worker-....tenant-a and api-worker-....tenant-b, are in the remote cluster. If I try to diff the remote pod's config with istiod it fails with different errors depending on which cluster context I use.

Primary cluster ctx:

$ istioctl proxy-status api-worker-5b5d9f6d4f-crvzv.tenant-a --context ${PRIMARY_CLUSTER_CTX}
Error: failed retrieving pod: pods "api-worker-5b5d9f6d4f-crvzv" not found

Remote cluster ctx:

$ istioctl proxy-status api-worker-5b5d9f6d4f-crvzv.tenant-a --context ${REMOTE_CLUSTER_CTX}
Error: unable to find config dump in Istiod responses

It makes sense that this would be an issue since the remote pod and istiod are in different clusters. I just wanted to point out this additional issue, since it makes the diffing tool unusable in a situation where the potential for syncing issues is pretty high.

Let me know if this would be better entered as a separate issue.

[stevenctl]

not stale

[istio-policy-bot]

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2021-06-28. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.

[hanxiaop]

not stale