TestXdsProxyStatus failures

[howardjohn]

https://prow.istio.io/view/gs/istio-prow/pr-logs/pull/istio_istio/37162/integ-pilot-istiodremote_istio/1489422911918837760

Here

istio/pkg/test/framework/components/istio/operator.go

Line 628 in 7227423

os.Setenv("ISTIOCTL_XDS_ADDRESS", istiodAddress.String())

we set the Istiod Address

This is a pod IP
Istiod autoscales up to 2 pods, because the istiodless code doesn't set the same config as in cluster (which has autoscaling disable)
We picked the wrong istiod in that code above, so we hit an istiod with 0 connections and fail

[frankbu]

In the integration test environment, an external control plane's istiod is exposed as a loadbalancer-type service:

istio/tests/integration/iop-externalistiod-primary-integration-test-defaults.yaml

Line 19 in b268554

type: LoadBalancer

The framework should use the service IP if there is one, but the problem is that kind doesn't support LoadBalance out of the box, so we use the pod IP instead:

istio/pkg/test/framework/components/istio/util.go

Line 108 in 7227423

if !s.LoadBalancerSupported {

So, the problem is because the integration framework uses kind without metallb or some other loadbalancer support. We could conceivably setup the cluster using metallb, like we suggest users do for testing with kind, or we could set up a gateway, like we do in the external control plane docs.

I'm inclined to think that since this is just for integration testing, and there's no real benefit to scaling up istiod for these tests, why not just fix this by disabling autoscaling of istiod, that is, I think the temporary workaround for this issue, #37170, is the permanent fix.

@howardjohn WDYT?

[howardjohn]

I think for all of our tests we always deploy metallb. So I feel like we can disable scaling long term and use load balancer -- since that is what any real user would need anyways.

[frankbu]

I wasn't aware that we always deploy metaldb in the integration test env. I ran the tests locally and confirmed that, in fact, s.LoadBalancerSupported is true, so I don't understand how this part of the issue description can be true:

we set the Istiod Address

This is a pod IP

The code uses the service IP for the istiodAddress if there is loadbalancer support.

Is the postsubmit configuration different, i.e., no metaldb deployment)?

@howardjohn Am I missing something here?

[frankbu]

Opened new issue #37217.

This issue is fixed by #37170.

[howardjohn]

https://prow.istio.io/view/gs/istio-prow/pr-logs/pull/istio_istio/39896/integ-pilot-istiodremote_istio/1548796671507304448, https://prow.istio.io/view/gs/istio-prow/pr-logs/pull/istio_istio/39896/integ-pilot-istiodremote_istio/1548635578541543424

[howardjohn]

https://prow.istio.io/view/gs/istio-prow/logs/integ-pilot-istiodremote_istio_postsubmit/1549524310748565504