Plaintext Endpoints via PeerAuthentication aren't filtered out in multi-network

[stevenctl]

We filter out non-tls endpoints from cross-network LoadBalancing. Without TLS we don't have SNI, required to do routing at the gateway.

istio/pilot/pkg/xds/ep_filters.go

Lines 79 to 82 in 86b87f2

	if tlsMode := envoytransportSocketMetadata(lbEp, "tlsMode"); tlsMode == model.DisabledTLSModeLabel {
	// dont allow cross-network endpoints for uninjected traffic
	continue
	}

As part of #28621 and #28720 we're seeing the test apply the following PeerAuthentications, but the unreachable endpoints aren't being removed from the LoadBalancing rotation.

Global mtls disable

# mTLS is disabled without destination rule.
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  annotations:
    test-suite: "beta-mtls-off"
spec:
  mtls:
    mode: DISABLE

Port level mtls disable

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "mtls"
spec:
  selector:
    matchLabels:
      app: server
  mtls:
    mode: STRICT
  portLevelMtls:
    8090:
      mode: DISABLE
    8092:
      mode: DISABLE
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: server
spec:
  host: server.%s.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
    portLevelSettings:
    - port:
        number: 8090
      tls:
        mode: DISABLE
    - port:
        number: 8092
      tls:
        mode: DISABLE

Does the PeerAuthentication set the tlsMode early enough in our EDS code to be filtered out properly?

[stevenctl]

cc @jtrbs

[jtrbs]

Related issue #28766

[yangminzhu]

cc @incfly

[incfly]

I clicked through the link and having a bit trouble to connect the dots.

 if tlsMode := envoytransportSocketMetadata(lbEp, "tlsMode"); tlsMode == model.DisabledTLSModeLabel { 

This line is only looking at the endpoint label. The label is the result of the injection. And it's not taking PeerAuthenticatoin into account.

but the unreachable endpoints aren't being removed from the LoadBalancing rotation.

What are the characteristics of those enpdoints? With sidecar injected but configured as plaintext, therefore cross cluster traffic fails?

[stevenctl]

Yes, we require TLS for SNI routing through the gateway. This doesn't fail for cross-cluster, single-network. Only cross-network traffic fails.

[incfly]

I see. To answer the question, yes, I think we gather the PeerAuthentication config at the time when push context is initiated. If we consult that, it should be able to tell whether endpoints are plaintext or not.

istio/pilot/pkg/model/push_context.go

Line 1146 in 71b7265

if ps.AuthnBetaPolicies, initBetaPolicyErro = initAuthenticationPolicies(env); initBetaPolicyErro != nil {

Though I would look into how the client side auto mTLS decision is made, in cluster.go,

istio/pilot/pkg/networking/core/v1alpha3/cluster.go

Line 589 in 8b10dcb

tls, mtlsCtxType = buildAutoMtlsSettings(tls, opts.serviceAccounts, opts.istioMtlsSni, opts.proxy,

trying to reuse as much as we can.

[stevenctl]

So if i understand correctly this is mostly related to the TLSContext on the cluster, not info directly on the lb endpoints. I'll have to look into a way to reference that in the eds flow with minimal recomputation.

I did notice that we surround the buildAutoMtlsSettings call with a check that we're not in sni-dnat mode.

istio/pilot/pkg/networking/core/v1alpha3/cluster.go

Line 624 in f1aa586

if opts.clusterMode != SniDnatClusterMode && opts.direction != model.TrafficDirectionInbound {

We want to filter these non-tls endpoints from the logic that converts endpoint address to gateway address, but we want to filter the endpoints entirely when building eds for the gateways themselves which have ISTIO_META_ROUTER_MODE: sni-dnat

istio/samples/multicluster/gen-eastwest-gateway.sh

Lines 97 to 98 in a43810c

	- name: ISTIO_META_ROUTER_MODE
	value: "sni-dnat"

Not sure how to handle this without recomputing some of the traffic policy logic inside of eds