v2 <-> v3 TlsContext causes temporary downtime

[howardjohn]

Title: v2 <-> v3 TlsContext causes temporary downtime

Description:
Switching the TLS context version on a cluster causes a downtime

Apply this change (full cluster below) to a v2 cluster via XDS:

>           "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
<           "@type": "type.googleapis.com/envoy.api.v2.auth.UpstreamTlsContext",

Fails a few requests with client disconnected, failure reason: TLS error: Secret is not supplied by SDS.

Repro steps:
I can easily reproduce this with Istio control plane just swapping out the version. I assume it could be reproduce with file based XDS but haven't produced a minimal reproducer

Full cluster:

{
    "transportSocketMatches": [
        {
            "name": "tlsMode-istio",
            "match": {
                "tlsMode": "istio"
            },
            "transportSocket": {
                "name": "envoy.transport_sockets.tls",
                "typedConfig": {
                    "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
                    "commonTlsContext": {
                        "tlsCertificateSdsSecretConfigs": [
                            {
                                "name": "default",
                                "sdsConfig": {
                                    "apiConfigSource": {
                                        "apiType": "GRPC",
                                        "grpcServices": [
                                            {
                                                "envoyGrpc": {
                                                    "clusterName": "sds-grpc"
                                                }
                                            }
                                        ]
                                    }
                                }
                            }
                        ],
                        "combinedValidationContext": {
                            "defaultValidationContext": {
                                "matchSubjectAltNames": [
                                    {
                                        "exact": "spiffe://cluster.local/ns/testyomesh/sa/default"
                                    }
                                ]
                            },
                            "validationContextSdsSecretConfig": {
                                "name": "ROOTCA",
                                "sdsConfig": {
                                    "apiConfigSource": {
                                        "apiType": "GRPC",
                                        "grpcServices": [
                                            {
                                                "envoyGrpc": {
                                                    "clusterName": "sds-grpc"
                                                }
                                            }
                                        ]
                                    }
                                }
                            }
                        },
                        "alpnProtocols": [
                            "istio-peer-exchange",
                            "istio"
                        ]
                    },
                    "sni": "outbound_.80_._.testyomesh-1.testyomesh.svc.cluster.local"
                }
            }
        },
        {
            "name": "tlsMode-disabled",
            "match": {},
            "transportSocket": {
                "name": "envoy.transport_sockets.raw_buffer"
            }
        }
    ],
    "name": "outbound|80||testyomesh-1.testyomesh.svc.cluster.local",
    "type": "EDS",
    "edsClusterConfig": {
        "edsConfig": {
            "ads": {}
        },
        "serviceName": "outbound|80||testyomesh-1.testyomesh.svc.cluster.local"
    },
    "connectTimeout": "10s",
    "circuitBreakers": {
        "thresholds": [
            {
                "maxConnections": 4294967295,
                "maxPendingRequests": 4294967295,
                "maxRequests": 4294967295,
                "maxRetries": 4294967295
            }
        ]
    },
    "filters": [
        {
            "name": "istio.metadata_exchange",
            "typedConfig": {
                "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                "typeUrl": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
                "value": {
                    "protocol": "istio-peer-exchange"
                }
            }
        }
    ]
}

[lizan]

@howardjohn I could only reproduce (w a mocking config) this behavior on 1.14.x branch, i.e. istio proxy 1.6.x, but not with 1.15.x, 1.16.x or latest master. Can you confirm that's the case? If that's the case the upgrade path will be update to 1.7.x proxy first then migrate to v3 transport socket.

I'll try to figure out which commit exactly fixes this and report back.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.