Gateway with wildcard hosts for different namespaces causes duplicate filterChainMatch in envoy listener

[maxbischoff]

Bug description

When creating a gateway that has a host section with identical hosts on different namespaces on port 443 with simple-mode TLS, the resulting listener config is always rejected by envoy

  - hosts:
    - namespace1/*.example.com
    - namespace2/*.example.com
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: <key-path>
      serverCertificate: <cert-path>

The log message in the gateway deployment's envoy is

 [2019-08-27 15:51:48.555][22][warning][config] [external/envoy/source/common/config/grpc_mux_subscription_impl.cc:73] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) 0.0.0.0_443: error adding listener '0.0.0.0:443': multiple filter chains with overlapping matching rules are defined

The proxy diff (retrieved with istioctl proxy-status <gateway-proxy>) contains following listener:

-         "listener": {
-            "name": "0.0.0.0_443",
-            "address": {
-               "socketAddress": {
-                  "address": "0.0.0.0",
-                  "portValue": 443
-               }
-            },
-            "filterChains": [
-               {
-                  "filterChainMatch": {
-                     "serverNames": [
-                        "*.example.com",
-                        "*.example.com"
-                     ]
-                  },
-                  "tlsContext": {

...

Affected product area (please put an X in all that apply)

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[x] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Expected behavior

Pilot should deduplicate the wildcard hosts in this case, as no real conflict exists.

Steps to reproduce the bug

In an istio cluster with default ingress gateway create following test gateway:

echo 'apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: test
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - namespace1/*.example.com
    - namespace2/*.example.com
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/certs/tls.key
      serverCertificate: /etc/certs/tls.crt
' | kubectl apply -f -

Then retrieve the proxy-status of your ingress gateway. Observe that LDS is STALE:

istioctl proxy-status
NAME                                                   CDS        LDS                            EDS               RDS        PILOT                            VERSION
istio-ingressgateway-5f54f8875b-dt7ns.istio-system     SYNCED     STALE (Never Acknowledged)     SYNCED (100%)     SYNCED     istio-pilot-77bfb56b6b-xpjp4     1.2.4
istio-ingressgateway-5f54f8875b-zjnmb.istio-system     SYNCED     STALE (Never Acknowledged)     SYNCED (100%)     SYNCED     istio-pilot-77bfb56b6b-xpjp4     1.2.4

Obtain the diff using istioctl proxy-status istio-ingressgateway-5f54f8875b-dt7ns.istio-system (replace with your gateway pod's name)

Proxy diff

Clusters Match
--- Pilot Listeners
+++ Envoy Listeners
@@ -191,162 +191,14 @@
                   "name": "envoy.listener.tls_inspector"
                }
             ]
          }
       },
       {
          "listener": {
-            "name": "0.0.0.0_443",
-            "address": {
-               "socketAddress": {
-                  "address": "0.0.0.0",
-                  "portValue": 443
-               }
-            },
-            "filterChains": [
-               {
-                  "filterChainMatch": {
-                     "serverNames": [
-                        "*.example.com",
-                        "*.example.com"
-                     ]
-                  },
-                  "tlsContext": {
-                     "commonTlsContext": {
-                        "tlsCertificates": [
-                           {
-                              "certificateChain": {
-                                 "filename": "/etc/certs/tls.crt"
-                              },
-                              "privateKey": {
-                                 "filename": "/etc/certs/tls.key"
-                              }
-                           }
-                        ],
-                        "alpnProtocols": [
-                           "h2",
-                           "http/1.1"
-                        ]
-                     },
-                     "requireClientCertificate": false
-                  },
-                  "filters": [
-                     {
-                        "name": "envoy.http_connection_manager",
-                        "config": {
-                              "access_log": [
-                                    ],
-                              "forward_client_cert_details": "SANITIZE_SET",
-                              "generate_request_id": true,
-                              "http_filters": [
-                                       {
-                                                "config": {
-                                                         "default_destination_service": "default",
-                                                         "forward_attributes": {
-                                                                  "attributes": {
-                                                                           "source.uid": {
-                                                                                    "string_value": "kubernetes://istio-ingressgateway-5f54f8875b-dt7ns.istio-system"
-                                                                                 }
-                                                                        }
-                                                               },
-                                                         "mixer_attributes": {
-                                                                  "attributes": {
-                                                                           "context.reporter.kind": {
-                                                                                    "string_value": "outbound"
-                                                                                 },
-                                                                           "context.reporter.uid": {
-                                                                                    "string_value": "kubernetes://istio-ingressgateway-5f54f8875b-dt7ns.istio-system"
-                                                                                 },
-                                                                           "source.namespace": {
-                                                                                    "string_value": "istio-system"
-                                                                                 },
-                                                                           "source.uid": {
-                                                                                    "string_value": "kubernetes://istio-ingressgateway-5f54f8875b-dt7ns.istio-system"
-                                                                                 }
-                                                                        }
-                                                               },
-                                                         "service_configs": {
-                                                                  "default": {
-                                                                           "disable_check_calls": true
-                                                                        }
-                                                               },
-                                                         "transport": {
-                                                                  "check_cluster": "outbound|15004||istio-policy.istio-system.svc.cluster.local",
-                                                                  "network_fail_policy": {
-                                                                           "base_retry_wait": "0.080s",
-                                                                           "max_retry_wait": "1s",
-                                                                           "policy": "FAIL_CLOSE"
-                                                                        },
-                                                                  "report_cluster": "outbound|15004||istio-telemetry.istio-system.svc.cluster.local"
-                                                               }
-                                                      },
-                                                "name": "mixer"
-                                             },
-                                       {
-                                                "name": "envoy.cors"
-                                             },
-                                       {
-                                                "name": "envoy.fault"
-                                             },
-                                       {
-                                                "name": "envoy.router"
-                                             }
-                                    ],
-                              "http_protocol_options": {
-                                    },
-                              "normalize_path": true,
-                              "rds": {
-                                       "config_source": {
-                                                "ads": {
-                                                      },
-                                                "initial_fetch_timeout": "0s"
-                                             },
-                                       "route_config_name": "https.443.https.test.istio-system"
-                                    },
-                              "server_name": "istio-envoy",
-                              "set_current_client_cert_details": {
-                                       "cert": true,
-                                       "dns": true,
-                                       "subject": true,
-                                       "uri": true
-                                    },
-                              "stat_prefix": "0.0.0.0_443",
-                              "stream_idle_timeout": "0s",
-                              "tracing": {
-                                       "client_sampling": {
-                                                "value": 100
-                                             },
-                                       "operation_name": "EGRESS",
-                                       "overall_sampling": {
-                                                "value": 100
-                                             },
-                                       "random_sampling": {
-                                                "value": 1
-                                             }
-                                    },
-                              "upgrade_configs": [
-                                       {
-                                                "upgrade_type": "websocket"
-                                             }
-                                    ],
-                              "use_remote_address": true
-                           }
-                     }
-                  ]
-               }
-            ],
-            "listenerFilters": [
-               {
-                  "name": "envoy.listener.tls_inspector"
-               }
-            ]
-         }
-      },
-      {
-         "listener": {
             "name": "0.0.0.0_8060",
             "address": {
                "socketAddress": {
                   "address": "0.0.0.0",
                   "portValue": 8060
                }
             },

Routes Match

Version (include the output of istioctl version --remote and kubectl version)

istioctl version --remote
client version: 1.2.4
citadel version: 1.2.4
galley version: 1.2.4
galley version: 1.2.4
ingressgateway version: 1.2.4
ingressgateway version: 1.2.4
pilot version: 1.2.4
pilot version: 1.2.4
policy version: 1.2.4
policy version: 1.2.4
sidecar-injector version: 1.2.4
telemetry version: 1.2.4
telemetry version: 1.2.4

kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?

Using the official helm chart

Environment where bug was observed (cloud vendor, OS, etc)

Custom bare-metal cluster

[maxbischoff]

@howardjohn @hzxuzhonghu can you tell me, whether that behavior is intended or not? I will gladly fix this issue, if you can point me in the right direction. My current approach would be deduplicating the gateway hosts before they are used to generate the listener config.

[howardjohn]

Definitely not intended -- anytime you see Envoy rejecting config from pilot it is not intended. Maybe add some more cases to

istio/pilot/pkg/model/gateway.go

Line 207 in 454fe74

func checkDuplicates(hosts []string, knownHosts map[string]struct{}) []string {

. Or in NamesForNamespace maybe

[maxbischoff]

Thanks, I'll take a look at that

[maxbischoff]

Is another release for 1.2 planned? If yes, I would like to cherry-pick this to 1.2, too.