Weird Behavior of Istio Gateway with HTTPS

[bangau1]

Describe the bug
VirtualService route to the same gateway causing https failure

Expected behavior
It should working fine by terminate the SSL in the first layer Gateway

Steps to reproduce the bug
See the description below

Version

$istioctl version
Version: 1.0.0
GitRevision: 3a136c90ec5e308f236e0d7ebb5c4c5e405217f4
User: root@71a9470ea93c
Hub: gcr.io/istio-release
GolangVersion: go1.10.1
BuildStatus: Clean

$kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T10:09:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.5-gke.5", GitCommit:"9aba9c1237d9d2347bef28652b93b1cba3aca6d8", GitTreeState:"clean", BuildDate:"2018-12-11T02:36:50Z", GoVersion:"go1.10.3b4", Compiler:"gc", Platform:"linux/amd64"}

Installation
Using helm installation

Environment
GKE

Issue Description

Hi Guys,

Before I describe my issue is, I want to explain briefly about my use case:

• We have 2 backends: foo and bar in namespace team.
  • foo is accesible by public hostname of foo-team.company.com
  • bar is accessible by public hostname of bar-team.company.com
  • Each backend has its own CICD pipeline that utilizes Istio related stuff to split traffic between stable vs canary.
    So, for example for foo backend, this is how its like:

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: foo-app-gateway
  namespace: team
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - foo-team.company.com
    port:
      name: http-example-app
      number: 80
      protocol: HTTP
  - hosts:
    - foo-team.company.com
    port:
      name: https-example-app
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: foo-app-virtualservice
  namespace: team
spec:
  gateways:
  - foo-app-gateway
  - mesh
  hosts:
  - foo-team.company.com
  - foo.team.svc.cluster.local
  http:
  - route:
    - destination:
        host: foo.team.svc.cluster.local
        port:
          number: 8080
        subset: stable
      weight: 70
    - destination:
        host: foo.team.svc.cluster.local
        port:
          number: 8080
        subset: canary
      weight: 30

Imagine resources above is managed by the team (not by Central team), that can change the canary and stable weight.

Now, the team want to consolidate the backend public hostname into 1 and routed by path. Example:

• foo -> accessible from team.company.com/foo
• bar -> accessible from team.company.com/bar
  Surely we can use 1 Istio Gateway, Virtual Service and DestinationRule that consolidate the rules for these 2 backends and put that into 1 centralized repository.
  However, making it centralized makes it harder for each backend to do automated CICD pipeline with canary split traffic.

So, to overcome this problem, I introduce another Gateway and VirtualService resource like below:

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: public-gateway
  namespace: central-team
spec:
  selector:
    istio: central-ingressgateway
  servers:
  - hosts:
    - team.company.com
    port:
      name: http-app
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: false
  - hosts:
    - team.company.com
    port:
      name: https-app
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: public-gateway-virtualservice
  namespace: central-team
spec:
  gateways:
  - public-gateway
  - mesh
  hosts:
  - team.company.com
  http:
  - match:
    - uri:
        prefix: /foo/
    rewrite:
      authority: foo-team.company.com
      uri: /
    route:
    - destination:
        host: istio-ingressgateway.istio-system.svc.cluster.local
        port:
          number: 80
  - match:
    - uri:
        prefix: /bar/
    rewrite:
      authority: bar-team.company.com
      uri: /
    route:
    - destination:
        host: istio-ingressgateway.istio-system.svc.cluster.local
        port:
          number: 80

That resource is managed in a centralized style (located in namespace: central-team), regardless on how each backend has its own traffic management rules. Note that the centralized resource is using istio: central-ingressgateway as the selector, while the backend's one is using istio: ingressgateway. So they are using different Ingress Gateway Controller pods. Also, the centralized resources is rewriting the authority and uri so that it can be routed by the team's istio ingress gateway. The difference of traffic flow is like this:

• End user access foo.company.com -> Team's Istio Ingress Gateway Pods (istio:ingressgateway) -> foo.team.svc.cluster (it has istio-proxy on it).
• End User access team.company.com/foo -> Central Istio Ingress Gateway Pods (istio:central-ingressgateway) -> Istio Ingress Gateway Pods (istio:ingressgateway) -> foo.team.svc.cluster (it has istio-proxy on it).

The approach above is working fine, meaning team.company.com/foo successfully routed to foo backend, either https or http plain text.

However, as you can see, there are 2 Istio Ingress Gateway deployment being used: that has selector istio: central-ingressgateway and istio: ingressgateway. Out of curiousity, I tested the central resource by using the same selector istio: ingressgateway but somehow accessing via HTTPS failed but HTTP plain text is working fine.

curl https://team.company.com/foo
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to team.company.com/foo:443

But access the https://foo-team.company.com is okay. Both are using *.company.com SSL certificate.

Do you guys have any idea why this behavior occur?

[cyx]

I'm seeing the same behavior with multiple Gateway definitions, each with their own TLS settings. So far the only solution has been to consolidate to one Gateway.

My sense is that this is related to not using SDS for certs, and might be addressed in the upcoming 1.1 release.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in the next 30 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had activity in the last month and a half. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.