Egress Gateways with TLS Origination AND TLS passthrough

[justinkillen]

Bug description

In the article 'Egress Gateways with TLS Origination' (https://istio.io/docs/tasks/traffic-management/egress/egress-gateway-tls-origination/), there is no consideration for https request from the client pod. Once the setup is complete, http request function correctly (e.g. 'http://edition.cnn.com/politics'), but https request hang for 3 minutes then times out:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 151.101.193.67:443...
* Connected to edition.cnn.com (151.101.193.67) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /cacert.pem
  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
  0     0    0     0    0     0      0      0 --:--:--  0:04:59 --:--:--     0* Operation timed out after 300287 milliseconds with 0 out of 0 bytes received
  0     0    0     0    0     0      0      0 --:--:--  0:05:00 --:--:--     0
* Closing connection 0
curl: (28) Operation timed out after 300287 milliseconds with 0 out of 0 bytes received

In particular, note in the proxy log from the curl pod that direct connections are being attempted, which defeats the purpose of pushing traffic to a gateway.

[2020-05-12T01:17:22.147Z] "- - -" 0 - "-" "-" 517 0 300202 - "-" "-" "-" "-" "151.101.65.67:443" outbound|443||edition.cnn.com 172.25.69.92:52798 151.101.193.67:443 172.25.69.92:33354 edition.cnn.com -

Expected behavior

One of the primary use cases in my eyes for pushing TLS origination to a gateway (as opposed to in the client sidecar as in https://istio.io/docs/tasks/traffic-management/egress/egress-tls-origination/) is that it provides a chokepoint to enforce security policy. So, the expected behavior would be that outbound https connections are routed through the egress gateway utilizing TLS passthrough mode.

Steps to reproduce the bug

Follow the guide (section "Perform TLS origination with an egress gateway"), but in step 5 use https instead of http in the curl command.

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

client version: 1.5.1
control plane version: 1.5.1
data plane version: 1.5.1 (11 proxies)

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T23:41:24Z", GoVersion:"go1.14", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-eks-af3caf", GitCommit:"af3caf6136cd355f467083651cc1010a499f59b1", GitTreeState:"clean", BuildDate:"2020-03-27T21:51:36Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

Helm not used

How was Istio installed?

istioctl manifest apply --set values.gateways.enabled=true --set values.gateways.istio-ingressgateway.sds.enabled=true --set values.global.k8sIngress.enabled=true --set values.global.k8sIngress.enableHttps=true --set values.global.k8sIngress.gatewayName=ingressgateway --set values.kiali.enabled=true 

Environment where bug was observed (cloud vendor, OS, etc)

AWS EKS 1.15, using AWS images for the worker nodes

[justinkillen]

Okay, I think I have it working. Proposed update for the docs:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: edition-cnn-com-egress
  namespace: istio-system
spec:
  host: istio-egressgateway
  subsets:
  - name: tls-origination
  - name: tls-passthrough
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: edition-cnn-com-origination
  namespace: istio-system
spec:
  host: edition.cnn.com
  subsets:
  - name: tls-origination
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
      portLevelSettings:
      - port:
          number: 443
        tls:
          mode: SIMPLE
  - name: tls-passthrough
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
      portLevelSettings:
      - port:
          number: 443
        tls:
          mode: DISABLE
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: edition-cnn-com
  namespace: istio-system
spec:
  selector:
    istio: egressgateway
  servers:
  - hosts:
    - edition.cnn.com
    port:
      name: http-port-for-tls-origination
      number: 80
      protocol: HTTP
  - hosts:
    - edition.cnn.com
    port:
      name: tls
      number: 443
      protocol: TLS
    tls:
      mode: PASSTHROUGH
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: edition-cnn-com
  namespace: istio-system
spec:
  hosts:
  - edition.cnn.com
  ports:
  - name: http
    number: 80
    protocol: HTTP
  - name: tls
    number: 443
    protocol: TLS
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: edition-cnn-com
  namespace: istio-system
spec:
  gateways:
  - edition-cnn-com
  - mesh
  hosts:
  - edition.cnn.com
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway
        port:
          number: 80
        subset: tls-origination
  - match:
    - gateways:
      - edition-cnn-com
      port: 80
    route:
    - destination:
        host: edition.cnn.com
        port:
          number: 443
        subset: tls-origination
      weight: 100
  tls:
  - match:
    - gateways:
      - mesh
      port: 443
      sniHosts:
      - edition.cnn.com
    route:
    - destination:
        host: istio-egressgateway
        port:
          number: 443
        subset: tls-passthrough
      weight: 100
  - match:
    - gateways:
      - edition-cnn-com
      port: 443
      sniHosts:
      - edition.cnn.com
    route:
    - destination:
        host: edition.cnn.com
        port:
          number: 443
        subset: tls-passthrough
      weight: 100

[JimmyCYJ]

Thanks @justinkillen given this is solved, I will just close it for now. Feel free to reopen it.

[justinkillen]

I feel like it would be beneficial to drop the networking and security tags, but leave it open as a docs issue.

[ceastman-ibm]

@justinkillen I am trying to accomplish the same thing, have the microservice establish an https connection through the egressgateway. i deployed the yaml you have above but when i try to test it i get the following error: wondering if you saw this while testing?

Carls-MacBook-Pro:console-ui-jmeter-scripts carleastman$ kubectl exec -it workflow-api-6c7b7d4c8d-2znmt curl https://edition.cnn.com
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
Defaulting container name to workflow-api.
Use 'kubectl describe pod/workflow-api-6c7b7d4c8d-2znmt -n ondeck' to see all of the containers in this pod.
curl: (35) Unknown SSL protocol error in connection to edition.cnn.com:443
command terminated with exit code 35

before applying the yaml that same curl from the pod worked.

[justinkillen]

@ceastman-ibm can you provide envoy logs for the source pod proxy container and the egress pod proxy container? You'll need to enable envoy access logs if you don't already have them enabled.

[ceastman-ibm]

@justinkillen i got it figured out.

[justinkillen]

@JimmyCYJ can we re-open this with only the Docs tag

[JimmyCYJ]

@nschhina has implemented the SDS based solution for egress gateway TLS/mTLS origination. The user guide is here https://preliminary.istio.io/latest/docs/tasks/traffic-management/egress/egress-gateway-tls-origination-sds/. Could you follow this link the let us know if that solves the issue?

[JimmyCYJ]

@justinkillen I am reopening this per #23740 (comment)

[nschhina]

http request -> port 80 to 443 using Virtual Service is targetting to the 443 Listener we open on Egress Gateway using gateway API, just change VS rule from port 80 to 443 and then your request will be routed properly for 'https'. Does that make sense? This is not an issue with docs/feature and imo should stay closed. All Egress docs are now automated tested so there can't possible be a bug. cc: @JimmyCYJ

[justinkillen]

@nschhina
If I deploy this as-is today, from the test pod a curl to https://my-nginx.mesh-external.svc.cluster.local won't work. If I change the VS rule to port 443, the routing will be okay, but it won't work, because instead of http in a tunnel, it's http in a tunnel in a tunnel.