One egress gateway TCP port does not route to different endpoints for TCP traffic

[ashleythomasv28]

Bug description
My use case is to route TCP traffic to different external endpoints. I create a gateway resource with 1 port for TCP protocol. Additionally, I create service entry, virtual service with destinationSubnets set as the external IP address to access but I observed that only the first endpoint's cluster come up in egress gateway's listeners

[ ] Docs
[ ] Installation
[x] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Expected behavior
Egress gateway should have filters to match all TCP based external endpoints.

Steps to reproduce the bug

1. Create gateway resource with 1 port for TCP protocol

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 81
      name: tcp
      protocol: TCP
    hosts:
    - "*"

2. Create service entry for 1st service

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: external-tcp-endpoint-1
spec:
  hosts:
  - external-endpoint-1.com
  addresses:
  - 10.20.30.40
  location: MESH_EXTERNAL
  ports:
  - number: 1234
    name: tcp
    protocol: TCP
  resolution: STATIC
  endpoints:
  - address: 10.20.30.40

3. Create virtual service for 1st service

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: external-tcp-endpoint-vs-1
spec:
  hosts:
  - external-endpoint-1.com
  gateways:
  - istio-egressgateway
  - mesh
  tcp:
  - match:
    - gateways:
      - mesh
      destinationSubnets:
      - 10.20.30.40
      port: 1234
    route: # route requests from mesh to egress gateway 
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        port:
          number: 81
      weight: 100
  - match:
    - gateways:
      - istio-egressgateway
      destinationSubnets:
      - 10.20.30.40
      port: 81
    route: # route requests from egress gateway to external service
    - destination:
        host: external-endpoint-1.com
        port:
          number: 1234
      weight: 100

4. Check listeners for egress gateway:
   a. Can see cluster of endpoint 1 appearing for listener 0.0.0.0:81
   b. But do not see the following on egress gateway, which is seen for other pod's sidecars. That means virtual service's destinationSubnets does not take into effect on egress gateway.

"prefixRanges": [
                       {
                           "addressPrefix": "10.20.30.40",
                           "prefixLen": 32
                       }
                   ]

5. Create service entry for 2nd endpoint

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: external-tcp-endpoint-2
spec:
  hosts:
  - external-endpoint-2.com
  addresses:
  - 10.50.60.70
  location: MESH_EXTERNAL
  ports:
  - number: 8080
    name: tcp
    protocol: TCP
  resolution: STATIC
  endpoints:
  - address: 10.50.60.70

6. Create virtual service for 2nd endpoint

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: external-tcp-endpoint-vs-2
spec:
  hosts:
  - external-endpoint-2.com
  gateways:
  - istio-egressgateway
  - mesh
  tcp:
  - match:
    - gateways:
      - mesh
      destinationSubnets:
      - 10.50.60.70
      port: 8080
    route: # route requests from mesh to egress gateway 
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        port:
          number: 81
      weight: 100
  - match:
    - gateways:
      - istio-egressgateway
      destinationSubnets:
      - 10.50.60.70
      port: 81
    route: # route requests from egress gateway to external service
    - destination:
        host: external-endpoint-2.com
        port:
          number: 8080
      weight: 100

7. Check listeners for egress gateway:
   a. Can only see cluster of endpoint 1 appearing for listener 0.0.0.0:81
   b. Even when accessing endpoint 2, egress gateway accesses endpoint 1

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
istioctl version --remote
client version: 1.6.8
control plane version: 1.6.8

kubectl version
Major:"1", Minor:"18"
Major:"1", Minor:"18"

How was Istio installed?
Using istioctl install

Please let me know if I am missing something. Or is there some other way of accessing TCP traffic. Also why doesn't virtual service's destinationSubnets take effect on egress gateway?

[howardjohn]

I don't think its possible to configure this, because the destination IP is lost by the connection from sidecar to gateway - the destination IP will always be the gateway IP, which is not what we want

[istio-policy-bot]

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2020-10-22. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.

[gadiener]

Hi @ashleythomasv28, have you found a solution for this? I'm facing the same issue. Thanks

[ashleythomasv28]

@gadiener I didn't try but with istio's smart DNS, when service entries are created with resolution as DNS, there should be individual listeners for each external endpoint (due to class E IP address allocation) pointing to their own cluster. You can check with this feature enabled in Istio 1.8

[gadiener]

@ashleythomasv28 Thanks for your answer. Unfortunately I don't see Smart DNS as an option as it is in alpha as we speak. For internal policy we cannot use it in production yet.

I'm fixing it using the SNI host of the mTLS to route the traffic in the egress, I just did some tests but it seems to fix the issue.