Egress proxy for external services

[kyessenov]

For services that don't have pods backing them, there's just an external IP and no deployment with a proxy. We need to deploy a set of proxies to handle cluster egress traffic, and configure them to capture and route traffic.

@vaikas-google

[rshriram]

Do we need to create a set of such dummy services adding another unnecessary hop? If we augment the routing proto to include static IPs or fqdns, what then? It might complicate code a bit but it won't require the end user to declare what services are external or be surprised to see a pod for each such service or service instance. (E.g. lb1.foo.maps.com, lb2.foo.maps.com).

[kyessenov]

This issue is specifically for the ExternalName type services. Here is an example:

apiVersion: v1
kind: Service
metadata:
  name: my-app
spec:
  ports:
  - port: 12345
  type: ExternalName
  externalName: myapp.whatever.google.com

This service does not get a cluster IP and does not have pods in the cluster.
Imagine we want to monitor the traffic through this service and enforce control policies like quota.
We need to route traffic through Istio Proxy, so we create a deployment with proxies. We then rewrite the external name in the service "my-app" to point to Istio proxies, and make sure Istio proxies route the incoming traffic to "myapp.whatever.google.com". With this change, Kube DNS sends proxy address to Istio and non-Istio in-cluster clients. For Istio client-side proxy-enabled clients, we can skip Kube DNS and use Envoy discovery service.

[kyessenov]

Initial design for this feature.

1. Redirect traffic to external services to Istio proxy.
   Manager rewrites externalName field to point to a dedicated k8s service that selects proxy-egress pods running Istio proxy. The original externalName is stored as an annotation on the service definition.
   Manager also requires port definitions on the external services. Ports are optional for externalName services starting from k8s 1.6, but Envoy needs explicit listener ports in its configuration to capture traffic and choose the right protocol filter. Manager rejects external services with no port declarations.

2. Deploy egress proxies.
   There are two choices for deploying dedicated egress proxies:

2a) Deploy proxies per each external services.
Manager defines a new service for each external service to point to these proxies, which has the same port declarations and serves as the target DNS name.

2b) Share egress proxies between external services.
There is one shared deployment and one service for cluster-egress proxies.
The service is managed to list all ports across external services This shared service is the target DNS name in the external service externalName.
Proxies identify traffic using the external service hostname or VIP address.

I prefer option b), since it seems to be more in-line with sidecar proxy configuration.
Istio proxy does not need to use iptables port-rewriting since it's the sole container in its pod.

3. Handing cluster egress traffic for explicit IP addresses.
   Current k8s workaround: use a regular service without pod selector, and manually add an Endpoint with the IP and port.
   Proposed workaround: add an annotation for ExternalName services to override the DNS name with an explicit IP address. Currently, a custom endpoint is treated the same way as a regular pod, so it should be handled by existing sidecar client-side proxies. But non-istio pods need a dedicated server-side proxy deployment, which this design addresses.

[rshriram]

fixed by #463
Option 2b is implemented currently, wherein applications are expected to access the external service via their respective Kube services (over HTTP). The egress proxy will do https to the external service. Support for traffic with explicit external IP address, tcp communication with external service (and https) is TBD

[rshriram]

Closing