Proposal: e2e SSL for fun and profit

[krancour]

Overview

One persistent shortcoming of Deis v1.x is something that was once referred to as "mullet TLS." That is, we had security in the front and a party in the back.

Ref deis/deis#4611, deis/deis#4629, deis/deis#4757

The Deis platform's shift to k8s and the new deis-router offer us the unique possibility of dramatically simplifying some of the previously discussed approaches to solving this.

tl;dr:

Proposal: Give developers or operators the option to "move" the router closer to their application.

Challenges

• In order to meet various regulatory requirements (e.g. those imposed by HIPAA or PCI-DSS) some customers require end-to-end SSL.
• Terminating SSL and re-initiating it can incur a costly performance penalty and adds undesirable complexity. Examples of this, which we would ideally avoid include:
  • Terminating SSL at the edge / external (to k8s) load balancer (e.g. Amazon ELB) and then re-initiating SSL to the (platform) router.
  • Terminating SSL at the (platform) router and then re-initiating SSL to the application (pod).

Solution

In order to provide end-to-end SSL without the overhead or complexity involved in terminating and re-initiating secure connections, the best option may be to accomplish as much routing as possible at L4, leveraging the flexibility of k8s and the new Deis router component. What follows are three options varying in security and complexity. Each is well suited to a variety of requirements and use cases. Only the third achieves end-to-end SSL and is a novel proposition that requires development effort. Thus it is the third option that accounts for the bulk of this proposal. The point of documenting all three options is to provide a basis for comparison and to demonstrate how the router can facilitate different scenarios through varying configuration and "placement." (This has the makings of an excellent blog post.)

Option 1: Terminate SSL at the edge / load balancer

This is an option that we've seen Deis operators exercise before. It's plausible in cases where security requirements are not stringent and minimal customization of the Deis platform is needed or desired. In a case such as this, a single wildcard certificate (or more, if supported) can be used to terminate SSL at the edge / load balancer and pass only unsecured requests through to the Deis router(s).

Benefits:

• Extremely easy to setup

Deficiencies:

• Weakest security
• Anywhere that a load balancer can be configured with only a single wildcard certificate (e.g. Amazon ELB), SSL support for additional "custom" domains will not be possible unless additional load balancers are added for the express purpose of terminating SSL for those other domains.

Use cases:

This could work very nicely in a cluster that strictly serves non-production applications.

Option 2: Terminate SSL at the Deis router(s)

This is the most common case. (It's how Deis works out-of-the-box.) The edge / load balancer will pipe raw TCP traffic through to the Deis router(s) and SSL is terminated there. These requests are proxied through to upstream application pods without SSL.

Benefits:

• Moderately secure
• Can easily support numerous "custom" domains
• Easy to setup (using deis CLI)

Deficiencies:

• Only moderately secure

Use cases:

This works very nicely for non-production clusters and for production clusters hosting workloads that are not subject to strict security requirements.

Option 3: Terminate SSL at the pods(s)

Both scenarios above are easily achieved. Another attribute that they have in common is that they do not require an additional edge / load balancer (and associated costs; more on this in a moment).

Before continuing, it's worth asking how we can leverage the flexibility of k8s itself by "dropping to k8s" to solve problems that aren't easily solved within the bounds of the conventional Deis workflow. k8s already exposes a resource that can be associated (usually seamlessly) to its own external (to the cluster) load balancer (on most platforms). Namely, these are services of type: LoadBalancer. Depending on one's underlying infrastructure and distribution of k8s, creating such services also programmatically creates the corresponding load balancer. For instance, this works brilliantly on kube-aws and on GKE. (It doesn't work so well on Vagrant or bare metal; but we will still cover how to deal with those.)

k8s services of type: LoadBalancer, on supported infra, will programmatically create external / edge load balancers that pipe raw TCP traffic (on ports of your choice) through to consistently numbered and highly numbered host ports that are programmatically allocated on every node and listened to by the kube-proxy-- which in turn routes the traffic to (once again, ports of your choice) on the service's endpoints (backend pods). If this seems familiar, it's because it's how the Deis router(s) already work-- how they already get their traffic from the outside world into the cluster.

The salient point in the above is that at every turn, it is just TCP packets that are being routed. No SSL termination takes place. This means that such a configuration achieves a secure route from the end-user's agent all the way through to a service endpoint (pod). The only remaining question is how to terminate SSL within the pod.

Again, this is where the flexibility of k8s comes to the rescue. Pods may contain multiple containers. Multiple containers within a single pod may share disk, network, etc., and are physically located on a single node. Running a "sidecar" "terminator" container in each application pod would permit termination of SSL, with unsecure traffic proxied to an upstream over the local interface. In other words, terminating SSL in a sidecar would allow applications to not incur this responsibility themselves, but would still ensure that no unencrypted traffic is ever flowing "over the wire." (Compare this to the "old days" when one would put Apache or Nginx on each application host.)

At this juncture, the question becomes how best to terminate SSL in an application sidecar. The proposal: the Deis router component can already do that very well, and since the use of the service of type: LoadBalancer would effectively bypass the (platform) router(s), it makes sense to re-introduce the Deis router into this configuration, but within the pod. i.e. as a sidecar. This has numerous benefits:

• SSL termination at the pod (e2d SSL, finally)
  • Even for multiple "custom" domains, each protected by its own cert!
• Brings along all the router's other features and capabilities
  • Which can be tweaked further to meet the needs of the specific application in ways that would be inadvisable for a router instance serving the whole platform
• Requires fairly minimal code changes to adapt to such a scenario

To tie up loose ends:

• On infra such as Amazon, for instance, this strategy amounts to one extra ELB per application requiring e2e SSL. Ballbark, that is a cost of $10/month.
• This won't work well on Vagrant. Who cares? Who needs e2e SSL on Vagrant?
• This won't work so nicely on bare metal-- but it is not impossible. While automatic provisioning of of an external load balancer might be out in response to a service of type: LoadBalancer, what one still gets it consistent, high numbered ports allocated on every node, with the kube-proxy listening to those and forwarding traffic to service endpoints. With this being the case, assuming you do have external load balancer(s) of some sort at your disposal, it's remarkably easy to configure/reconfigure those manually to support this.
• It remains the responsibility of the operator to make/modify DNS entries that will route traffic for the appropriate domain(s) to the new load balancer.

Option 3: Proposed implementation:

First, this feature would be strictly opt-in, on an app-by-app basis.

When an app owner opts into this feature (via deis CLI) this should modify the existing k8s service definition for the application (just as this definition is already modified each time changes are made to a domain or certificate associated to the application). Namely, it should change the type to LoadBalancer and remove the router.deis.io/routable: "true" annotation. This will stop the platform router(s) from handling and further traffic for the application. Some other annotation like router.deis.io/terminable: "true" probably takes its place.

Similarly, the replication controller definition for the application will need to be modified. The modification would include a second container described in the template for application pods. This definition would look something like the following:

• Would use the same deis-router image as the platform router(s).
• Would be configured slightly differently:
  • Would set POD_NAMESPACE environment variable to the same namespace as the application.
  • Would set some TBD environment variable or annotation to cue the router to run in "terminator mode."
  • Would set some TBD environment variable or annotation to cue the router on what port the upstream application process is listening on.
  • Would mount a service account belonging to the application's namespace.
    • Workflow would have to manage this service account as well. (Cannot assume k8s will handle this automatically as that depends what admission controls are in place.)

All of the above should be reversed if the operator reverses their opt-in to e2e SSL.

cc/ping @helgi, @slack, and @technosophos in particular for feedback.