Proposal: make Let's Encrypt a first-class entity in Workflow

[jmspring]

At the Azure / Deis hackfest at in December, there was time to work on additional enhancements. One such was a PoC for adding support for Let's Encrypt certificates (to enable TLS) when new applications are deployed to Workflow.

The initial PoC takes on two parts:

1. An application that makes periodic checks for new applications: https://github.com/jmspring/deis_certificate_manager

2. A worker process that the (1) spawns to do the actual Let's Encrypt/ACME handshake:
   https://github.com/jmspring/deis_cert_letsencrypt_worker

This is just a PoC and real support should directly work with the ACME protocol and not necessarily shell out another process to do the handshake.

Further, some research into limitations in/around Let's Encrypt certificate issuance policies (limits per domain, etc) should be considered.

Also, the PoC only retrieves the initial certificate, Let's Encrypt certs have ~90 day life time so renewal as well as revocation as apps are removed should be considered as well.

@mboersma - Jason Hansen mentioned you were Workflow lead.

[bacongobbler]

related: deis/router#134

[jmspring]

@bacongobbler, @mboersma - thoughts on moving forward on this? I have cycles to work on it.

[bacongobbler]

This sounds good to me!

Personally I'd like to see this fleshed out in the client as a new deis certs:generate command, which would essentially work with the ACME protocol to retrieve a certificate from Lets Encrypt and upload it to the server via deis certs:add. The POC could also look into re-running deis certs:generate to renew the certificate. That implementation may need to be performed by a new deis certs:update command, which currently does not exist. I know others have asked for that feature as well.

I think this is something that should be supported as a first-class citizen in the CLI rather than a sidekick container. Other than that, the idea of generating certs for domains using Let's Encrypt and using existing commands to upload/update/delete the cert would probably be the best approach.

What do you think?

[jmspring]

Part of the Let's Encrypt Handshake (for issuance) currently requires some form of validation of the domain. The PoC was one quick hack for such. I'll dive into the protocol a bit more this week and solidify my understanding both of issuance and renewal -- the certs are only 90 day certs, so renewal (as well as revocation will be important).

My concern if it is CLI only and requires said handshake, what the best approach is. As for updating nginx (per the thread you linked to -- the hash of a cert makes sense to trigger the update).

I'll get some more info by Thursday.

[MartinodF]

We're investigating deis and one of the main issues we found is the lack of Let's Encrypt integration. Our current solution on Kubernetes relies on kube-lego together with nginx-ingress.

kube-lego monitors the kubernetes cluster for ingresses with a specific annotation. It then reads the spec.tls.hosts field from those and compares the result with the available certificates. When some are missing or near the expiry date, it requests a new certificate, creating a new ingress rule which matches the acme challenge path and intercepting the Let's Encrypt validation request. It then saves the new certificates in a secret, ready to be used by the application.

I believe such an approach would work for deis too, wouldn't it?

[bacongobbler]

@MartinodF just following up on this comment, but with #732 in place, yes. You could just run ingress and kube-lego on the cluster as usual.

[bacongobbler]

Now with the experimental native ingress feature in v2.13.0, I've been able to get a Workflow cluster up and my apps being backed by letsencrypt running using a combination of kube-lego and the nginx ingress controller. I'll hopefully get to writing a blog post about the entire setup so others can learn about the internals and see where we can write a feature around it.

[johnschultz]

@bacongobbler really interested in how you got that to work. Is that blog post up somewhere or can you point me in the right direction? Thanks!

edit: apologies, didn't realize commenting would re-open the issue.

[mariusmarais]

@johnschultz Native ingress via Traefik with its built-in Let Encrypt support is working quite well for us too. Default app names work perfectly, but deis domain:xxx needs manual changes to the ingress for the app, which I'm assuming will be handled by deis/controller#1277. This is probably true for nginx too.

[ConnorJC3]

Any progress on this? Yes, I can use native ingresses, but using Deis's router is preferable.

[bacongobbler]

No progress. The "hack" I did was merely just manually editing ingress routes and installing the nginx ingress controller, along with kube-lego. No code to contribute at this time.

[ConnorJC3]

Does anyone who's setup native ingresses know if Deis adds the annotations for TLS (Or if there is a way to configure them)? Because the way that kube-lego works is that it looks for annotations to indicate which domain to pick.

[bacongobbler]

It does not; I had to manually edit the ingress resources to add those annotations in.

[Cryptophobia]

This issue was moved to teamhephy/workflow#43