Ingress controller can generate real certs via letsencrypt client

[bprashanth]

Filing random idea. The missing piece is hooking up dns, which can even be human driven. For an ingress like :

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: echomap
spec:
  tls:
  - host: foo.bar.com
    port: 443
    tlsSecret:
      name: fooSecret
  rules:
 ... irrelevant...

• Run letsencrypt client
• Get a challenge like provision .well-known/acme-challenge/ at foo.bar.com:80 (https://letsencrypt.github.io/acme-spec/)
• Create the l7 for 1.2.3.4, program DNS for foo.bar.com->1.2.3.4, point urlmap of foo.bar.com:80/.well-known/acme-challenge to localhost backend serving file with expected contents
• Delete challenge url from urlmap, hook up HTTPS backend according to Ingress rules
• Keep renewing cert

[mattbates]

This sounds pretty neat @bprashanth. Can we pick this up and help contribute?

[bprashanth]

@mattbates yes. I'd be happy to answer random questions about the controller etc or review a brief sketch.

[jimmidyson]

An alternative to using the letsencrypt client is to do it programmatically using lego (https://github.com/xenolf/lego). This is used in caddy to provide the automatic cert issuance & works brilliantly. One benefit of this is being able to use multiple challenge types, e.g DNS, without needing to change configuration/reloading of l7 router.

[bprashanth]

Lego looks promising.

@mattbates the fastest way to get off the ground with certs today is probably:

• Create an nginx service with a conf that looks something like:

http {
  server {
  listen 80;
  location / {
    root /siteroot/;
  }
  location ^~ /.well-known/acme-challenge/ {
    autoindex on;
    root /siteroot/;
    }
  }
}

And you directory layout looks something like:

nginx-pod $ ls -aR /siteroot/
/siteroot/:
.            ..           .well-known  404.html     index.html

/siteroot/.well-known:
.               ..              acme-challenge

/siteroot/.well-known/acme-challenge:
.
..
EJJXvSUhbQIsvFwF8zBkt4-B_aoGiS79QsTOLOG1FaI
foo

• Make your default backend an nginx server

  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
  name: foo
  spec:
  backend:
    serviceName: nginx
    servicePort: 80
  rules:
  - http:
      paths:
      - backend:
          serviceName: yourSvc
          servicePort: 80
          path: /foo

• Point dns of yoursite.com to the public IP allocated to this ingress

• Run the normal letsencrypt flow (manually, i.e just https://gethttpsforfree.com/), you'll get a challenge to serve http://yoursite/.well-known/acme-challenge/foo (contet: bar)

• Put the file:

$ kubectl exec nginx-endpoint-pod -- bash -c "echo bar > /siteroot/.well-known/acme-challenge/foo"

A good first step would be to explore if we can create some sort of nginx/letsencrypt/lego helper pod that we can use in place of the given nginx backend, capable of automating some of this.

[bgrant0607]

Ref #12732, #18112

[bgrant0607]

Also ref #20439

[simonswine]

I've started working on this. My idea is a service pod that does all the communications with the ACME server and finally stores the certificates into kubernetes secrets. For now I think I will limit to one cert per ingress resource and simple monitoring the domains used within the rules and if they mismatch the domains specified in the TLS section, I will request a cert via ACME. Here is the repo for this service daemon: https://github.com/simonswine/kube-lego

[bprashanth]

@simonswine great! I'll leave this bug assigned to myself since I can't assign it to you, but keep off working on it assuming you are. Let me know if you need code review etc.

[simonswine]

@bprashanth thanks for offering your help on this. I've got a very basic (and stupid API polling) example working now, but I will polish it and write some docs how to set it up.

I've basically started implementing a reverse proxy (https://github.com/simonswine/kube-ingress-proxy) to consume the certificates and the ingress resource.

If by any chance you are in London the next two days, I am happy to show the working prototype.

[bprashanth]

@simonswine unfortunately I am not. But maybe you can give a 10-15m demo on one of the kubernetes community hangouts? (http://blog.kubernetes.io/2015/06/weekly-kubernetes-community-hangout.html). This is a topic the community at large will be interested in.

[0xmichalis]

cc: @deads2k

[goblain]

I have a quick and dirty caddy controller at the moment using SIGUSR reloads of fresh config generated on kube ingress changes - https://github.com/Goblain/caddycontroller
From what I see, Caddy can't really be used for automated TLS handling as per normal scenario as it would need some native clustering for LE enabled so that having multiple ingress caddies does not violate access to the URLs required during LE cert requests (obviously ingress controller runs in HA setup).
At the moment my plan is to use Caddy as the Ingress but handle certificates with a separate pod in k8s that will register ingress path for $DOMAIN/.well-known/acme-challenge/ so that the ingress controller passes the challenge request to a letsencrypt --standalone ...

[munnerz]

I've been working on a small microservice that'll monitor the apiserver for new ingress resources and automatically obtain a certificate if necessary (incl. keeping the certificates up to date).

It's available at https://github.com/munnerz/acme-secrets - it's still WIP and not production ready yet, although I am running it on my own personal cluster just fine without issue. Contributions and comments are highly appreciated!

The idea is to be ingress-controller agnostic (providing the ingress controller handles particular failure scenarios (eg. certificate secret not existing) gracefully and retries a few minutes later, as some already do)

[bprashanth]

@munnerz great! suggest opening a pr to get it into contrib. It doesn't need to be 100% ready (like a lot of the other projects in contrib) as long as we're clear about the limitations. This should get it in front of a larger audience and potential contributors. It will also force us to do code review, unittests, document etc :)

[munnerz]

Will do - I'm just making a few changes to clean up the code as well as write some info on how people can get it up and running (the user registration step is currently quite difficult as it requires the user to manually request it and store it).

I'll be submitting a PR in the next week. Is it the intention to have this functionality in the Kubernetes code in future?

[bprashanth]

I'll be submitting a PR in the next week. Is it the intention to have this functionality in the Kubernetes code in future?

We currently have this concept of "cluster-addons", which run as pods in a cluster and solve commonly encountered problems when running a distributed application. Examples include cluster DNS, monitoring and loadbalancing. Cluster addons are managed by the master and run in the kube-namespace, so there's a clear indication to the user that we think it's a good idea that you leverage this functionality. At the same time they're not "core" in the same way scheduling or in-cluster Service VIPs are.

I think a letsencrypt controller would make a really nice addon, either as a separate pod or as a sidecar with existing Ingress controllers. Suggest starting off with a pr in contrib/ingress (https://github.com/kubernetes/contrib/tree/master/ingress).

[simonswine]

I've just "released" 0.0.2 of kube-lego. I think it has the minimal needed features in for a release. See the README

I added documentation and examples and think the code is much more mature...

It would be great if you could give it a go in your cluster. I am keen to get some early feedback.

@bprashanth I am not really familiar with this community hangouts, but I want to create slides for a short demo anyhow, so it would not be a problem to give a demo in one of this hangouts. What's the process of getting a topic in one of the next hangouts?t

[bprashanth]

@simonswine great! I've added a bullet to https://docs.google.com/document/d/1VQDIAB0OqiSjIHI8AWMvSdceWhnz56jNpZrLs6o7NJY/

[bprashanth]

@simonswine don't have you gmail so I can't tag you on the doc, @sarahnovotny had a question about the date. Can you confirm? Thanks!

[wernight]

I hope it'll be available in the Nginx Ingress Controller. My understanding is that no matter the solution, the Ingress Controller should refresh its TLS sometimes.

[bprashanth]

@wernight I believe this already works with kube-lego, all that remains is clear documentation on the ingress docs on how to acquire and renew certs cross platform, and some form of e2e testing

[aledbf]

I hope it'll be available in the Nginx Ingress Controller

Already present:

• kube-lego provides the Let’s Encrypt integration (nothing to be done by in controller)
• if the a TLS secret is updated, the ingress controller will update the certificate and reload
  (this should not take more than 5 seconds after the secret update)

[MDrollette]

The docs for using kube-lego with the nginx ingress controller can be found here https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx#automated-certificate-management-with-kube-lego

[donpark]

Unless I'm mistaken, kube-lego say it's not ready for production use and I have not found a concrete timeline nor plan to bring it to production quality.

[wernight]

Doesn't provide one to consider it stable or bug-free enough to use it in production.

[phutchins]

I've written a service that does all of this and works quite nicely in my production environment. It is still a work in progress but it works.

https://github.com/phutchins/kubernetes-ssl-manager

[cmluciano]

Is this issue looking for an owner? @bprashanth

[bprashanth]

We need a volunteer to try the available options and add:

1. A working example here: https://github.com/kubernetes/ingress/tree/master/examples#tls
2. Preferably somem form of integration/e2e testing

1 is a great start, let me know if you need more info on how to go about doing 2.

[ixdy]

cc @spxtr

[phutchins]

I'd be happy to post my plans for my project on github and would love to hear any feedback from others using it...

[bprashanth]

It would be nice to streamline, we have 3 options at this point, kube-lego, Philip's ssl-manager and ACME/autocert golang libraries that're still under development (eg: https://github.com/google/acme, https://godoc.org/golang.org/x/crypto/acme/autocert).

Perhaps it's time for an incubator project? (in addition to an example demonstrating how to rotate certs using existing ingress controllers). If we decide to do that, we'll need some volunteers, and we should discuss it in the networking sig.

[cmluciano]

Another option could leverage something like Hashicorp Vault pki

[phutchins]

The way that my ssl-manager is built, it is the automation around other tools like certbot and what kube-lego and autocert look to be. Currently it runs the letsencrypt binary and I have plans to migrate that to using the certbotauto bin but it could really wrap any of these.

The missing pieces for me was something to sit in my cluster, find services (pods, deployments...) that needed certificates by label, use those labels to find out what cert to register, then push those certs to secrets that can the be picked up in a consistent way by an ingress controller.

I'd be glad to create an example and would definitely like to get some tests written. @bprashanth, is there a preferred or accepted way to do integration/e2e tests for things like this?

[bprashanth]

Currently we test cert upgrades in the ingress conformance suite: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/ingress_utils.go#L133, we just update the hostname in the test and make sure it reaches the lb (https://github.com/kubernetes/kubernetes/blob/master/test/e2e/ingress_utils.go#L149).

It sounds like your cert manager doesn't actually have an Ingress dependency, which brings up the question of how you acquire the cert, since you need to demonstrate ownership over the hostname. We should aim to tie in ingress/DNS/cert so that it just works, eg: I create an Ingress with some hostname, the ingress controller provisions an lb, the DNS controller assigns the records, and the letsencrypt controller acquires certs and manages rotation.

So I guess my question is, in this mode, how do you tell the ingress controller to proxy /well-known to your controller? I think that's the integration point we should really target in a test.

[bprashanth]

So I guess my question is, in this mode, how do you tell the ingress controller to proxy /well-known to your controller? I think that's the integration point we should really target in a test.

I believe kube-lego does this by inserting a rule, eg an ingress like

spec:
  rules:
  - host: echo.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: echoserver
          servicePort: 80

becomes

  spec:
    rules:
    - host: echo.example.com
      http:
        paths:
        - backend:
            serviceName: kube-lego-gce
            servicePort: 8080
          path: /.well-known/acme-challenge/*
        - backend:
            serviceName: echoserver
            servicePort: 80
          path: /

Though there are multiple ways to do this

1. as described in the first comment you can also add a rule as the default backend
2. you can create a second ingress that acquires /.well-known (this is currently dependent on the controller, the nginx ingress controller will "join" ingresses such that if it finds 2 with the same hostname and different paths, they mux onto a single IP, but the GCE controller will create 2 loadbalancers one per ingress. The joining problem is described in: Ingress claims  #30151)

Opinions welcome on if we need better ways to support this out of the box (maybe a standard default backend for all ingresses that want certs, that advertises /.well-known only when required, and advertises a user supplied 404 otherwise? thinking out loud)

[phutchins]

So before we go diving into what is there to solve the problem, maybe we can spec out how we would like the user experience to look, then the resulting actions in the backend service that interacts with letsencrypt and kubernetes to make it all happen (including cleaning up after itself if necessary).

I think it should look something like this...

Prerequisites:

• An ingress defined that directs all traffic for /.well-known/acme-challenge/ to the ssl manager service
• A backend service that...
  • Exposes a port
  • Has an appropriate service grouping label

User Experience:

Adding TLS

1. User adds a label or labels to a deployment or pod **(see below for discussion on this)

• To define the URL dnsUrl
• To define what port dns should point to
• To define what port dns should serve from
• To define if the certificate should be removed after the definition containing the label that it was created from goes away

2. An Ingress is created for the user with a default path (say /) that points to their service and specifies the TLS block with the appropriate secret name

Removing TLS

1. The user removes the pod/deployment
2. The ingress is removed
3. After X time (or when the cert is expired) the secret is removed

Backend:

(magic tag could be a tag or label)

1. Service watches for:
   a) changes on in the entire cluster for resources being given a magic tag indicating that they would like SSL/TLS generated for them
   b) changes where services no longer exist that had the magic tag
   c) changes where a resource no longer has the magic tag bug still exists
2. Service handles changes to resources

• magic tag added:
• magic tag removed:
• resource with magic tag removed:

3. Service automatically renews certs at a set interval or when needed? Needs to be efficient so we don't hit the rate imit.
4. Service saves the certificate as a secret named such that the ingress matches
5. Service Cleanup should clean up unused resources but should be very careful as this could be a potential place for errors that create service outages

Notes

*@bprashanth It would be nice to be able to define this once end not once for every ingress. Being able to define a catch all for anything trying to hit a certain path might be nice. Or maybe we just use labels to select everything with the SSL label we use to kick off fetching a LE cert.

**The other part where we could use some discussion is around where the label adding the dns name would go. It would have to specify a dns name and a port as pods/deployments could expose more than one port.

Integration w/ DNS Creation

Also, I'd like this to seamlessly work with whatever I'm using to automatically create a DNS entry in my cloud provider. It would be best if we didn't have to create a label or entry for both the DNS and SSL/TLS. Do we have a standard way of having DNS created? If so, we could use that and add an additional param that simply enables fetching SSL for the already specified address.

Thoughts?

[phutchins]

@bprashanth re kube-kego, looks like it does. That's fairly similar to what my ssl-manager does. I may give it a try and compare.

[simonswine]

@phutchins thanks for writing down the a user experience. I am the author of kube-lego and happy to join forces to improve the integration for Let's Encrypt.

Many users are quite happy with kube-lego, but a good start would be to develop your notes, and maybe create a proposal/design document. In the last release I switched to the golang.org/x/crypto/acme library with much better interfaces.

The DNS thing is quite tricky, my thinking was always as long kubernetes provides no DNS integrations I did not want to go down the route of supporting DNS validations. I think kube-cert-manager supports that.

My next priority would be to integrate E2E into the kubernetes repo.(I am now quite familiar, with e2e tests)

[luxas]

@phutchins @simonswine +100 for collaboration and making things as official as possible.
It might be worth considering having an official letsencrypt-kube-integration controller in kubernetes/ingress.

@bprashanth WDYT?

[bprashanth]

I'm still reading the propsal, some quick thoughts.

@simonswine has done a great job with kube-lego, as many members of the community will attest to. Collaboration seems like the right way forward, especially since Philip's proposal formalizes how normal Services can acquire certs.

Many users are quite happy with kube-lego, but a good start would be to develop your notes, and maybe create a proposal/design document. In the last release I switched to the golang.org/x/crypto/acme library with much better interfaces.

Heard tell that this will make it into the stdlib soon

The DNS thing is quite tricky, my thinking was always as long kubernetes provides no DNS integrations I did not want to go down the route of supporting DNS validations.

@justinsb has a promising proposal:
https://groups.google.com/forum/#!topic/kubernetes-dev/2wGQUB0fUuE

[phutchins]

Re kube-lego, it looks like it's a really good start and fairly close to what I imagine we'll land on in the design doc. I'd be more than happy to help contribute to kube-lego and retire my project if that makes the most sense.

@simonswine, would you mind providing some feedback on the design proposal and let us know if it is in line with a direction that fits your vision of kube-lego? I'll definitely be testing kube-lego and will give my feedback as well.

[bprashanth]

Let's restrict discussion to external -> internal https and letsencrypt certificates. I'm not talking about certs for intra cluster Services. We're still figuring out an api for "internal lb", or intra-cluster Service gateways.

The first question I'd like to tackle is: Do we really need to include Pods and Services in the design?

Adding TLS
User adds a label or labels to a deployment or pod **(see below for discussion on this)

I'd like to not expose this UX for pods. I think they're the wrong fundamental unit when we're talking about ingress TLS.

I'm on the fence about exposing it for Services. They're already a grab bag of functionality ripe for an api refactor (various different types, firewalls settings, headless/selectorless all in one object).

The big problem with squirrelling annotations in Services is that we will break api compat unless we're very careful, and when we do, a user's website will break on cluster upgrade in the most frustrating way. It won't break when they upgrade the master or nodes, it probably won't break when they upgrade the LE Service, it'll break after they've done all that, when certs expire.

Moreover, annotations are meant for experimentation. They usually don't have e2es, unittests or show up in swagger docs. They're hard to access via kubectl -o jsonpath, and often don't have validation. They're inferior in almost every way if we know what we want and already have an api object to express it.

I think the right way to do this is through an Ingress, which already has a hostname and a pointer to a secret. We should start simple and leave room for expansion. If the user doesn't want any of the other loadbalancing features of Ingress (eg they just want a Cert for a Service of Type=LoadBalancer), they'd create an ingress with ingress.class=kube-lego pointing to a single Service.

Service watches for:
a) changes on in the entire cluster for resources being given a magic tag indicating that they would like SSL/TLS generated for them
b) changes where services no longer exist that had the magic tag
c) changes where a resource no longer has the magic tag bug still exists

This puts a heavy burden on the controller, because it needs to watch different resources, the user, because they need to pay attention to manual cross resource validation, and the developer who needs to write test suites that check Service annotations that essentially express most of what's available in the ingress. It would be great if this was just "Service watches Ingresses (and maybe secrets, which are mostly static)". Vastly simplifies things.

Shoot holes through this?

[phutchins]

@bprashanth I agree that we should keep the scope to external -> internal https and lets encrypt certs. That being said, I always like to consider how a feature or addition would/could interact or change user experience with current or future features/changes. While DNS is not and should not be a part of this service, it might be helpful to at least keep in mind how the two might play together...

The first question I'd like to tackle is: Do we really need to include Pods and Services in the design?

Connecting the two at the Ingress makes perfect sense to me. Its simple, clear and would group the most similar tasks into one place. If you're already setting the tls.secretName in spec in the Ingress, it would be quite simple to add a tls.tlsProvider or something along those lines.

I'd like to clarify how you imagine this working with other ingress classes. For example, I use gce as my ingress controller and would very much like to stay as close as possible to my current workflow when adding a service to control fetching my certs.

I still want all of the same things to happen, but would want the cert fetching service to grab the required cert, using the dns name already specified in the ingress, and upload it, using the secret name already specified in the ingress, then by default my gce ingress controller would check for the existence of that secret at an interval until it finds it and continue to do its thing.

I imagine it looking something like this...

{
  "apiVersion": "extensions/v1beta1",
  "kind": "Ingress",
  "metadata": {
    "name": "my-service",
    "namespace": "staging"
  },
  "spec": {
    "tls": [
      {
        "secretName": "cert.my-service.staging.domain.com",
        "certManager": "kube-lego"
      }
    ],
    "rules": [
      {
        "http": {
          "paths": [
            {
              "path": "/*",
              "backend": {
                "serviceName": "my-service",
                "servicePort": 80
              }
            }
          ]
        }
      }
    ]
  }
}

Currently, I have an additional path...

            {
              "path": "/.well-known/acme-challenge/*",
              "backend": {
                "serviceName": "ssl-manager",
                "servicePort": 80
              }
            }

...which is how I forward the validation request to the ssl-manager. This should simply happen behind the scenes but I'm not sure if/how we could make this happen without making a separate ingress class but then we end up with two ingress configs required if we want to continue to use our existing class.

I'm not familiar enough with the internal architecture around this yet to determine if it would be possible or make sense to do it in one. If all of the data in the spec is routed directly to the selected ingress controller, (i.e. gce) then this may not work.

This puts a heavy burden on the controller, because it needs to watch different resources, the user, because they need to pay attention to manual cross resource validation, and the developer who needs to write test suites that check Service annotations that essentially express most of what's available in the ingress. It would be great if this was just "Service watches Ingresses (and maybe secrets, which are mostly static)". Vastly simplifies things.

Sounds great to me to simplify it like this.

[nambrot]

There hasn't been movement on this issue, but I just wanted to ask quickly whether this is something we can expect as kubernetes end users to be built into kubernetes soon, or rely on excellent solutions such as https://github.com/jetstack/kube-lego for the foreseeable future?

[InAnimaTe]

@nambrot I doubt soon. I'm working with @simonswine and others to put more development work into kube-lego to reach better stability and features. While I'd love to see this ability built-in, kube-lego does a pretty wonderful job now and isn't too hard to get going.

[kfox1111]

if its built in, then each ingress controller would be required to implement it themselves? I kind of like the idea of sharing the implementation.

[thockin]

This should definitely be in a controller of its own, such as kube-cert-manager.