Autocert for updating the secret that has the certificates

[vikramkhatri]

Hi - The simplicity of your implementation is indeed very impressive.

I have a use case. My Istio ingress certificates are stored in a secret and then the Ingress pod consumes it from the secret and then pushes it to the memory of the target service. For example:

kubectl -n istio-system create secret generic httpbin-keys --from-file=key=$HOME/step/httpbin.key --from-file=cert=$HOME/step/httpbin.crt --from-file=cacert=$HOME/step/ca-chain.crt

Now, the above will be consumed by the Istio Ingress gateway automatically since I have defined a gateway that defines the secret name httpbin-keys for the httpbin.istio.io - which resolves to a local IP address - Just for testing. For example:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: mygateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
  - port:
      number: 443
      name: bookinfo
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: bookinfo-keys
    hosts:
    - bookinfo.istio.io
  - port:
      number: 443
      name: httpbin
      protocol: HTTPS
    tls:
      mode: MUTUAL
      credentialName: httpbin-keys
    hosts:
    - httpbin.istio.io

Since the secret is in istio-system namespace, it is protected as the users will not have access to this namespace and the certificates are then loaded in the memory of the pod instead of mounting them as files for security reasons through Secret Discovery Service of Istio.

Is there a way, you could recommend as what is the best way to automatically update the secret based upon when TTL of the x.509 is about the expire.

If this is easy to do, it will solve one of the problem that I am facing is to automatically renew the certificates for the external names for the virtual services.

Thank you.

[mmalone]

Hey @vikramkhatri, sorry for the delayed response.

Short answer... If you're running autocert, and assuming the istio ingress will notice when the secret is updated, the easiest solution here is probably to simply launch a container that gets a certificate from autocert and runs a job to periodically put it in the kubernetes resource for your ingress controller.

The podspec would look something like:

apiVersion: apps/v1
kind: Deployment
metadata: {name: ingress-cert-manager, labels: {app: ingress-cert-manager}}
spec:
  replicas: 1
  selector: {matchLabels: {app: ingress-cert-manager}}
  template:
    metadata:
      annotations:
        autocert.step.sm/name: bookinfo.istio.io
      labels: {app: ingress-cert-manager}
    spec:
      containers:
      - name: ingress-cert-manager
        image: ingress-cert-manager

With the appropriate cluster permissions, the ingress-cert-manager container could just run:

kubectl -n istio-system create secret generic httpbin-keys --from-file=key=/var/run/autocert.step.sm/site.key --from-file=cert=/var/run/autocert.step.sm/site.crt --from-file=cacert=/var/run/autocert.step.sm/root.crt

in a loop, sleeping for an hour or so in between invocations.

Longer answer...

Autocert was intentionally designed to keep private keys out of kubernetes secrets, instead persisting them only on the local file system inside pods/containers, because it's more secure (fun fact: most kubernetes distros don't encrypt secrets in the storage backend!). Often there's a security/usability trade-off. But there isn't one here. Storing certs on disk offers the same developer/operator experience because, ultimately, containers just need to find certs & keys on disk. Autocert does that without ever moving a private key over the network, which is a very good thing.

That said, we're aware that certain kubernetes primitives (like ingress controllers) require private keys (and certificates) to be made available via a kubernetes API resource. For those use cases we have no choice. We might add functionality to autocert to support this use case, but we're not currently working on that.

There are other ways to make this work though. When you install autocert you get a vanilla step-ca installation running on your cluster. (You can also install step-ca on kubernetes separately from autocert.) You can use step to work directly with step-ca, create a private key & certificate, and stick it in a kubernetes resource yourself. It sounds like that might be what you're doing already..? Monitoring that resource using step would be pretty straightforward. You could use a command like:

step certificate inspect mike.crt --format json | jq .validity.end

in a script to pull the expiration date, renew using step ca renew, and then update the relevant kubernetes resource. You could run this command in a pod if it had the appropriate cluster permissions. Alternatively, you could create a container that mounts the certificate & key pair and just runs:

step ca renew --daemon --exec "./resource-update-script.sh" srv.crt srv.key

where ./resource-update-script.sh is a little script that updates the cert / key resource.

You might also want to take a look at cert-manager for this. By default, cert-manager gets Web PKI certificates from Let's Encrypt. But we're working on an integration that will let cert-manager manage certificates issued by step-ca. If I'm understanding your use case correctly, once it's ready, this integration will do exactly what you want here without any custom integration on your party.

Finally, we're working on a direct integration between envoy and step-ca using their SDS protocol. It should be launched in the next month or so. We could give you early access if you think it'd be useful, but I don't think it's what you need here.

[servicemeshbook]

Finally, we're working on a direct integration between envoy and step-ca using their SDS protocol. It should be launched in the next month or so. We could give you early access if you think it'd be useful, but I don't think it's what you need here.

Hi @mmalone - actually that will be very interesting. Please let me have access and I will be willing to test this in my Istio set up. Istio's SDS is pretty good as cert and key are not mounted and they remain in memory - I think a better implementation from security standpoint. Thanks for step. This is awesome.

[servicemeshbook]

One suggestion - Linkerd does service to service communication mTLS automatically and they have their own CA that does this. It does not secure traffic between an Ingress gateway to the edge service as that is specific to a type of ingress controller one is using. Linkerd leaves that function to the end-users. Istio provides its own Ingress and Egress gateways but Istio does not cover certificate rotations of its own Ingress / Egress gateways either. It good be nice if you could fill that gap through your CA for both Istio and Linkerd.

[sourishkrout]

Finally, we're working on a direct integration between envoy and step-ca using their SDS protocol. It should be launched in the next month or so. We could give you early access if you think it'd be useful, but I don't think it's what you need here.

Hi @mmalone - actually that will be very interesting. Please let me have access and I will be willing to test this in my Istio set up. Istio's SDS is pretty good as cert and key are not mounted and they remain in memory - I think a better implementation from security standpoint. Thanks for step. This is awesome.

@vikramkhatri @servicemeshbook as Mike Malone mentioned for early access please see https://github.com/smallstep/step-sds we just made public. It's a direct integration between step-ca and envoy using the Secret Discovery Service protocol. Please let us know how it goes!

[sourishkrout]

Closing. Feel free to reopen if you have more questions.