Add ca.crt to TLS secret generated by ACME issuers

[ajpauwels]

Is your feature request related to a problem? Please describe.
The ca.crt field is added to the TLS secret generated by CA issuers, but is null for ACME issuers.

Describe the solution you'd like
The Issuer uses the Link header returned by the ACME protocol as defined by the ACME standard to fetch the issuer's cert and places it inside the TLS secret under the ca.crt key.

ACME standard
The Link header is mentioned in Section 6.5 Certificate Issuance of the ACME standard draft. Refer to the last paragraph on page 30 here: https://tools.ietf.org/html/draft-ietf-acme-acme-02

Environment details (if applicable):

• Kubernetes version (e.g. v1.10.2): v1.12
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): AKS
• cert-manager version (e.g. v0.4.0): v0.7.0
• Install method (e.g. helm or static manifests): static

/kind feature

[munnerz]

If this is something we are able to obtain from the ACME server (and it sounds like that is possible), then a +1 from me on this! 😄

If anyone is interested in putting together a PR for this, it'd be greatly appreciated 😄 it may require some changes to our ACME client to expose enough information for us to determine what to use. We'll also need to somehow 'passback' this information via the Order resource. Perhaps having a new field, caBytes (or similar) that sits alongside the existing order.status.certificate field?

[rolfdalhaug]

Hi

I'm not sure if this is related to an issue we're experiencing but would welcome some insight. We've implemented a ACME Issuer which has created a certificate which all checks out using SSL Server Test at SSL Labs. We can connect with SSL for browser connections etc. The issue we're having is when trying a connection with node (specifically Realm.io Data Adapter) we're getting a "The server certificate was not signed by any root certificate" "Verify return code: 21 (unable to verify the first certificate)". We're new to kubernetes / ingess SSL but doing some reading this seems to be an issue with the ca.crt?.. is this correct?.. On checking the secret we do have a 'null' ca.crt.

Are you able to advise on what the issue is / how to resolve?..

Thanks

[cpu]

If this is something we are able to obtain from the ACME server (and it sounds like that is possible)

If the OP means the root certificate and not an intermediate it isn't possible. ACME makes no provisions for communicating roots: these are meant to be present on the relying party systems out of band.

The Link header is mentioned in Section 6.5 Certificate Issuance of the ACME standard draft. Refer to the last paragraph on page 30 here: https://tools.ietf.org/html/draft-ietf-acme-acme-02

Note that draft-02 is extremely old and doesn't match up with Let's Encrypt's ACME v1 API or ACME v2. Also the Link headers were for following from an end entity certificate to the intermediate(s) that issued it, not the root. The full chain (minus the root) is always present in the response to a POST-as-GET for the order certificate URL in RFC 8555.

[ajpauwels]

Sorry for the late follow-up on this but @cpu is totally right. The tls.crt field contains the entire chain, from the leaf cert to the intermediate cert (not the root cert, but all I need is an intermediate cert). That chain is all I needed to complete TLS verification. If you require splitting off the leaf cert from the intermediate cert, here's a short snippet to alias that can do that on most systems:
csplit -f tls- -b "%02d.crt.pem" -s -z "$1" '/-----BEGIN CERTIFICATE-----/' '{1}'

Closing as the intermediate certs are already provided and root certs cannot be retrieved according to the standard.

[davgordo]

I respectfully disagree with the conclusion of this issue. It's well understood that the ACME spec itself does not explicitly support the retrieval of an issuer's root CA certificate. However the ca.crt key in certificate secrets doesn't necessarily need to represent the top-level root. It just needs to represent the issuing authority, which could be an intermediate.

Clearly the issuing intermediate can be derived [1] [2]. But these tricks represent an operational burden. Cert-manager could at least optionally support automatic population of the ca.crt key in the case of ACME issuers, based on the intermediate.

In my opinion the most important benefit of populating ca.crt for ACME issuers is that it provides a consistency (to the secret consumer) regardless of the type of issuer. There are use cases for consuming the ca.crt key (clearly, or else it wouldn't exist). The issuer type is purely irrelevant to the validity of those use cases.

Please consider re-opening this issue, or if it is more appropriate to create a new issue, I'm happy to do that.

[1] #1571 (comment)
[2] goharbor/harbor-helm#261 (comment)

[wichert]

For what it's worth having an empty ca.txt key makes it needless difficult to use ejabberd with cert-manager managed certificates: ejabberd will not start if ca.crt is present but invalid.

[aranair]

Also, not un-related, the empty field that ends up in the k8s secret ca.crt: null also prevents the k8s secret from being modified/copied to any other namespaces (because it fails validation).

[XanderAtBackboneSystems]

Also related, using istio (1.4.2) with cert-manager and ingress-sds will not work due to the missing ca.crt with the following log line:

failed load server cert/key pair from secret <SECRET-NAME>: server cert or private key is empty

[jamestutton]

Also related, using opendistro-es does not work with cert-manager le certs as it expects the cert, key and ca to all exist under a single secret. It is also not possible to use for example /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem as it expects the ca file to exist within its config path. If the LE ca itself can not be included could not a copy of normal OS level ca bundles not be includable perhaps via a flag? Or even just a flag so we can manually set the ca.crt contents and cert-manager will include this in the secret as ca.crt

[rectacoda]

+1 here. We can't use our cert-manager ACME Issuer with Couchbase Autonomous Operator. The DAC need a secret containing a ca.crt. And for security reasons, storing our private key in a k8s secret for a CA Issuer is not an option. Please consider reopening this issue.

[munnerz]

This isn't something that is going to be resolved in cert-manager for the reasons already discussed above - if you are looking for a project/controller to help with CA certificate/trust distribution, take a look at cert-manager/trust instead 😄

[pschichtel]

@munnerz I'm a little late to the party, but as the problem still persists:

While cert-manager/trust seems neat, it's not the solution to the problem described here (as shown by the reactions to your last comment). The problem with ACME certificates is that the resulting secrets don't meet the expectation of many applications that tls.key, tls.crt and ca.crt exist. For example I've been deploying bitnami's openldap container and I just now wanted to enable TLS, but that container requires a ca.crt. Sure I could customize the container to split the tls.crt as suggested in #1571 (comment), but I'd need to do that for every single container that requires a ca.crt.

So what's being asked for, if I understand people correctly, is just some way to have cert-manager split the tls.crt into tls.crt and ca.crt for me, so I can immediately use the certificate. I could imagine this as a flag on the issuer.

[Jiawei0227]

+1 on open this for more discussion. The biggest problem is a lot of software assumes a secret contains tls.key tls.crt and ca.crt for usage. And the fact that ACME issuers generates secret without ca.crt is making software hard to consume using automation.

[Sicaine]

Facing the same issue now with opensearch.

Sounds ridicoulous annoying to me :-(

We can now either create a secret syncer, which will take the secret and add ca.crt or patch it into certmanager. I have no clue why this problem is so common but still no real solution is provided.

[wallrj]

@pschichtel If you are using a Let's Encrypt certificate for your openldap server, try configuring openldap to use the Let's Encrypt root cert that is included in the container image:

LDAP_TLS_CA_FILE: /etc/ssl/certs/ISRG_Root_X1.pem # CA file

# OR

LDAP_TLS_CA_FILE: /etc/ssl/certs/ca-certificates.crt # CA bundle

• https://hub.docker.com/r/bitnami/openldap/#Securing%20OpenLDAP%20traffic

$ docker run --rm bitnami/openldap:latest cat /etc/ssl/certs/ISRG_Root_X1.pem | openssl x509  -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            82:10:cf:b0:d2:40:e3:59:44:63:e0:bb:63:82:8b:00
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Validity
            Not Before: Jun  4 11:04:38 2015 GMT
            Not After : Jun  4 11:04:38 2035 GMT
        Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1

I can't tell how openldap uses that CA certificate / bundle. The documentation suggests that it is used to verify the signatures of client TLS certificates:

Typically a single CA will have issued the server certificate and all of the trusted client certificates, so the server only needs to trust that one signing CA.
-- https://www.openldap.org/doc/admin24/tls.html#TLS%20Configuration

But that seems inappropriate when the certificate is issued by Let's Encrypt, since it would allow any client with a Let's Encrypt signed client certificate to connect to the server.

[michal-rybinski]

Was just researching this issue and came across this discussion thread.
The inclusion of ca.crt is also a requirement of Istio Gateway to be able to configure it to work in mTLS mode:

https://istio.io/latest/docs/tasks/traffic-management/ingress/secure-ingress/#configure-a-mutual-tls-ingress-gateway

"The server uses the CA certificate to verify its clients, and we must use the key ca.crt to hold the CA certificate."

I am using cert-manager along with istio-csr to issue certificates within istio using internal ACME services.
Due to the lack of ca.crt in secret generated by cert-manager, I am able to only use simple TLS mode and not mTLS what is kind of bummer as mTLS is in the core of Istio's philosophy.

Theoretically it could be solved by introducing some additional field/flag in cert-manager's "Certificate" resource (https://cert-manager.io/docs/concepts/certificate/), to manually provide contents of ca.crt that will be included in secret with certificate but it seems much cleaner and easier to include it as it was discussed above.

[michal-rybinski]

ha, just after posting I realized that what I am asking about is already in progress and was linked above... cross linked it to here.

[wallrj]

The inclusion of ca.crt is also a requirement of Istio Gateway to be able to configure it to work in mTLS mode

@michal-rybinski The Istio Gateway API docs say:

For mutual TLS, cacert: and crl: can be provided in the same secret or a separate secret named -cacert.
https://istio.io/latest/docs/reference/config/networking/gateway/#ServerTLSSettings

So can you create a separate Secret called <credentialName>-cacert with a file called cacert, containing the CA certificate that you sign your client certificates with?

Afterall, the client certificates need not (should not?) be signed by the same CA as signed the Gateway server certificate.
Which seems to be the motivation for supporting this mode:

• Unable to use different CAs for server / client ingress mTLS istio/istio#29856

[wallrj]

/reopen

Because I think the documentation should at least explain why the ACME issuer doesn't (won't?) create ca.crt file and explain what the preferred alternatives are.

[jetstack-bot]

@wallrj: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[michal-rybinski]

@michal-rybinski The Istio Gateway API docs say:

For mutual TLS, cacert: and crl: can be provided in the same secret or a separate secret named -cacert.
https://istio.io/latest/docs/reference/config/networking/gateway/#ServerTLSSettings

Huh... I cannot believe I missed that.... thank you very much for pointing that out! I'll give it a go and test.

But, regarding the problem this issue is about, I suppose good docs around the reasoning behind absence of such option would be something that would prevent such requests in the future.
Thanks!

[Pela2silveira]

Hi there. I am issuing the same problem with oci native ingress controller (oracle K8s engine - oke). Load balancer expect ca.crt in TLS secret. So the process fails an none tls listener is configured.

[vkumar85]

I am also facing the same problem with securityAdmin in opensearch deployment where it won't work because of missing ca.crt file in the secret which that job uses.

[linhng98]

I'm having same problem with emqx cluster, to enable ssl/tls feature we need ca.crt field

[Elyytscha]

having the same issue with bank vault :(

[shayktrust]

Any updates? I'm encountering the same issue with RabbitMQ:
Warning FailedMount 23s (x25 over 35m) kubelet MountVolume.SetUp failed for volume "certs" : references non-existent secret key: ca.crt

[shayktrust]

Btw, you can use letsencrypt like the following:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-cluster-issuer
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: shay@xxxx.io
    privateKeySecretRef:
      name: letsencrypt-cluster-issuer-key
    solvers:
    - dns01:
        route53:
          region: eu-north-1
          hostedZoneID: xxxx

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: rabbitmq-tls
  namespace: rabbitmq
spec:
  secretName: rabbitmq-tls
  issuerRef:
    name: letsencrypt-cluster-issuer
    kind: ClusterIssuer
  dnsNames:
  - mq.stage.xxxx.io

and in the rabbitMQ to set the following block:

tls:
            enabled: true
            existingSecret: "rabbitmq-tls"
            existingSecretFullChain: true

existingSecretFullChain: Existing secret with certificate content be mounted instead of the ca.crt coming from caCertificate or existingSecret/existingSecretFullChain.

[alextricity25]

I also think it would be nice to have this behavior implemented.

I'm trying to configure mTLS between an ingress-nginx ingress controller and another client running on a K8s cluster. I would like cert-manager to manage these certs, but because the ca.crt key is not present on the secrets that cert-manager generates, I can't simply reference the secret in my ingress-nginx configurations. Although ingress-nginx does seem to support retrieving the ca from a seperate, pre-made secret, it would be extravagant if cert-manager could split out the CA chain and leaf cert into two keys, ca.crt and tls.crt.