Allowing alternative Secret output formats (e.g. single .pem file priv/cert output)

[julien-hedoux]

/kind feature

When we describe the generated secret, we can see tls.key and tls.crt. But if we want to get the pem output, we have to concat both to obtain it.

The problem : If a container need a secret field (PEM file) as a mounted volume, we can't mount it from the secret api.
Could be great if the CertManager generate the PEM and store it as a key to the secret.

Now we have this :

yaml
volumes:
        - name: volume-site-tls
          secret:
            secretName: site-secret-tls
            items:
              - key: tls.key
                path: certificate.key
              - key: tls.crt
                path: certificate.crt 

Could be great to do this :

yaml
volumes:
        - name: volume-site-tls
          secret:
            secretName: site-secret-tls
            items:
              - key: certiticate.pem
                path: certificate.pem

[munnerz]

Thanks for the request.

There have been similar requests for additional fields or alternative field names to be provided in the resulting secret resource.

My main gripe here, is that this creates extra complexity, and thus confusion for users, in how we should behave if a certificate resource is 'out-of-sync', i.e. if the secret is missing a certificate.pem key (because a user has migrated from an older version, or are supplying their own certificates during a migration).

There's numerous edge-cases to handle, and I'd like to know that you problem can't be otherwise solved before we introduce the complexity 😄.

Could you describe your use-case a bit further? For something the likes of Apache2, which does not (as far as I know) support having separate key and cert files, I see the following:

• You want to easily configure Apache2, which requires a concated pem file
• You want certificates to be rotated automatically when they change on disk/when they are renewed

In order to trigger Apache2 to actually reload the cert pem, you will need to provide some glue that sends a SIGHUP when the file changes.

If you have to create/configure this glue already, an additional piece that observes either the key or pem changing on disk and automatically calls cat tls.key tls.crt > certificate.pem is fairly trivial to add.

This ultimately achieves what you are looking for, and will still allowing inotify events etc to be fired, meaning you can easily trigger reloads in reaction to these files changing on disk 😄.

Perhaps your use case cannot be solved with this kind of solution? It'd be great to hear more 😄

[julien-hedoux]

Hi.

First, thank you for your reply.
Actually, it's not a piece of cake. I understand why you ask it before.

• From Apache/Nginx or other, you have to use key and cert. So you are right and we have all data we need.
• But for other, for a TLS proxy like Hitch, we need to get the PEM file (https://github.com/varnish/hitch/blob/master/docs/certificates.md) in the config.

Here, I would like to fetch pem from the Let's encrypt Issuer, but I didn't found it and how to get it.
So, as I usually do, and you told it, I concat key et cert to obtain the pem file. Without the CA cert, be I should do if I will have.

As I can't store it into the container into the dockerfile,

• I have to pass key et cert files as mounted volumes. Then
• When the container is starting, I concat them into a PEM file in a emptyDir volume
• I use it as PEM file in my Hitch configuration.

Here my actual config :
The container part

- name: wp-julienhedoux-fr-hitch
        image: gcr.io/poc-docker-87/hitch:1.4.6-15
        imagePullPolicy: Always
        volumeMounts:
          - mountPath: /ssl
            name: wp-julienhedoux-fr-pem
          - mountPath: /secrets
            name: wp-julienhedoux-fr-tls
        ports:
        - containerPort: 443

The volume part :

volumes:
      - name: wp-julienhedoux-fr-pem
        emptyDir: {}
      - name: wp-julienhedoux-fr-tls
        secret:
          secretName: wp-julienhedoux-fr-tls
          items:
          - key: tls.key
            path: tls.key
          - key: tls.crt
            path: tls.crt

Then In the docker entrypoint script, I do this :

cat /secrets/tls.key /secrets/tls.crt > /ssl/certificate.pem

And use finally the /ssl/certificate.pem as PEM into my hitch configuration.
I the cert manager was added the pem content into the k8s secret, as tls.pem for exemple. A simple mount will share the PEM from secret to the container. Instead of regenerated it ;)

An other way I tried, was to patch the secret by adding the pem into the secret. Like this :

kubectl patch secret wp-julienhedoux-fr-tls --patch "$(cat pem.yaml)"

With a yaml like this

data:
  tls.pem: _PEM_STRING_

but here, I got an error about base 64 encode string too long

Do you have a better way?

[ksemaev]

+1 I have to do exactly the same for my Haproxy, which is starting to be default ingress in Kuber, and require .pem format as well

[alexanderilyin]

Without it, it's a huge pain in the ass to make HAProxy (and all other software which needs a single pem file as a certificate) work with certificates generated by cert-manager.

[white-hat]

+1 it's a blocker for me too

[claudiocleberson]

Guys that would a great addition.
I have a Ejabbed cluster that uses a .pem key. I will probably have to implement a container to server the cert.pem. But I'd rather prefer to user the Cert-manager which is already serving a cert to my Traefik proxy backend.

[retest-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[kbruner]

Maybe this could instead be implemented by allowing a "post-processing" step before the secret is created, if that's somehow simpler than specifying alternative output formats?

[retest-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[retest-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

[moritz31]

/remove-lifecycle rotten

We have a similar need for getting .pem files as output. Our use case is, that we are running opendistro for elasticsearch in a kubernetes cluster and do not want to build the pki ourself.
Since opendistro requires .pem files, we need to build a big chain on top of cert-manager, to convert the certificates to .pem format. That could be more easy if there is an option to output a .pem file instead of an .crt/.key file.

[munnerz]

This is something we'd like to support in cert-manager, but we're uncertain on how it should best be done.

If anyone has any insight on what we could do here, a few things at the front-of-mind:

• How does this play into alternate certificate formats? i.e. JWKs, DER encoded certs
• This ticket is actually requesting support for concatenating the two files we already produce, which is also slightly different once again.

Should these fields be related? How can we represent this in our API?

/lifecycle frozen

[bhagyas]

This is an issue when installing applications with built-in certificate management (eg. elasticsearch and xpack security).

Is there any known workaround to continue using cert-manager in such instances?

[jroper]

How does this play into alternate certificate formats? i.e. JWKs, DER encoded certs

There are multiple axes of encodings standards, there's PEM and DER, and there's PKCS*. So you can have PEM encoded PKCS1, DER encoded PKCS8, PEM encoded PKCS8 etc. PKCS* is more commonly referred to as the format (in some places its called the syntax), while PEM and DER are the ascii and binary encodings of whichever format you've chosen, and so are more commonly referred to as the encoding. That said, openssl just calls them all formats. It's really a mess.

If it were me, I this is what I would do:

• Rename keyEncoding to keyFormat.
• Create a new config, keyEncoding, to specify if the key should be PEM or DER encoded.
• For backwards compatibility, if keyFormat is not set, support PKCS8 or PKCS1 in the keyEncoding config, and treat that as the key format.

JVM apps, with the out of the box JDK crypto libraries, can only read keys in either PKCS8 DER encoding, or PKCS12 key stores (there's also JKS but let's just pretend that doesn't exist). PKCS12 key stores is in some cases more convenient, a PKCS12 file contains both the key and the cert chain, and you can make many of the standard network APIs in Java use your key store by setting a single system property. However, there are many cases where a PKCS8 DER encoded private key is necessary, for example, the postgres jdbc driver requires a PKCS8 DER encoded private key. So, I think it would be useful to add PKCS12 support. Note that if PKCS12 were used, you would probably call the output file keystore.p12, and have no other files, apart from perhaps the CA certificate.

[bergerx]

Along with #586, we keep getting hit by this. We are not comfortable about creating a PR since our knowledge about these is limited and we keep getting lost in the terminology used by openssl and java tools, i can see some of the reasons in @jroper's above post. I guess many others are in a similar status, so for so long we've not been getting a PR this :(

[kaceo]

Adding my support to the PEM format here. I am using HAProxy in docker (edge server outside cluster) and all the PEMs are in one directory. If I use Cert-manager I get a list of individual secrets, which I would need to copy individually, reload the HAproxy config, every time there is certificate events. I am too new to Kubernetes to know how to do this, perhaps if someone has sample code to show the strategy.

[fiberoptic89]

I am also requesting .PEM format support. I am using dovecot inside a docker image that runs inside a k8 pod.

[meyskens]

now we have certificate.spec.keystores we can start thinking about adding these under there

[meyskens]

We're taking on this for the v1.1 release, pkcs#12 and jks support already solves many of the usecases described above. Similar to this it would be nice to add "single PEM" support in something like:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: crt
spec:
  secretName: crt-secret
  dnsNames:
  - foo.example.com
  - bar.example.com
  issuerRef:
    name: letsencrypt-prod
  keystores:
    pkcs12:
      create: true
    cobinedPEM:
      create: true

using the logic of the already present alternate keyStores.

[GrandChaman]

Would it be possible to add support for DER encoded certificates and keys, to support postgres jbdc for mutual TLS ?

[meyskens]

@GrandChaman hey can you file a separate issue for that? that will make it easy to track and assign this!

[2tim]

Adding an additional use case. The spec for the webhookclientconfig takes a PEM encoded bundle. If the secret produced by cert-manager produced a pem file it would drastically reduce the overhead for the setup/maintenance for the ValidatingWebhookConfiguration for the nginx-ingress controller.

[rajivml]

We also have this exact similar use case for MongoDB deployments, Mongo expects certs to be in pem format , so we are converting the certs generated by cert-manager into pem format but as a result when ever cert-manager rotates certs mongo can't consume the same directly and even if I write a k8's cron job it won't be full proof because there will be some time delay on when certs are renewed vs when cron triggers and there may be some intermediate downtime

[rajivml]

@meyskens the pem creation, is this being picked in any of the upcoming releases ?

[javierguzman]

Another one seeking this

[maelvls]

Maybe we could have a CSI driver that would do exactly that? I was imagining something like:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: my-frontend
          image: busybox
          volumeMounts:
          - mountPath: "/tls"
            name: tls
          command: [ "sleep", "1000000" ]
      volumes:
        - name: tls
          csi:
            driver: pem-merger.csi.cert-manager.io
            volumeAttributes:
              pem-merger.csi.cert-manager.io/secret: my-tls-secret

The CSI driver would read the TLS Secret, and write a tls.pem file that contains the content of tls.key and tls.crt concatenated:

-----BEGIN RSA PRIVATE KEY-----
# Content of tls.key
MIIEpAIBAAKCAQEAz5DYA7iEBFq/SrCOTsjiYSHlHbTUdLyzselos5cE2++Huon3
tSK2ayFX1wQ3PuEmewAogy/20tWo80cr556AXA62Utl2PzLK30Db8w==
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
# Content of tls.crt
MIID4DCCAsigAwIBAgIJAJzTROInmDkQMA0GCSqGSIb3DQEBCwUAMFMxCzAJBgNV
kQ7ALfUfUh/RUpCV4uI6sEI3NDX2YqQbOtsBD/hNaL1F85FA
-----END CERTIFICATE-----

The CSI driver would also have to watch the Secret and reload the tls.pem when the Secret gets rotated.

[SpectralHiss]

A CSI driver seems like a good idea to add this feature for those who need it, and could even be a generic secret transformer driver containing different templates for other kinds of transformations!

[maelvls]

Woah, CSI drivers seem quite involved. I read through cert-manager's csi-driver and csi-driver-hostpath and I felt overwhelmed.

Maybe the "Secret controller" road is easier? I propose using a tiny controller secret-transform that takes the tls.key and tls.crt and concatenates them into a new field tls.pem using an annotation:

apiVersion: v1
kind: Secret
metadata:
  annotations:
    cert-manager.io/secret-transform: tls.pem
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FU...CBDRVJUSUZJQ0FURS0tLS0tCg==
  tls.key: LS0tLS1CRUdJToCi0tLS0tRU5EIF...SBQUklWQVRFIEtFWS0tLS0tCg==

The new data key will be created with the name tls.pem and the value
contains the key and certificate concatenated:

apiVersion: v1
kind: Secret
metadata:
  annotations:
    cert-manager.io/secret-transform: tls.pem
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FU...CBDRVJUSUZJQ0FURS0tLS0tCg==
  tls.key: LS0tLS1CRUdJToCi0tLS0tRU5EIF...SBQUklWQVRFIEtFWS0tLS0tCg==
  tls.pem: LS0tLS1CRUdJTiBSUXc0ZHk3NTNl...kQgQ0VSVElGSUNBVEUtLS0tLQo= # ✨

What do you think?

(Please try out https://github.com/maelvls/secret-transform so that we can decide whether it is a good approach or not)

[maelvls]

@bhagyas @moritz31

We are running opendistro for elasticsearch in a kubernetes cluster and do not want to build the pki ourself. Since opendistro requires .pem files, we need to build a big chain on top of cert-manager, to convert the certificates to .pem format. That could be more easy if there is an option to output a .pem file instead of an .crt/.key file.

and

This is an issue when installing applications with built-in certificate management (eg. elasticsearch and xpack security).

It seems like OpenDistro for Elasticsearch (or "OpenSearch" nowadays) can be configured using a PKCS#8 PEM-encoded private key file and a chain of PEM-encoded X.509 certificates, which means it is compatible with Secret resources' tls.key and tls.crt.

I also looked at the Open Distro Helm chart and it seems like everything can be configured with plain Secret resources.

---

@jroper @GrandChaman

The postgres jdbc driver requires a PKCS8 DER encoded private key.

and

Would it be possible to add support for DER encoded certificates and keys, to support postgres jbdc for mutual TLS?

As of 42.2.9 (Dec 2019), the PostgreSQL JDBC driver is compatible with PKCS#12 using the sslkey field. Does it work for you?