Support loading identity trust roots from a Secret

[bdun1013]

Feature Request

What problem are you trying to solve?

I am using cert-manager to create the CA Certificate and corresponding Secret for the identity component. The trust bundle is available in the ca.crt field of the Secret. I would like to pull the bundle from this Secret instead of creating a ConfigMap to hold the same data.

How should the problem be solved?

I would recommend adding a .Values.identity.externalCAFromSecret which defaults to false. When both .Values.identity.externalCA and .Values.identity.externalCAFromSecret is set to true, the ca-bundle.crt will be pulled from the ca.crt field on the linkerd-identity-trust-roots Secret that will be managed by cert-manager

What do you want to happen? Add any considered drawbacks.

This would add more flexibility to the deployment.

Any alternatives you've considered?

Is there another way to solve this problem that isn't as good a solution?

I could create a ConfigMap with the ca bundle, but this makes automation more difficult.

How would users interact with this feature?

If you can, explain how users will be able to use this. Maybe some sample CLI
output?

$ helm repo add linkerd https://helm.linkerd.io/stable

$ helm template linkerd/linkerd2 --set identity.externalCA=true --set identity.externalCAFromSecret=true --set identity.issuer.scheme=kubernetes.io/tls

This should not create the linkerd-identity-trust-roots ConfigMap

All references to the CA data should use the ca.crt key in the existing linkerd-identity-trust-roots Secret

[mateiidavid]

Hey @bdun1013, thanks for raising this! Is there a specific reason why a Secret works better than a ConfigMap for you?

To integrate with cert-manager, did you follow the steps from our documentation? If you create the cert-manager Certificate resource, you'll have a Secret created called linkerd-identity-issuer. This secret will contain the issuer keypair and CA public key/cert. You can use this secret when installing linkerd. e.g installing with CLI: linkerd --identity-external-issuer.

If you follow the documentation steps, Linkerd will pull in the CA from the secret created by cert-manager and use it to create the linkerd-identity-trust-roots ConfigMap. Control plane components mount the ConfigMap so proxies have the CA in memory on start-up. It sounds like what you'd be looking for?

[bdun1013]

Hey @mateiidavid, thanks for the reply. I have been following those steps in the documentation, but with one small tweak. I am using cert-manager instead of step to generate my trust anchor CA. Currently, I am just using cert-manager to generate a Self Signed CA, but soon I would like to pull a CA from Hashicorp Vault.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: linkerd-self-signed-issuer
  namespace: linkerd
spec:
  selfSigned: {}

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: linkerd-trust-anchor
  namespace: linkerd
spec:
  isCA: true
  commonName: root.linkerd.cluster.local
  secretName: linkerd-identity-trust-roots
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: linkerd-self-signed-issuer
    kind: ClusterIssuer
    group: cert-manager.io

So I would like to use the ca.crt data stored in the linkerd-identity-trust-roots Secret as my identity trust anchor.

With helm, my current options are to:

• Set identity.externalCA to true and create the linkerd-identity-trust-roots ConfigMap with the data that is already in the linkerd-identity-trust-roots Secret before installing
• Manually copy the CA data from the linkerd-identity-trust-roots Secret and pass it in to helm install with identityTrustAnchorsPEM as per the documentation

So I would just like to make helm chart changes to pull the ca.crt from the linkerd-identity-trust-roots Secret when a new value identity.externalCAFromSecret is set to true

I will throw up a draft PR to help visualize my recommended changes

[mateiidavid]

@bdun1013 thanks for submitting the PR and being so proactive in fixing this :)

This particular painpoint has been felt by Linkerd users for some time. For example, #3843 was opened in 2019 to address this issue specifically. We have made changes to the way we manage identity since then (e.g added support to read trust roots from configmaps).

We are very simpathetic and understand this makes it harder for people to integrate Linkerd with cert-manager (or other similar solutions). However, the solution is not very feasible for us, mostly because of security implications. The Secret itself holds the entire keypair of the CA; aside from the public key/certificate, it also holds the private key. Pods other than the identity controller in Linkerd (which is responsible for signing key requests and distributing certificates) will have access to the Secret and consequently the private key. This same exact reason was also quoted in #3843. There are also some more implementation details that we'd have to cover if we were to read the trust roots from a Secret.

With that being said, we'd ideally people to be able to automate all of this. We were tracking the following issue cert-manager#3949 which addresses the same issue we have: separating the keys from the certificate itself. There hasn't been much progress there, but I have looked at the suggestion from one of the cert-manager maintainers to use trust to distribute trust bundles across namespaces in ConfigMaps. Here's what I came up with after some tinkering:

• Install cert-manager and trust in the same namespace.
• Create self-signed Issuer and Linkerd root CA. Create an Issuer based on the CA so we can have our intermediate Linkerd issuer. You'll notice that this differs a bit from the tutorial; I made the CA issuer a ClusterIssuer so we can sign certificates across namespaces:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: linkerd-self-signed-issuer
  namespace: cert-manager
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: linkerd-trust-anchor
  namespace: cert-manager
spec:
  isCA: true
  commonName: root.linkerd.cluster.local
  secretName: linkerd-identity-trust-roots
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: linkerd-self-signed-issuer
    kind: ClusterIssuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: linkerd-trust-anchor
  namespace: cert-manager
spec:
  ca:
    secretName: linkerd-identity-trust-roots

• Create issuer certificate and linkerd namespace. This will give us a namespace scoped issuer to be used by linkerd (and it will also create the issuer secret we need to install linkerd):

apiVersion: v1
kind: Namespace
metadata:
  name: linkerd
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: linkerd-identity-issuer
  namespace: linkerd
spec:
  secretName: linkerd-identity-issuer
  duration: 48h
  renewBefore: 25h
  issuerRef:
    name: linkerd-trust-anchor
    kind: ClusterIssuer
  commonName: identity.linkerd.cluster.local
  dnsNames:
  - identity.linkerd.cluster.local
  isCA: true
  privateKey:
    algorithm: ECDSA
  usages:
  - cert sign
  - crl sign
  - server auth
  - client auth

• Create a Bundle resource to distribute CA certificate in linkerd namespace as a configmap (source is taken from the namespace trust was installed in, i.e cert-manager):

apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: linkerd-identity-trust-roots
spec:
  sources:
  - secret:
      name: "linkerd-identity-trust-roots"
      key: "ca.crt"
  target:
    configMap:
      key: "ca-bundle.crt"

• Install linkerd, I used the CLI but you can also use helm. We'd have to pass externalCA=true and identity-external-issuer so we can make use of the issuer secret and trust root configmap.

:; linkerd install --identity-external-issuer \
       --set "identity.externalCA=true" \
       | kubectl apply -f -

Here's how everything looks at the end:

:; k get secrets -n cert-manager
NAME                                       TYPE                                  DATA   AGE
linkerd-identity-trust-roots               kubernetes.io/tls                     3      64m

:; k get secrets -n linkerd
NAME                                 TYPE                                  DATA   AGE
linkerd-identity-issuer              kubernetes.io/tls                     3      62m
linkerd-identity-token-7tndg         kubernetes.io/service-account-token   3      58m

:; k get cm -n linkerd
NAME                           DATA   AGE
kube-root-ca.crt               1      62m
linkerd-identity-trust-roots   1      62m
linkerd-config                 1      59m

It's probably not the easiest way to do it, but it helps us with our security concerns and allows you to use your self-signed CA without having to manually pull it out of the secret. Hope this makes sense?

[bdun1013]

@mateiidavid Thank you so much for playing around with this! It makes perfect sense that we don't want to give Pods access to a Secret containing the private key material.

I wasn't aware of cert-manager/trust, but it looks great and worked for me. I will close this issue and the associate PR.