Managed ca-certs

[tallclair]

/kind feature

What happened:

A common requirement of container base images is to provide ca-certs. For example, many of our addons are based on alpine purely to install the ca-certificates package.

I believe this is an anti-pattern:

• Revoking or adding new ca-certificates requires rebuilding & redeploying all workloads (that need it)
• If my organization wants to add our internal CA, or decides we don't trust a "standard" CA, we need to build our own images.
• There is poor visibility into which ca-certs are trusted across the cluster.

A related problem is how to mount the Kubernetes root CA into pods. See kubernetes/community#1973 for a discussion.

What you expected to happen:

There is an opportunity for Kubernetes to provide some value-add here with support for cluster-managed root CA bundles. This is already feasible to some extent by putting the ca-certificates.crt bundle in a config map and mounting it into containers at /etc/ssl/, but there are some shortcomings to this approach:

1. No cross-namespace config map references, so the CM needs to be cloned into every namespace
2. Every pod-spec needs to be modified to mount the ca-certs.
3. Lack of standardization means most containers will still build in the certs

An ideal solution would:

1. Have a single cluster-scoped copy of the ca-cert data
2. Custom certs could easily be added as additional resources or k-v pairs
3. Certs are automatically injected (optionally disabled, a la automountServiceAccountToken)
4. Clusters include ca-certs by default (either Kubernetes provides a bundle, or load the master's host ca-certificates on startup)

I'm not sure whether a new resource type (or CRD) is necessary here, or if ConfigMaps (either via x-namespace references or a new ClusterConfigMap) would be sufficient.

WDYT?

/sig auth

[tallclair]

/cc @mikedanese

[smarterclayton]

/etc/ssl is only correct for debian :( centos/fedora are /etc/pki

[kksriram]

Reacting to just the thought of a "managed root CA bundle".

The CA certs one deployment wants to trust vs another deployment will vary. By shipping a root CA bundle, we’d end up requiring someone to curate and manage the trusted Root CAs. 

When we add apps deployed into the cluster, this gets worse with one App wanting to have a different trusted cert set from underlying K8S cluster. 

That can quickly degenerate to an empty set of Root CAs.
Or, at the other end of the spectrum, something like this.

Just an initial reaction.

[tallclair]

@kksriram I was thinking more about the general case, where an application needs the ability to connect to an arbitrary set of servers on the public internet. It may be the case that this is actually a relatively small use case, in which case the discussion of cluster-level config maps & automounting doesn't make sense. Perhaps we don't need to make any changes, and just promote ca-certs through namespace-level config maps as a best practice?

[mikedanese]

Since we want these to be global, can we consider a dedicated type in the certificates.k8s.io API Group? Strawman:

In certificates.k8s.io:

type CertificateBundle struct {
  // Certificates is an array of DER encoded certificates that make up the bundle.
  Certificates [][]byte
}

Then create a VolumeProjection:

type CertificateBundleProjection string {
  Path string
  CertificateBundles []string
}

To construct the volume, kubelet PEM encodes all certificates in all referenced certificate bundles and coalesces them into a single file at Path. I could see a GKE cluster wanting a bundles for:

• The cluster root CA only to replace the old service account secrets
• Roots used for Google APIs to distribute to our logging and monitoring components that only need to talk to googleapis.com
• A bundle for a deployed Istio mesh
• A bundle containing all public roots trusted by ChromeOS

[tallclair]

xref: https://security.stackexchange.com/questions/191512/managing-trusted-certificates-revocation-in-containers-in-as-cloud-native-mean

[tallclair]

@mikedanese I like the overall idea. But why group certificates into bundles, rather than having "certificate" top level resources? It seems like that would make managing the certs easier, though I suppose it makes it more work to create the projection. Maybe a label selector over the cluster certs?

Also, we still need to consider whether this is just a special case of cluster-scoped configmaps...

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[yuko11]

@tallclair isn't it a good idea to add your internal company's CA certificate into underlying host CA certificate bundle file and mount it into each POD? It would allow you to get public CA updates with underlying host OS update, and you can easily manage internal company's CA (except of you need to reboot all cluster hosts to change it). Having Public CA bundle + internal CA certs in configmaps would lead to never update Public CA bundle.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[mleneveut]

Any update on this ?

I need my pods to trust a self-signed certificate for our dev environments. I tried to add the ca-cert to my pod and run update-ca-certificates but I can't make it work.

Any help would be appreciated :)

kind: Deployment
metadata:
  name: bff
  namespace: ci
spec:
  template:
    spec:
      initContainers:
      - command:
        - sh
        - -c
        - |
          mkdir /etc/ssl/certs
          cp -f /tmp/ssl/* /usr/local/share/ca-certificates
          update-ca-certificates
        image: my-image
        name: add-certs
        securityContext:
          runAsUser: 0
        volumeMounts:
        - mountPath: /tmp/ssl
          name: is4-cacerts
        - mountPath: /etc/ssl
          name: shared-ca-certificate-vol
      containers:
        image: my-image
        name: bff
      volumes:
      - configMap:
          name: is4-cacerts
        name: is4-cacerts
      - emptyDir: {}
        name: shared-ca-certificate-vol

kind: ConfigMap
apiVersion: v1
metadata:
  name: is4-cacerts
  namespace: ci
data:
  kubernetes.crt: |-
    -----BEGIN CERTIFICATE-----
    MIID
    ...
    q4Neg/JZ
    -----END CERTIFICATE-----

In the Node's log I got these errors when calling an https request to a service in the same namespace :

Error: self signed certificate in certificate chain

[fabianotessarolo]

@mleneveut which distro is based your image? Have you checked its location?

https://blog.confirm.ch/adding-a-new-trusted-certificate-authority/

[mleneveut]

@fabianotessarolo It' ubuntu 16.

I already tried to put the cert in /usr/local/shared/ca-certificates and run the update-ca-certificates, but my trouble was to do this before the container's main process (nodejs or dotnet run) starts.

I ended up to put the .crt file in /usr/shared/ca-certificates directory of every Kubernetes node, copy it to /usr/local/shared/ca-certificates with a volumeMount and modify the ENTRYPOINT of my Dockerfile with a CMD :

CMD["sh", "-c", "update-ca-certificates && npm start"]

It's not ideal, but working. If we could put the .crt file in the CA cert of Kubernetes and automatically inherits it in the pods, it would be better.

        volumeMounts:
        - mountPath: /usr/local/share/ca-certificates
          name: ca-certificates
      volumes:
      - name: ca-certificates
        hostPath:
          path: /usr/share/ca-certificates

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[neolit123]

/reopen
/remove lifecycle-stale

[k8s-ci-robot]

@neolit123: Reopened this issue.

In response to this:

/reopen
/remove lifecycle-stale

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mikedanese]

/lifecycle frozen
/priority backlog

[costinm]

/cc costinm

Istio ( in particular multi-cluster and SDS provisioning ) would also benefit of this.

[avanier]

Just putting a ping in here. We're scratching our head on how to go about doing this sanely. We have a few ideas in mind, from doing full path validation to subscribing to different bundles / lists. There's quite a few ways to skin that particular cat and all of them seem like 🐰 🕳️ s.

[tallclair]

We don't have any updates or progress on this, but would welcome a KEP if anyone is interested in picking it up.

[mikedanese]

cc @happinesstaker

[howardjohn]

It would be useful to consider if we want a specialized API (mounted ca certs) vs a more generic case like ClusterConfigMap. If we do go for the specialized API, we probably need to careful construct it to not allow "abusing" the API to mount all sorts of other things in the API, in which case we just end up with a poor form of ClusterConfigMap. Because if you we just build an API to mount generic files into a pod automatically but and just call it CertificateBundleMount or something then inevitably someone (probably me) will start trying to use it to distribute other files.

[avanier]

From the short conversations I've had with colleagues I believe we're seeing a specialized API more than a generic one. I'm not sure a ClusterConfigMap would be sufficient to perform the task of merging CA bundles, well not dynamically anyway. One could certainly use ClusterConfigMap that to provision static or externally maintained bundles.

Mildly related, this article explores the discrepancies across distros of CA trust-stores and their location and behaviour. The article doesn't really touch on anything that would be directly in scope for a CA-bundle-merge-o-tron-mount-a-mabob API, but it depicts well the reality a end-user trying to mount a trust-store inside a container would have to contend with. Trying to put the words "quality of life" and CA trust stores in the same sentence is sort of hilarious, but this is a good spot of test cases to see if the idea holds up.

@tallclair I'll see if I can get some bandwidth when I come back from vacation to put together a KEP. This is an interesting 🐰 🕳️ .

[mikedanese]

I agree that we should consider a generic ClusterConfigMap approach but we should also think about what we would gain from bundle specific semantics. A couple things that come to mind:

• Reference support in API extensions that require embedded bundle data today. Non-portability of e.g. dynamic webhook configuration has annoyed me:

kubernetes/staging/src/k8s.io/api/admissionregistration/v1/types.go

Lines 526 to 529 in b19de7f

	// `caBundle` is a PEM encoded CA bundle which will be used to validate the webhook's server certificate.
	// If unspecified, system trust roots on the apiserver are used.
	// +optional
	CABundle []byte `json:"caBundle,omitempty" protobuf:"bytes,2,opt,name=caBundle"`

• Any extra validation specific to trust bundles.
• Generic object format, with support for OS specific file structure configured at the volume level.
• Flexibility to implement some custom rollout strategy. I suspect that the maintainers of cluster wide bundles often won't be the same set of people that maintain application configuration. This isn't the case with ConfigMaps. We need to do some thinking about how we want to decouple these responsibilities while still supporting safe rollout of changes.

[happinesstaker]

I am planing to work on this trust distribution issue and compiled a draft of design options and consideration. It is still in early stage so I would like to involve more people here to give their brilliant idea and help analyze pros&cons of options.

Since it would be too long to paste the entire content here, I put them in a Google Doc at:
https://docs.google.com/document/d/1zC7BQdMMFRPuZpMd9Efd0nNNsTKETRuV6xDMZ2EDYr4/edit?usp=sharing

[Siegfriedk]

I would like to add my use case: Running an image as non root makes it quite complicated to update a trust store with a certificate which is deployment specific;

Like for a proxy, we generate the proxy ca with the deployment and would need to tell the deployment to trust that certificate.

Other tools like cilium (i believe) make this easier as you have that root ca available earlier, so you could bake it into an image but thats far from convenient.

It is also not far away, to have network policies allowing you fqdn with tls which would solve plenty of problems.

We also spend quite a lot of time of thinkering with updating the trust store of the underlying os, java and python.

[gfiasco]

Just an FYI openshift can achieve this https://docs.openshift.com/container-platform/4.6/networking/configuring-a-custom-pki.html

[GauntletWizard]

The recently-accepted ClusterTrustBundle KEP will provide this mechanism.