Removing need to have service account tokens in etcd AKA 'dynamic secrets'

[smarterclayton]

Today we generate a service account token into etcd for every service account created. A controller watches for new service accounts, generates the token as a secret, and then links the secret to the service account. The secret must contain a ca.crt, a token, and optionally a namespace for use in contacting the apiserver. An admission controller then automatically mounts that token (unless requested otherwise) into every pod that runs that service account. This is a stable API for clients that must be preserved.

The token is a JWT signed token that identifies the secret name, UID, service account name, namespace, and a few other values. It does not expire (but can be expired by a user by deleting the secret). The ca.crt is admin controlled, but can potentially be large, and is identical for almost all clients.

Some integrations may take the service account token and transform it - OpenShift does so to generate pull secrets to an integrated registry, I've seen it used to generate other cryptographic entities.

Challenges today:

1. secrets grow very large when ca.crt is large, most of the data is duplicated, and so on dense clusters (O(namespaces) ~ O(pods)) this can grow very quickly (we have a 10k namespace cluster which has 3 service accounts per namespace and 3 secrets per namespace).
2. The pattern whereby the secret is created is a controller, but requires the secret to be transformed into the secret up front - for secret values that should be rotated frequently, this is non desirable because it may increase the write load on the master
3. storing the service account token in etcd leads to accidental escalation attacks in general - access to all secrets is equivalent to having union(permissions_of_any_service_account) which is often cluster admin
4. some cluster administrators don't want service account tokens to be accessible to end users

Future direction:

1. We believe secrets in many cases should not have to live in etcd, especially those that can be dynamic, managed externally, or which should expire frequently. We've said we would prefer if "retrieving the contents of a secret" could be decoupled from "retrieving the metadata of a secret"
2. Service account tokens, being frequently tied to permissions available on the cluster, should not be accessible by someone reading all secrets
3. Any change to the behavior of service account secrets would have to be opt-in and has implications for the kubelet to adapt changes. The type field of a secret defines a schema of the data fields - it's not difficult to imagine that additional metadata might be added to the secret to identify where it resides
4. Pods today can only reference secrets and PVCs. It's somewhat undesirable to have pods need to reference other forms of secrets, since every new secret type would require a new volume definition. The entity that provides the secret is orthogonal to how the secret is consumed - anyone expecting token and ca.crt in a directory doesn't necessarily care how Kube provides that (it's a detail of the platform). Other secrets, like a service identity certificate, a pod client certificate, or a standard kerberos keytab, could be modeled as external volumes - but the semantics of how those are managed on the host is "as a secret".
5. The new authorization mechanism to constrain secrets to nodes that pods reference would need to constrain any mechanism where the kubelet asks for a secret on the user's behalf to inject into a pod. That could be in pod, in volume plugin, or in kubelet, but in all cases a check has to be done

Goals:

1. Reduce size of static secrets stored in etcd
2. Make it still possible to get access to service account tokens, but via an API, not reading the secret
3. Make it easier to use dynamic secrets from pods, and have other mechanisms like PodPresets able to use those dynamic secrets
4. Reduce the scope and lifetime of service account tokens to individual users / contexts / pods
5. Allow service accounts to have identity that is coupled to an external security system in some fashion

Possible changes:

1. Make it unnecessary to store the same ca.crt in every secret for the master in etcd
2. Generate a service account token on demand for a pod that is tied to the lifetime of the pod (uuid), rather than the lifetime of a secret on the master
3. Make it possible for a secret to be a reference to some other secret providing entity - the kubelet would read the secret (which has no contents, but does point to a SecretDataSource) and then delegate out to the remote entity
4. Implement a new set of flex volumes that handle all service account token logic, and instead of having the master admission controller link a secret into a pod definition, have it link a flex volume (still requires master config)

Assumption of most external dynamic secret generators is that they need to be able to tie the requesting entity (node or pod) to a permission or fact (node A runs pod B under service account C) in order to release the secret.

[smarterclayton]

We are very incentivized in the very short term to address the secret duplication problem (we're hitting scale limits that we have to solve in some fashion) - I'd prefer to take some baby steps to dynamic secrets.

[smarterclayton]

@kubernetes/sig-auth-feature-requests

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[liggitt]

/remove-lifecycle stale
/lifecycle frozen

[liggitt]

tracked in https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/svcacct-token-volume-source.md and kubernetes/enhancements#542

/close

[k8s-ci-robot]

@liggitt: Closing this issue.

In response to this:

tracked in https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/svcacct-token-volume-source.md and kubernetes/enhancements#542

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.