High-level discussion of k8s' security model for user-facing surfaces.

[Q-Lee]

This is a little different from Clayton's threat model (#504), but has a non-zero intersection.

I would like to nail down the direction for Kubernetes' security model.

[smarterclayton]

Service accounts can limit what secrets can be mounted. However, since the point of most secrets is to be used in pods, I think the danger is probably more the other way "secrets not intended for use in pods can be accessed by pods". Both directions are valid, of course.

[Q-Lee]

That's a good point. I wasn't trying to create a comprehensive list, but just show some examples. Do you any others?

[davidopp]

Service accounts can limit what secrets can be mounted

Dumb question: How does that work?

[deads2k]

Dumb question: How does that work?

ServiceAccounts have a list of secrets which are allowed to be mounted by a pod using that ServiceAccount. There's an admission plugin which inspects the secrets being mounted by the pod and any secret not in the ServiceAccount.Secrets list will cause a rejection.

[deads2k]

That's a good point. I wasn't trying to create a comprehensive list, but just show some examples. Do you any others?

Mutation of ReplicationControllers allows the ability to delete pods the user couldn't create. Without a concept of ownership (difficult in a multi-mutator system), solving the "problem" is intractable.

[deads2k]

Without a concept of ownership (difficult in a multi-mutator system),

Ah, I see you've tried to define it below. A namespace level owner doesn't seem to fix this concern and the "intentional" owner from below doesn't seem to provide value above RBAC since, "group of people who can edit" is the same as "people bound to role which can edit".

[Q-Lee]

You're right: protecting k8s' objects is insufficient to protect their integrity. The relationships between objects must also be protected (e.g., services <> pods, deployment <> pods, pods <> secrets).

Scoping permissions to a namespace protects most inter-object relationships, because most are bound to a namespace. There are some difficulties with this approach

1. Objects not bound to a namespace

2. The loss of versatility could make separating privileges hard

3. The cluster administrator is responsible for keeping namespaces in sync with ownership

The advantages with namespace are that it can be done today with RBAC.

[feiskyer]

Does this include host path volumes? hostPath should also be controlled for arbitrary mounting.

[smarterclayton]

Note that separating privileges within a namespace has never been a core goal of Kubernetes - the original security model explicitly describes namespaces as a security boundary, rather than within a namespace. Sub-namespace separation should be possible, although I don't know that I think it is required if namespace separation can better model actual user intent.

[Q-Lee]

If we make namespace the unit of security, then we'll need to special case every separation within a namespace, and every non-namespaced object.

Maybe that's not so bad. Most objects are namespaced, and few objects will ever be as precious as secrets. But it will need to be explicit in our security model, so that we can check for special cases when we're reviewing new features.

[smarterclayton]

As an alternative - why not model namespaces as being "owned"?

[Q-Lee]

That's essentially what the namespace argument boils down to. Most objects belong to a namespace, and you can own a namespace.

[smarterclayton]

A true ownership model however requires that a user leaving the organization transfer ownership of all resources. There are two predominant models of tenancy:

1. resources owned by the user
2. resources owned by the organization

The former is best described as "when you leave, these are deleted". The latter is defined as "when you leave, everything keeps working but you no longer have access".

If namespaces are owned by individuals (1) or groups (2), is there a significant difference between the granular ownership model and the current model?

[smarterclayton]

Only because no permission model applies to either the actor or the namespace. If namespaces or the owning service account required access to affinities and tolerations before pods could use them, this would not be an issue.

[davidopp]

Dumb question - what does it mean to say a service account owns an object? Just that there is an RBAC rule that allows the service account to perform some verbs on the object?

[Q-Lee]

@smarterclayton - Exactly, we'll need to special access controls for non-namespaced objects.

@davidopp - I think "ownership" is being kept intentionally vague, since it's a concept under discussion. For instance, an admission controller could check that you have permissions to use affinities and tolerations.

[smarterclayton]

Only if users are granted permission on all resources. Granting access to all labels is oversharing - if access is only granted by labels for resources across contexts, then the system fails closed.

[Q-Lee]

That's true, but I dread a k8s where I need permissions for every label. I don't see how access management is simplified by doing this.

[smarterclayton]

List secrets is already a special case, in that secrets are escalating resources. I shouldn't be able to see contents unless I have permission to do so. The act of creating a secret without protection is the problem, not secrets implicitly.

[Q-Lee]

Arguably, all objects need similar protections. It's so much more important for secrets that we had to do something to protect them earlier.

[smarterclayton]

Secrets can be protected by label ACLs if a few other conditions are met:

1. creation of a secret requires user to specify intent, or intent is defaulted to a restrictive mode
2. access to secret contents is restricted by intent or context
3. secrets are write only

[Q-Lee]

Let me see if I understand your points correctly.

1. Intent: some form of "SecretSecurityPolicy" (e.g., {podConsumable: true, initContainer: true, runas: foo-service}? This would default to a fairly inaccessible secret.
2. An admission controller prevents creation of pods that mount a secret without matching the policy.
3. I'm not sure why secrets need to be immutable. Is it because I can "upgrade" the classification (e.g., locking down the runas), but the secret will still be accessible by a grandfathered object?

Rather than special casing access controls for every object-uses-object relationship, I would prefer a generic means of control. It could then be re-used for pod-uses-secret, pod-uses-node, and even service-uses-pod.

[smarterclayton]

The model being proposed is that access to secrets is gated by label. Integrations do not have access to read all secrets, they only have access to read secrets by a particular label. For instance, label kubernetes.io/for-use-by-nodes=1. This would then not be special case - it would be normal case. All "for use by" relationships can be modeled via ACL on labels for secrets.

I do agree that targeted "-uses-" relationships are valuable, but only a small subset of those relationships matter, so I'm not sure that generic "uses" has significant value to users (I think it has to be proved that uses is a strictly better model than buckets + roles for the majority case, unless you're proposing to add "uses" as an additional dimension). I also don't see that "-uses-" addresses confused deputy on controllers (unless I missed that in this doc), which is the more pressing concern. And with controllers, if you grant a controller access to create anything that can use anything else, you also have to enable it to either transitively refer to all uses (thus requiring a fairly large set of permissions and reducing to the current model) or explicitly specify all uses (which for many users will result in them bulk selecting "all related resources" to grant access to).

[derekwaynecarr]

Is it safe to assume that if in the service catalog, we want to be able to reference secrets for use in service Instance or Binding parameters, we would potentially need our users to add a label kubernetes.io/for-use-by-service-catalog=1 label, and then any secrets created by the service catalog as a result of a Binding object being created would need to have a way of defining labels to apply on the secret the catalog provisions?

/cc @kubernetes/sig-service-catalog-misc @pmorie

[Q-Lee]

Let's set up a call to discuss [just] access controlled labels. I think this calls for something with a faster back and forth than a PR.

@deads2k @smarterclayton @davidopp @cjcullen @destijl

[davidopp]

Please include me in the call. (But please notify me of when it will be held though some means other than Github, which I don't catch enough of.)

[smarterclayton]

Ownership doesn't defend against all threats - confused deputy, for example, is just as difficult with ownership. A controller that has access to update a set of services likely does so in response to other inputs, which then means that controller can propagate updates outside of group 'foo' as easily as in the current model.

[Q-Lee]

This is exactly why there needs to be control on object-uses-object relationships. Whether this is implicit with namespace (with special cases for non-namespaced objects), or done by run-time permission checking (e.g., when a service wants to add a pod as a backend) is less important than that it happens at all.

[liggitt]

both noun and verb dimensions are supported by authorization rules

[Q-Lee]

The security model should list what practices are required.

[smarterclayton]

This may not be sufficient for all scenarios, especially with self hosted components like controllers that run with rights across many resources. Policies controlling privileged access to the node must at some level be imposed by the hosts, rather than the cluster. It's easier to build a ring of moats with decreasing privilege than to attempt to cross lines. Outside of the core rings, placement policy can be enforced safely by the apiserver.

[Q-Lee]

I meant to exclude controllers with "user containers". If that's unclear, can you suggest another phrasing?

Are we considering nodes that don't trust the apiserver? It sounds fine for nodes to be the final authority on what they execute, and therefore reject pods.

1. I don't see why the cluster's control plane shouldn't know these restrictions for placement. If 90% of my fleet wants to reject privileged containers, then I would have quite a few mishaps before I found a stable home.

2. Nodes could still be the authority. For instance, the kubelet could pull a "NodeSecurityPolicy" out of a local yaml file, and then write it back to the apiserver. The apiserver will respect the node's policy, but if they get out of sync, then the node will still reject forbidden workloads.

[smarterclayton]

I would rephrase this as:

Kubernetes is not intended to be a key storage solution. It is intended to assist in the transport, rotation, and access to keys.

I would agree that Kubernetes should limit the investment in secret management - but a KMS still must actually trust the Kubernetes system for two reasons: identity and placement (as you note below), and thus an external KMS can only partially limit the spread of an attack.

[Q-Lee]

Yes, but I've been hearing quite a lot of discussion on how identity should be established.

@mikedanese @destijl @cjcullen - any input?

[destijl]

I've been thinking of k8s secrets as supporting two approaches:

1. a built-in, functional secrets store for basic use; and
2. a simple way to interface with enterprise-grade external stores such as Google/Amazon KMS and Hashicorp Vault etc.

[smarterclayton]

Can be - the question is whether that separation is valuable enough to justify the additional complexity. The classic unix "drop privileges" approach is relevant and can be modeled solely with keys delivered to an init container. A KMS can enforce policy that only delivers secrets to pods with a particular shape (an init container that has access to the pod identity volume and is running an image signed with a particular cert).

[timstclair]

Agreed. We have never considered containers within the same pod to be a trust boundary. Per-container privileges are useful for defense-in-depth/least-privileges, but I think we should avoid over-engineering this boundary.

[Q-Lee]

I think we've always intended to support container scope privileges. If we want to bootstrap a KMS with k8s secrets, then it's pretty straight forward.

@mikedanese
@cjcullen

https://kubernetes.io/docs/concepts/configuration/secret/#use-case-secret-visible-to-one-container-in-a-pod

[destijl]

I think we need to design for container identity. There are already use cases, and trying to bolt it in later will cause problems. We can encourage people to split into more pods wherever it's possible to reduce the need for container identity, but in some cases it won't be. A SPIFFE doc had a good use case: a migration of a large monolithic app into a set of microservices. First container deployment might have only broken up the app into a couple of services, so pods have containers with very different roles that need different identities but have to run together on the same pod as the devs work on splitting it out into separate services.

[smarterclayton]

Why is this a problem?

[smarterclayton]

Hosts in existing clusters already require denormalized network policy - denormalizing network policy into namespaces is no worse.

[Q-Lee]

On reflection, I prefer the denormalized model. It beats cutting a ticket to networking, and the scope helps it retain some its purpose (so cruft is more manageable).

In fact... I like it so much that now I want access defined on Services and Deployments :(

[smarterclayton]

:)

[davidopp]

I'd suggest to add a title and some kind of introduction that explains what its intent is. (Preferably something more detailed than "high-level discussion of k8s' security model for user-facing surfaces.")

[davidopp]

Service accounts can limit what secrets can be mounted

Dumb question: How does that work?

[davidopp]

Dumb question - what does it mean to say a service account owns an object? Just that there is an RBAC rule that allows the service account to perform some verbs on the object?

[deads2k]

How is this providing value above knowing the set of people bound to roles which allow them to update the resource. It seems to be a different way of describing the same information. Only doing it twice.

[Q-Lee]

".... The service's backends would also be limited to pods owned by a group."

Explicit ownership makes access control between objects possible. Namespace satisfies this requirement with the caveat that objects can only relate to others with the same "owner".

If we want to avoid special cases (e.g., secrets, nodes, and volumes), or allow separation of privileges within a namespace (a nice, but admittedly dubious, property), then all objects will need to know an owner.

[feiskyer]

I think a section of container runtime isolation should also be added, e.g. via hypervisor-based runtimes (refer frakti).

Hypervisors could provide a more consolidate isolation boundary than namespace-based runtimes. And it could ensure the minimal attack surface in multi-tenant environments.

[timstclair]

Oops, never published old comments.

[timstclair]

As a central requirement of this document, I think it's worth defining "multi-tenant environment" in some detail. E.g. are we talking soft or hard multitenancy? What level of resource sharing is happening (e.g. untrusted code running in application sandboxes set up by a single tenant vs. potential adversaries sharing access to a single apiserver)

[timstclair]

Could this be solved with pod anti affinity instead?

[timstclair]

Agreed. We have never considered containers within the same pod to be a trust boundary. Per-container privileges are useful for defense-in-depth/least-privileges, but I think we should avoid over-engineering this boundary.

[davidopp]

This doc raises good issues but I'm worried it could be confusing to have too many docs about multi-tenancy checked into our repo. @Q-Lee maybe you and I could work together with @smarterclayton on something comprehensive rather than attacking it piecemeal (this PR, my doc on multi-tenancy models, and Clayton's doc have some overlap, while none by itself is comprehensive, and I'm worried people will just get confused if we merge them as separate docs).

[k8s-github-robot]

This PR hasn't been active in 109 days. Closing this PR. Please reopen if you would like to work towards merging this change, if/when the PR is ready for the next round of review.

cc @Q-Lee @cjcullen

You can add 'keep-open' label to prevent this from happening again, or add a comment to keep it open another 90 days