Policies on Labels and Selectors

[erictune]

Suppose there is one namespace with two users.

• Trustworthy Tammy is trusted to manage the prod pods.
• Fat-fingered Fred is only trusted to manage the dev pods. fred is prone to making mistakes.

Tammy makes a service that selects pods with "type=prod".

Now, say Fred starts some buggy code in pods. He meant to apply label "type=dev", but by accident he applies "type=prod". Now user requests are hitting his buggy servers.

We should have a way to control who can apply and select a certain label-key-value. ABAC policy could certainly support this.

This is relatively low priority, since people could use namespaces to separate Fred and Tammy's pods. But, in the long run, I think we want to encourage people to put more things in a single namespace.

[thockin]

Once we have identities, should ident be an automatic label? Select pods
owned by foobar-prod with the label type=prod ?

On Thu, Nov 6, 2014 at 12:56 PM, Eric Tune notifications@github.com wrote:

Suppose there is one namespace with two users.

• Trustworthy Tammy is trusted to manage the prod pods.
• Fat-fingered Fred is only trusted to manage the dev pods. fred is
  prone to making mistakes.

Tammy makes a service that selects pods with "type=prod".

Now, say Fred starts some buggy code in pods. He meant to apply label
"type=dev", but by accident he applies "type=prod". Now user requests are
hitting his buggy servers.

We should have a way to control who can apply and select a certain
label-key-value. ABAC policy could certainly support this.

This is relatively low priority, since people could use namespaces to
separate Fred and Tammy's pods. But, in the long run, I think we want to
encourage people to put more things in a single namespace.

Reply to this email directly or view it on GitHub
#2211.

[erictune]

I don't think so.
We have a notion of user identity, but I wasn't planning on using that for pods.
Pods may have access to credentials that they use to identify themselves to services, such as K8s apiserver, or github, etc. But those are not something I can see turning into labels. And you might want to have pods with different capabilities be selected by a selector.

[thockin]

Hmm, I sort of assumed that eventually we would have a "principal ident"
attached to pods and that it would become part of just about all forms of
namespacing.

Because your example is dead on, it should be possible for ME to craft a
selector that YOU can not possibly satisfy, and vice versa.

On Thu, Nov 6, 2014 at 1:16 PM, Eric Tune notifications@github.com wrote:

I don't think so.

We have a notion of user identity, but I wasn't planning on using that for
pods.
Pods may have access to credentials that they use to identify themselves
to services, such as K8s apiserver, or github, etc. But those are not
something I can see turning into labels. And you might want to have pods
with different capabilities be selected by a selector.

Reply to this email directly or view it on GitHub
#2211 (comment)
.

[erictune]

@thockin
The system you and I know and love does have the properties that:

• "container identity is also a namespace"
• "container identity may be the same as a human user or group of users"

GCE does not appear to be that way. It has these properties, as far as I can see:

• "projects are also namespaces"
• "VMs can act as service accounts" -- https://cloud.google.com/compute/docs/faq#serviceaccounts
• "projects can have several service accounts"

I see Kubernetes being more like GCE. It already has namespaces which are not container identity.

[erictune]

To answer your second thing, I think we could have policies that say:
"Only user-account Tim can create objects that have label tim=awesome"
"Only user-account Eric can create objects that have label eric=cool"
Then, you can satisfy selector "tim=awesome" and I can't, and vice versa.

[thockin]

So nothing will be tagged with originating and/or acting identity? I'm
concerned that this will make a very confusing system down the road, but
maybe you have some grand plan and I've just ignored it...?

Setting policy on who can create certain labels feels inverted to me -
labels should be arbitrary.

How do we get to a place where containers don't run as root, but as some
user ID?

On Fri, Nov 7, 2014 at 3:36 PM, Eric Tune notifications@github.com wrote:

To answer your second thing, I think we could have policies that say:
"Only user-account Tim can create objects that have label tim=awesome"
"Only user-account Eric can create objects that have label eric=cool"
Then, you can satisfy selector "tim=awesome" and I can't, and vice versa.

Reply to this email directly or view it on GitHub
#2211 (comment)
.

[erictune]

Certainly the audit log (#2203) will show the user-string who created each
object.
There could certainly be an index of UUID -> creating user string.
And UIs and CLIs might chose to show that alongside an object in
appropriate views.
I hadn't considered creating-user as a first-class part of an object's
type, or metadata, though.

Note that this matches how I understand GCE to work
You make a VM, and the VM belongs to the project.
gcloud compute instances list does not show "etune@google.com" next to my
VMs.
So, if you still don't like it, the some suggested lines of argument would
be:

1. I'm wrong about it being like GCE because X
2. GCE experience could be improved because of y.

I filed this bug after talking about it with @bgrant0607 who can maybe give
better articulated reasons about why labels
need policies. Internal label and policy experts outside the Kubernetes
team also seem to agree Can go into detail on that offline.

Containers should use user namespaces once docker supports it.
So root in container != root on machine.
So, from the standpoint of the client of the system, your last question is
a non-issue.

On Fri, Nov 7, 2014 at 8:52 PM, Tim Hockin notifications@github.com wrote:

So nothing will be tagged with originating and/or acting identity? I'm
concerned that this will make a very confusing system down the road, but
maybe you have some grand plan and I've just ignored it...?

Setting policy on who can create certain labels feels inverted to me -
labels should be arbitrary.

How do we get to a place where containers don't run as root, but as some
user ID?

On Fri, Nov 7, 2014 at 3:36 PM, Eric Tune notifications@github.com
wrote:

To answer your second thing, I think we could have policies that say:
"Only user-account Tim can create objects that have label tim=awesome"
"Only user-account Eric can create objects that have label eric=cool"
Then, you can satisfy selector "tim=awesome" and I can't, and vice versa.

Reply to this email directly or view it on GitHub
<
#2211 (comment)

.

—
Reply to this email directly or view it on GitHub
#2211 (comment)
.

[bgrant0607]

We don't want it to be possible for others to pollute the meaning of labels -- doing so will break lots of assumptions made by users and automated components. For instance, what if users applied labels used by the system? Without enforcement, namespacing of labels only prevents completely accidental collisions, not pollution or subversion caused by misguided or malicious acts. This is an important issue especially in the case that labels are used to attach flexible security to objects, but there are lots of scenarios, such as creating pods that are targeted by another's services or replication controllers, or creating labels that will match selectors for filtering monitoring or logging data. Without a solution to this problem, in my experience all types of selectors/filters just get more and more complex as people try to defend against all of these scenarios.

Object namespaces solve some of the problem by making objects in other namespaces invisible, but it doesn't solve the whole problem, such as abuse of system/tool labels.

[liggitt]

/cc @deads2k

[thockin]

This thread languished (sorry), but to Eric's last comment:

I think #2 "GCE experience could be improved" is true (my opinion).
Knowing the internal systems, and the second-level infrastructure services
that runs on top of the cluster manager but is used by ~everyone, I am not
sure what the mapping is to namespaces. If namespaces are ~projects, how
will app-namespaces in a cluster bridge into infra-service namespaces? Or
if I am thinking of it wrongly, how do we provide, e.g. a central
monitoring service that all other pods can use, without granting pod-owners
admin-access to the monitoring service?

Secondly, regarding root-in-namespace, I think we should steer people away
from using root, even in-namespace. There's some advantage to having
containers run with less privs, and even maybe to each container run as a
different UID in the namespace.

On Thu, Jan 8, 2015 at 6:41 AM, Jordan Liggitt notifications@github.com
wrote:

/cc @deads2k https://github.com/deads2k

Reply to this email directly or view it on GitHub
#2211 (comment)
.

[erictune]

@thockin

In the second paragraph of your last comment, I think you were asking: how are the monitoring pods of the monitoring service, in the monitoring-service namespace, allowed to read the resources and select by labels in a monitoring-client namespace. That is a good question which I certainly had not thought through. Off the top of my head, I'd say that the client needs to grant the monitoring service some kind of read permission on its resources, including the ability to select labels, or at least those labels which are relevant to monitoring. Giving the monitoring service the ability to read labels off your pods and select them seems pretty safe.

Were you envisioning that the monitoring service would need to set labels too? If so, elaborate.

For your third paragraph, I read the man page for user_namespaces but I'm still having trouble understanding what special abilities a root-in-namespace user has. Explanations or pointers welcome.

[thockin]

I guess I can buy the "you need to grant the monitoring team access to your
app by policy" answer for a pull model, but what about push? Maybe that is
just an open policy?

root in namespace can do things like write to all files in the container
chroot and signal other PIDs (assuming we get shared PID namespaces?). If
we could get to less privs by default, wouldn't that be better?

On Tue, Jan 13, 2015 at 9:34 AM, Eric Tune notifications@github.com wrote:

@thockin https://github.com/thockin

In the second paragraph of your last comment, I think you were asking: how
are the monitoring pods of the monitoring service, in the
monitoring-service namespace, allowed to read the resources and select by
labels in a monitoring-client namespace. That is a good question which I
certainly had not thought through. Off the top of my head, I'd say that the
client needs to grant the monitoring service some kind of read permission
on its resources, including the ability to select labels, or at least those
labels which are relevant to monitoring. Giving the monitoring service the
ability to read labels off your pods and select them seems pretty safe.

Were you envisioning that the monitoring service would need to set labels
too? If so, elaborate.

For your third paragraph, I read the man page for user_namespaces but I'm
still having trouble understanding what special abilities a
root-in-namespace user has. Explanations or pointers welcome.

Reply to this email directly or view it on GitHub
#2211 (comment)
.

[erictune]

One of the main cases for this is that services should only use backends they trust. If service load balancer used TLS to talk to service backends, then it would also be able to detect a rogue backend. This seems like a good way to do things.

If the other use cases for label/selector policy are more nuisances, then maybe we don't bother to implement this.

[bgrant0607]

Related: #15390

[bgrant0607]

Note that labels are scoped by namespace.

[erictune]

@olegshaldybin this is related to https://github.com/olegshaldybin/kubernetes/blob/protected-attributes/docs/proposals/protected-attributes.md and #27330

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

[bgrant0607]

/remove-lifecycle rotten
/lifecycle frozen