design: reduce scope of node on node object

[mikedanese]

Super mini design doc about centralizing reporting of some sensitive kubelet attributes.

@kubernetes/sig-auth-misc
@roberthbailey
@kubernetes/sig-cluster-lifecycle-misc as it relates to registration

[smarterclayton]

Since those fields change over time I'm not even sure that initializers are required for anything except strong exclusion rules.

[mikedanese]

I think you are right. I think we want strong exclusion of sensitive labels, taints and addresses though. I will keep the initializer and add a stanza that allows the central controller reconcile objects as well.

[liggitt]

"sensitive labels" is going to be really tough to pin down. Examples of node self-labeling I've seen just in the past two days are cpu policy and kernel version, both of which could be used for capability steering (a workload requiring a specific CPU policy or kernel version to run) or for security purposes ("find nodes running a kernel with a known vulnerability", "schedule pods to a known good kernel")

[smarterclayton]

Has come up a lot in security auditing.

[ConnorDoyle]

The label part seems minor in comparison to the problem of compromised nodes registering themselves at all.

[ConnorDoyle]

Which is another way to say, if we can't trust a node to set appropriate labels then why are we trusting it at all?

[liggitt]

labels allow steering workloads. we need to be able to set up labeled topologies of nodes to keep classes of workloads separate, and be able to isolate a compromised node by tainting it, unlabeling it, and know that it didn't steer workloads to itself by adding to its own labels.

[liggitt]

the same applies to allowing a node to remove taints (or delete its own Node API object while tainted)

[liggitt]

cc @kubernetes/sig-node-proposals

[mikedanese]

This was discussed in sig-auth and the proposal needs some further consideration:

• We need to dig into the label/taint whitelisting mechanism. We need to make it easy for kubelet to set some of these on itself. We need to make it clear that kubelet self-set labels cannot be used to implement strong exclusion.
• We need to clarify what the expectations are for a deployment running without a cloud provider.
• We need to decide whether a node should be able to delete itself, thereby potentially removing taints from itself.
• We may need to consider how we might need to adjust kubelet starts up.

[liggitt]

this also came up in the GCE cloud provider trying to determine what zones exist by looking at the node API objects and trusting whatever zone they reported they were in (kubernetes/kubernetes#52322 (comment))

[tallclair]

/cc @davidopp @mtaufen

[roberthbailey]

@mikedanese - are you aiming to get this merged and something implemented for 1.9? Or is this a longer term proposal?

[liggitt]

are you aiming to get this merged and something implemented for 1.9? Or is this a longer term proposal?

I'd like to see the labeling/tainting approach agreed on and implemented for 1.9

The node addresses are more involved and have more ties to cloud providers and variance between cloud/non-cloud environments.

[thockin]

I'm all for belts and suspenders, but is "an intruder registers a node into our cluster" a high-prio attack vector?

[smarterclayton]

It turns "can launch a VM inside a given infrastructure account" into "can root the entire infrastructure account" when you take into account that the masters need certain privileges on the infrastructure account. So I would say yes.

[thockin]

Maybe simpler to say:

kubernetes.io/hostname
kubernetes.io/os
kubernetes.io/arch
kubernetes.io/instance-type
[*.]beta.kubernetes.io/* (deprecated)
failure-domain.kubernetes.io/*
[*.]kubelet.kubernetes.io/*
[*.]node.kubernetes.io/*

Could we maybe argue to allow all of naked kubernetes.io ?

Or at least make it clear that this list may change in the future. Concretely, we might add more top-level things, we might enable new prefixes, and we might even provide policy rules to users.

[liggitt]

Maybe simpler to say
...

Updated to just enumerate allowed labels

Could we maybe argue to allow all of naked kubernetes.io ?

I'd prefer to start with this specific set, and document the allowed set may grow or shrink in the future.

Or at least make it clear that this list may change in the future. Concretely, we might add more top-level things, we might enable new prefixes, and we might even provide policy rules to users.

+1

[thockin]

Is this name already codified somewhere? I don't like the length or specificity of it.

spitballing:

protected.kubernetes.io
user.kubernetes.io
admin.kubernetes.io
local.kubernetes.io
site.kubernetes.io
my.kubernetes.io
x.kubernetes.io

Or maybe a distinct TLD?

[.]local/
[.].k8s/

Think through whether this prefix will be applicable in any other context (one advantage of node-restriction is that it is pretty clearly node-related).

Naming is hard, but I am OK with the rest of this proposal

[liggitt]

Is this name already codified somewhere? I don't like the length or specificity of it.

No, it's new to this proposal. I agree on the length, but I like the specificity.

Think through whether this prefix will be applicable in any other context (one advantage of node-restriction is that it is pretty clearly node-related).

having stewed on it for a few days, I actually like this prefix for this use:

• it is clearly node-related
• it only reserves a single specific label prefix
• it connects the label to the admission plugin that enforces it
• it connotes both a restriction on the node itself, and reads ok as a node selector (node-restriction.kubernetes.io/fips=true, node-restriction.kubernetes.io/pci-dss=true, etc)

[thockin]

I'll LGTM for merge now, but would appreciate just a BIT more thought on naming and scope.

/lgtm
/approve

[krmayankk]

who is working on the implementation ?

[liggitt]

I am, will update the PR today after thinking through Tim's last comments

[thockin]

See https://github.com/kubernetes/kubernetes/pull/70555/files for another proposed use of a label to control daemonset addons.

…

On Fri, Nov 9, 2018 at 3:52 AM Jordan Liggitt ***@***.***> wrote: I am, will update the PR today after thinking through Tim's last comments — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#911 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVGPte5wD9MFcCrFtMF-nFAxJT2_gks5utWyWgaJpZM4O2_Ym> .

[smarterclayton]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: smarterclayton, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• keps/sig-auth/OWNERS [smarterclayton]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[liggitt]

/hold cancel

[thockin]

/lgtm

…

On Mon, Nov 12, 2018 at 10:15 AM k8s-ci-robot ***@***.***> wrote: Merged #911 <#911> into master. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#911 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVO26YI_pZRtksm-ziTqYfGDsE3diks5uubrNgaJpZM4O2_Ym> .