[EKS] [request]: Kubernetes Restricted Label support for Managed Node Groups

[TBBle]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request

The Managed Node Groups API current rejects labels with prefixes containing kubernetes.io/, k8s.io/, eks.amazonaws.com/. I haven't checked, but I assume this is actually matching the behaviour of the NodeRestriction Admission Controller with the addition of eks.amazonaws.com/ as a further restricted domain, as motivated by the Bounding Self-Labeling Kubelets KEP.

This restriction is not mentioned in the Cloud Formation Managed Node Group API documentation.

Which service(s) is this request for?
EKS Managed Node Groups

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

Although this problem has been known for a while with, e.g., eksctl and unmanaged node groups (see eksctl-io/eksctl#2363), the only current solution on the table for that is kubernetes/cloud-provider-aws#110 which hasn't advanced in a while, sadly.

Managed Node Groups may want to implement the feature differently, or formalise/document the restrictions as "working as designed", or explicitly decide to build a solution on top of kubernetes/cloud-provider-aws#110 (or another solution) once it exists.

eksctl has a different bug tracking this use-case for Managed Node Groups, eksctl-io/eksctl#4007, and I only just noticed that there was no corresponding feature-request here for restricted-label support in Managed Node Groups.

Are you currently working around this issue?

Not using the restricted labels, e.g., if you don't populate node-role.kubernetes.io/, you don't see a role listed in the kubectl get output's dedicated column for this value.

Additional context

This is different from #854 and #733, which was a request to implicitly apply the Managed Node Group's name as the node-role.kubernetes.io/-prefixed label. This feature request is for general support of otherwise-restricted label prefixes (including but not limited to node-role.kubernetes.io) in the Labels field of the Managed Node Group API.

Attachments

None.

[adammw]

I note that the same restriction exists for taints - "Taint cannot contain reserved prefixes [kubernetes.io/, k8s.io/, eks.amazonaws.com/]"

[TBBle]

That's interesting, because the Kubernetes-side changes do not block taints, only labels. It makes sense for the EKS API to reserve their own prefix (eks.amazonaws.com) but for the rest, it's not mandated by k8s.

That said, the well-known taints are temporary taints in the node.kubernetes.io (managed by the Node Controller) or node.cloudprovider.kubernetes.io (managed by the Cloud Controller Manager) namespaces, so it doesn't really make sense for users to provide them as part of the NodeGroup definition.

[migueleliasweb]

Any updates on this?

Cluster: v1.21
TF EKS module: v17.24.0

I'm still receiving the following error:

Message_: "Label cannot contain reserved prefixes [kubernetes.io/, k8s.io/, eks.amazonaws.com/]"

I understand the rationale behind not enabling full control over these labels with known prefixes but at the same time there should at least be a way to set a node role via an API call when setting up the managed node group. This helps quite a bit when running kubectl get nodes and that shows the correct role.

[sidewinder12s]

Managed Node groups also are not automatically labeling with the node.kubernetes.io/lifecycle common label and now I cannot add it onto managed node groups manually either.

[TBBle]

That's interesting, node.kubenetes.io/lifecycle should not be being blocked by this, as node.kubernetes.io/* is considered a kubelet-settable namespace, specifically allowed by the NodeRestriction Admission Controller.

So that is actually a different problem, and if it's failing in the Managed Node Groups API, i.e. like #1451 (comment), then that probably should be opened as a new feature request/bug report.

This ticket was for providing a way in EKS to set the labels that the kubelet is not allowed to self-set, e.g., node-role.kubernetes.io/*.

[sidewinder12s]

Looks like the EKS Managed Node Group API blocks anything in those prefixes, not just the root.

[pascalwhoop]

this blocks people from following this
https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-can-i-scale-a-node-group-to-0
because if you cannot set the label that is needed to scale from 0, then one has to manually tag the ASG afterwards which is not possible with IaC so ... annoying

[TBBle]

@pascalwhoop That's a different problem, unless you're using one of the restricted labels for scaling (in which case, this would also affect vanilla k8s, and is probably making things harder for yourself, compared to using a local or locally-owned label).

For scale-from-zero on EKS Managed Node Groups, #724 (comment) links to the CA built-in support for that, introduced in EKS k8s 1.24. The FAQ you're looking at is for AWS-hosted non-Managed Node Groups, i.e. when the ASG is being managed directly by the user or infrastructure tooling, or via something like CloudFormation.

It's a shame the FAQ wasn't updated when the MNG support was added, maybe worth filing a bug with CA? The AWS-specific documentation talks about the MNG support for scaling from zero nodes, but probably could be clearer that if you're using the eks:DescribeNodegroup mechanism, the node template tags on the ASG are not necessary.

[enver]

Check out a simple workaround here: https://hub.docker.com/repository/docker/enver/label-nodes/general

[sudosoul]

Stumbled on this issue when attempting to implement Node Restriction via the node-restriction.kubernetes.io/ prefix.

https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-isolation-restriction

[TBBle]

That's indeed a general version of this issue. At this time, MNG relies on Nodes to self-set their labels on connection, and the whole point of the Node Restriction admission controller is to provide a set of labels that nodes cannot self-set, in order to support labels that are administrator validated.

kubernetes/cloud-provider-aws#110 was intended to provide a way to support elastic scaling and still be able to attach labels to nodes that are safe from rouge nodes or compromised nodes (since Cloud Provider AWS is already authenticating nodes against EC2 or ASG AFAIR), but it closed to due to lack of implementation interest.