Taint toleration configured on namespace level

[powereborn]

What would you like to be added:

The feature that would be interested to add is the capacity to prevent all pods in certain namespaces to be scheduled on some worker with a specific label.

Why is this needed:

It's needed notably in the case of a kubernetes multi-tenant cluster. Let's imagine a specific case, you have one customer per namespace. You want to force them to use specific workers with a simple offering. Let's say now you have a rook cluster and you want dedicated nodes on which customer cannot schedule any pods to avoid noise.
It would be quite useful in terms of security and features to be able to force all pods in a namespace not to be scheduled on some nodes.

/sig node
/sig scheduling

[mattjmcnaughton]

Thanks for starting a conversation on this @powereborn :)

I'm wondering if some of the comments in #77202 might be pertinent?

Specifically, it could be good to investigate #sig-node on slack and the weekly meetings to see if there are other ways to solve this problem or if there is general interest in your proposed solution.

For the former, it could be interesting to examine if Admission Controllers could achieve this.
If the later, please create a Kubernetes Enhancement Proposal (KEP) to track your work.

[powereborn]

Hi Matt,

Actually I think PodNodeSelector in AdmissionController are exactly what I needed !

https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/node-selector: name-of-node-selector
  name: namespace3

"PodNodeSelector allows forcing pods to run on specifically labeled nodes. Also see the PodTolerationRestriction admission plugin, which allows preventing pods from running on specifically tainted nodes"

It also says "Evaluate the pod’s node selector against the namespace node selector for conflicts. Conflicts result in rejection." so I guess it will prevent users from trying to schedule their pods on other nodes that have not the selector for their namespace. If I understand right.

[mattjmcnaughton]

Wonderful :) I'm glad that what already exists can help you address your use case.

If you no longer believe this feature is necessary, mind closing it?

As a side note, I'm curious on your thoughts about how we can better document that AdmissionController can be used for this purpose. Any thoughts?

[mikedanese]

Is this a dupe of #61185 ?

[powereborn]

I think it's indeed related to 61185 as well.

For the documentation, I would say highlight this feature adding it in https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ page for example,
or https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/

Closing this issue, thanks :)