Scheduling Pods with dynamic resources and using the Pod .spec.nodeName field fails

[uniemimu]

What happened?

The combination of using the Pod spec nodeName field with the DynamicResources scheduler plugin (currently feature-gated) doesn't work. No surprise really as the nodeName field effective bypasses the DynamicResources scheduler plugin running, and something rather clever would need to be invented to make things to work. But this is still a bit of a regression compared to the old device plugin model which doesn't have a similar limitation. @pohly as the author of the dynamic resources feature can explain what could maybe be created to fix this.

Granted, forcing scheduling even with a device plugin resource may or may not succeed at the node level depending on the availability of said plugin resources at the node, but with dynamic resources the failure seems to be currently guaranteed.

The use-case for this is such that the Kubernetes scheduler is effectively being preceded by some other scheduler which runs earlier (say, some HPC framework) which is giving the Pods the node names and somehow knows that there will be enough resources, and then the workloads ought to land in the specified node and the container related work is then handled normally from there on. So basically Kubernetes wouldn't do much in terms of scheduling, but takes over from there.

What did you expect to happen?

TBH I expected it to fail and it did. The resource claims won't work, since the scheduler plugin isn't getting a chance to do its thing.

How can we reproduce it (as minimally and precisely as possible)?

Just use DRA with Pod spec nodeName field, one needs a DRA driver obviously.

Anything else we need to know?

No response

Kubernetes version

Working with dynamic-resource-allocation branch from @pohly which recently got merged to the coming 1.26 release

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"5932fc8c", GitTreeState:"clean", BuildDate:"2022-11-15T12:54:32Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"5932fc8c", GitTreeState:"clean", BuildDate:"2022-11-15T13:06:30Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

N/A

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[pohly]

/triage accepted
/priority backlog
/sig node

One possible solution would be to have a new controller which specifically watches for pods in this state (nodeName set and reference claims). That controller then must trigger delayed allocation (if needed) by creating a PodScheduling object with selectedNode set. After allocation it must add the pod to the reservedFor list.

kubelet will immediately try to run the pod once it sees it because of the nodeName field. It will refuse to run it until eventually all claims are allocated and reserved.

Obviously using a scheduler which is not aware of dynamic resources is problematic. There is no guarantee that the chosen node even has the hardware that is needed for the pod and that it isn't in use. It would be good to hear from whoever has this use case whether the solution outlined above would work in practice.

The alternative is to build support for dynamic resource allocation into that other scheduler.

[sftim]

/retitle Scheduling Pods with dynamic resources and using the Pod .spec.nodeName field fails

[alculquicondor]

Should this be a beta blocker? Or we just want to mark this an unsupported behavior long-term?

[sftim]

My approach: provide a sample / “contrib” implementation of a restriction that prohibits specifying .spec.nodeName - other than by binding the Pod - for a Pod that also requests dynamic resources.

If ValidatingAdmissionPolicy gains a way to restrict this, we can provide a contrib ValidatingAdmissionPolicy as well, and recommend that tools for managing cluster lifecycle put that in place.

That'd be enough to move to beta. IMO.

[pohly]

Should this be a beta blocker?

I'm not sure whether it needs to be a blocker, but my intention is to work on this soonish - definitely before beta.

[pohly]

I intend to support this instead of prohibiting it.

[alculquicondor]

It sounds like it could be a non-trivial change to the design. Once you have something, please send a PR to the KEP.

[pohly]

Code PR: #118209
KEP PR: kubernetes/enhancements#4063