Node reboot leaving existing pod using resources stuck with error UnexpectedAdmissionError

[liuyuan10]

What happened?

When a node is rebooted, pod using resources allocated by device plugin will encounter UnexpectedAdmissionError error as below:

  Warning  UnexpectedAdmissionError  84s                kubelet            Allocate failed due to no healthy devices present; cannot allocate unhealthy devices xxx, which is unexpected

What makes it really bad is if it's a raw pod, it stucks in such state and never recover.

What did you expect to happen?

The pod should be retried until device plugin is ready

How can we reproduce it (as minimally and precisely as possible)?

1. create a pod requesting any resources allocated via a device plugin
2. reboot the node
3. observe pod failure

Anything else we need to know?

The behavior is introduced with #116376

And there are various issues opened around kubelet restart #118559 #124345

But this issue is about node restart. When node is restarted, kubelet started to rerun existing pods in random order. So a pod can run into this issue before device plugin pod is healthy on the node

Kubernetes version

kubectl version
Client Version: v1.30.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.1

Cloud provider

GKE

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[liuyuan10]

While this issue can be easily reproduced with node reboot, I believe it can also happen if device plugin suddenly becomes unhealthy after pod is scheduled and before kubelet allocates the resources.

The expected behavior is that the failure should be retried just like CNI failures.

[tamilselvan1102]

/sig node

[SamDecrock]

I can confirm this issue. Luckily for us the pods are managed by Deployments and they are restarted. But we're still stuck with 'ghost' pods in an UnexpectedAdmissionError state.

[ffromani]

What happened?

When a node is rebooted, pod using resources allocated by device plugin will encounter UnexpectedAdmissionError error as below:

  Warning  UnexpectedAdmissionError  84s                kubelet            Allocate failed due to no healthy devices present; cannot allocate unhealthy devices xxx, which is unexpected

Hi! This behavior wants to make evident and recoverable a previously hidden breakage when pod actually started, but noone allocated the device they requested: #109595

I don't know the mechanics of that specific GPU device plugin, but it seems likely to me that this problem can totally happen also for gpu devices. IOW, from what I gathered it was not device-specific but rather a flaw in how the devices are handled in kubelet, hence the flag.

What makes it really bad is if it's a raw pod, it stucks in such state and never recover.

This behavior can be surprising indeed, but when pod reach terminal state the system doesn't try to recover them, this is why the platform introduce and recommends higher level controllers like deployment

What did you expect to happen?

The pod should be retried until device plugin is ready

Yes, this is something we've being discussed since a while but there's unfortunately nothing concrete yet :\

[ffromani]

/triage accepted

kubelet retries are being discussed since a while and there's general agreement it's a desirable behavior

[neolit123]

The behavior is introduced with #116376

/assign @swatisehgal
for feedback as the PR author.