Terminated but not GC'd Pods should not be re-admitted on kubelet restart

[SergeyKanzhelev]

What happened?

Pods that already succeeded and waiting for finalizer are being readmitted, fail to be admitted and moving from Succeeded Phase to Failed (admission failed) phase.

What did you expect to happen?

The correct behavior would be to not try to re-admit the succeeded (or failed) Pod.

How can we reproduce it (as minimally and precisely as possible)?

Specific case:

• Node is exposing Devices using device plugin. Device plugin is implemented as a Pod
• Job's pod that is using the Device run to completion and waiting for Job Status finalizer
• kubelet is being restarted
• DevicePlugin pod is reconnecting to the kubelet. While it is reconnecting, kubelet don't have availability of devices
• kubelet re-admits the Pod, even though it is terminated already
• kubelet marks the pod as "failed to be admitted"

But the same can happen with the variations of GC speed and other admission handlers.

Anything else we need to know?

/sig node
/priority important-soon
/cc @bobbypage @smarterclayton

There is also a wider issue: #123980, that concentrates on running pods. This issue is about terminated Pods, which is less controversial.

Kubernetes version

Details

For sure on master, may be on earlier versions.

Cloud provider

Details

N/A

OS version

Details

N/A

Install tools

Details

N/A

Container runtime (CRI) and version (if applicable)

Details

N/A

Related plugins (CNI, CSI, ...) and versions (if applicable)

Details

N/A

[ffromani]

/triage accepted

I think we have at least another issue mentioning the same behavior but observed for jobs, I'll look it up and crosslink later

[dsxing]

/triage accepted

I think we have at least another issue mentioning the same behavior but observed for jobs, I'll look it up and crosslink later

#117955