JobTrackingWithFinalizers: Orphan pods might not get the finalizer cleared

[alculquicondor]

What happened?

I run into a situation where a pod didn't loose the finalizer, then it blocked the deletion of a namespace.

We do have an integration test for orphan pods

kubernetes/test/integration/job/job_test.go

Line 537 in 425ff1c

func TestOrphanPodsFinalizersCleared(t *testing.T) {

But I fear that it might not cover all cases

What did you expect to happen?

Finalizer to always be removed

How can we reproduce it (as minimally and precisely as possible)?

This is a theory, rather than a concrete set of steps:

1. Some pods of a job start.
2. A pod finishes (then it would get a deletion timestamp). Let's assume the job controller doesn't have time to count this pod.
3. A client deletes the job. The cascade would theoretically delete the pod, but since it already has a deletion timestamp, it's not updated again, so it doesn't have a chance to go to the work queue for orphan jobs.

Anything else we need to know?

No response

Kubernetes version

1.23.4

Cloud provider

GKE

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

@alculquicondor: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[alculquicondor]

/sig apps
/area batch

[alculquicondor]

Another theory for a repro: delete pods right before deleting the job.