Job spec.backoffLimit does not handle pod garbage collection

[ashu210890]

What happened?

We created a cronjob with backoffLimit of 6 in a cluster with a lot of cronjobs and frequent schedules (possibly cycling through more than 10k pods in a matter of half an hour). The launched pods always end up in failures to a code issue. Kubernetes is now executing infinite number of retries for this cronjob while the maximum retries should have been capped at six.

What did you expect to happen?

We expected that the number of retries will be capped at 6.

How can we reproduce it (as minimally and precisely as possible)?

This requires either
a) churning through 12500 pods in a small time frame or
b) reduce terminated-pod-gc-threshold (this option is not yet tested as we are running on GKE and cannot modify this)

1. Create a cronjob that always fails with a specified backoffLimit.
2. Wait for the amount of time for backoffs to finish
3. Notice that new runs are still being triggered.

Anything else we need to know?

This bug seems to have been introduced while trying to solve another related bug in this PR: #93779

Kubernetes version

$ kubectl version
# paste output here

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.13", GitCommit:"80ec6572b15ee0ed2e6efa97a4dcd30f57e68224", GitTreeState:"clean", BuildDate:"2022-05-24T12:40:44Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.12-gke.1700", GitCommit:"6c11aec6ce32cf0d66a2631eed2eb49dd65c89f8", GitTreeState:"clean", BuildDate:"2022-05-13T09:31:25Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

Google Kubernetes Engine

OS version

# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

$ uname -a
Linux Jupiter 5.13.0-52-generic #59~20.04.1-Ubuntu SMP Thu Jun 16 21:21:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

@ashu210890: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pandaamanda]

/sig apps

[gjkim42]

@ashu210890

Could you attach more information?

i.e.

kubectl describe job BROKEN_JOB
kubectl get job BROKEN_JOB -o yaml

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ashu210890]

@gjkim42 Sorry I missed the comment. This issue, however, is fixed in version 1.22 #98238

We deployed the new version and the it seems to be working fine. Going to close the issue.