BackoffLimit does not work for pods in Jobs (Kubernetes 1.19)

[kolomenkin]

What happened:

1. I deployed CronJob with non-existing docker image,
2. Job was created in time, Pod was created and got pull problems
3. Pod stay in the state "ImagePullBackOff" for 10 and more hours
4. If I deploy new version of CronJob with correct docker image tag it will never start.

What you expected to happen:

I expect Pod to use default BackoffLimit like in documentation:
Default value = 6
Delays = 10sec, 20sec, 40sec, 80 sec, 160 sec and 320 sec
Total delay is less than 12 minutes.

I expect Pod will fail to start in about 12 minutes after it was created.
With "restartPolicy: Never" I expect Job will finish after Pod will fail (in 12 minutes).
And I expect scheduler will create a new job from updated template when it is necessary by the schedule
(currently it never happens).

How to reproduce it (as minimally and precisely as possible):

See "What happened"

Anything else we need to know?:

It seems the counter of Back-off pulling retries is not incremented. Here is a part of POD log:

  Warning  Failed   13m (x439 over 113m)    kubelet  Error: ImagePullBackOff
  Normal   BackOff  3m25s (x484 over 113m)  kubelet  Back-off pulling image "my.private.artifactory/myimage:mytag"

It seems it makes a docker pull try every 12 seconds on average. It seems it waits only initial 10 seconds between retries.

Here is an earlier version of the same POD event log:

  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  24m                   default-scheduler  Successfully assigned mynamespace/mynamespace-mycronjob-1619601000-h5cmr to ip-10-0-45-27.eu-west-1.compute.internal
  Normal   Pulling    23m (x4 over 24m)     kubelet            Pulling image "my.private.artifactory/myimage:mytag"
  Warning  Failed     23m (x4 over 24m)     kubelet            Failed to pull image "my.private.artifactory/myimage:mytag": rpc error: code = Unknown desc = Error response from daemon: Get https://my.private.artifactory/v2/myimage/manifests/mytag: unknown: Authentication is required
  Warning  Failed     23m (x4 over 24m)     kubelet            Error: ErrImagePull
  Normal   BackOff    9m44s (x65 over 24m)  kubelet            Back-off pulling image "my.private.artifactory/myimage:mytag"
  Warning  Failed     4m45s (x87 over 24m)  kubelet            Error: ImagePullBackOff

Related (similar) issues:

• Job backoffLimit does not cap pod restarts when restartPolicy: OnFailure #54870
• Backoff Limit for Job does not work on Kubernetes 1.10.0 #62382

Environment:

• Kubernetes version (use kubectl version): 1.19

Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.6-eks-49a6c0", GitCommit:"49a6c0bf091506e7bafcdb1b142351b69363355a", GitTreeState:"clean", BuildDate:"2020-12-23T22:10:21Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: Amazon EKS (their SaaS Kubernetes)
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

[k8s-ci-robot]

@kolomenkin: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kolomenkin]

/sig architecture
/wg Reliability
/committee steering

[k8s-ci-robot]

@kolomenkin: The label(s) committee/-steering cannot be applied, because the repository doesn't have them.

In response to this:

/committee -steering

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kolomenkin]

/sig apps

[249043822]

If ErrImagePull happened, the containers of pod would not be created, the backoff only does ImagePull again, so the job's BackoffLimit could not perceive the ImagePullBackoff

[SergeyKanzhelev]

Glanced over. Is this dup of #87278?

[kolomenkin]

I would say it is not a full dulicate of #87278.

I can imagine someone can decide to keep those issue without a fix for Deployments. I'm not sure if Deployment yaml editing is affected,

But I point similar behavior in the context of CronJob. I.e. we need to success/fail a job by some schedule.

And it is extremely important when CronJob is updated (redeployed) with a different valid image reference. And these new changes are ignored infinitely.

[k8s-triage-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[kouk]

Does anyone have a workaround for this?

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Cuppojoe]

/reopen

[k8s-ci-robot]

@Cuppojoe: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.