Job back-offs get reset on controller-manager restart

[sathyanarays]

What happened?

Exponential back-off kicks in when a pod fails. The back-off logic is offloaded to the in-memory workqueue data structure. On controller-manager restarts, the information in the workqueue is lost. So, the back-off information is lost and the back-off gets calculated as if the job is a new one.

What did you expect to happen?

The pods to be created with correct back-offs even when the controller-manager restarts

How can we reproduce it (as minimally and precisely as possible)?

1. Start a job that always fail
2. Let the job fail till we see a visible back-off
3. Stop the controller-manager container
4. New pod gets scheduled immediately after controller-manager restart irrespective of the previously computed back-off

Anything else we need to know?

No response

Kubernetes version

In master as of commit d2504c9
Also present in latest release.

Cloud provider

NA

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

@sathyanarays: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sathyanarays]

/wg batch

[sathyanarays]

related to #114391 and #114651

[alculquicondor]

There is a more immediate problem for me:
All job syncs are tied to the workqueue delays. We should just be delaying pod creation, but not job status updates. This is specially important when the job hits the backoffLimit. We would wait X seconds before declaring the Job as failed, even though it could be done immediately.

I think the solution is somewhat simple: keep track of the elapsed time within the sync, instead of the workqueue and restrict pod creation specifically. The time should be measured based on the Pod status, to be resilient to restarts. But even if we can't do that immediately, just tracking in memory shouldn't be too bad to start.

[alculquicondor]

cc @mimowo

[sathyanarays]

@alculquicondor , @mimowo , please provide early feedback on these changes if they are in the right direction!

#114684

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[alculquicondor]

/close

I can't remember were we said this, but I think we agreed that handling this is overkill.

[k8s-ci-robot]

@alculquicondor: Closing this issue.

In response to this:

/close

I can't remember were we said this, but I think we agreed that handling this is overkill.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.