Automated cherry pick of #119434: Include ignored pods when computing backoff delay for Job pod

[mimowo]

Cherry pick of #119434 on release-1.27. PRs:

1. Include ignored pods when computing backoff delay for Job pod failures #119434: Include ignored pods when computing backoff delay for Job pod
2. Adjust the algorithm for computing the pod finish time #118413: Adjust the algorithm for computing the pod finish time

PR (2.) is added to ensure the new test passes.

For details on the cherry pick process, see the cherry pick requests page.

Fix computing backoff delay when using Job pod failure policy, by including in the backoff delay calculation pod failures ignored from the backoffLimit counter.

Also, compute the backoff delay more accurately for deleted pods.

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mimowo]

/assign @alculquicondor
One complication with the cherry-pick is that 1.27 doesn't have the refactoring of the computation of the finish time: #118413, and so in 1.27 the finishTime fallbacks to now.

Some of the existing unit test cases in TestSyncJobWithJobPodFailurePolicy (using the ignore rule), are sensitive to it and started to failed, because they expected wantStatusActive: 1, which would be 0 due to delayed pod recreation. Since the intention of the test cases is to check if the pod is ignored from counting I have adjusted such test cases to set FinishedAt as testFininishedAtTime which is in the past, to make sure they continue to pass.

A dedicated test case for delayed pod recreation with wantStatusActive: 0 is cherry-picked.

EDIT: an alternative would be to cherry-pick the new algorithm for finish time computation, but it does not seem to be warranted by this fix itself.

[alculquicondor]

I prefer we cherry-pick both PRs

[mimowo]

I prefer we cherry-pick both PRs

Ok, then we need to merge this one first: #119477.

[mimowo]

Ok, then we need to merge this one first: #119477.

And it this PR I reverted the test changes. Just left the conflict resolution changes related to syncJobCtx. This means that the unit tests will fail now, but will start passing once rebased after the other PR is merged

[alculquicondor]

I suggest you put them in one PR (separate commits)

[mimowo]

I suggest you put them in one PR (separate commits)

Ok, the previous PR included 2 commits, so cherry-picked them both here. I've also included the release not from the other PR here.

[alculquicondor]

Please update the description to reflect that there is more than one PR cherry-picked.

[alculquicondor]

/lgtm
/approve

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 7643579cc3bdf47a5cf08b2ed87ea8b5112a1c3b

[alculquicondor]

/kind bug
/cc kubernetes/release-managers

[xmudrii]

Doesn't qualify for cherry-picking to earlier branches: https://kubernetes.slack.com/archives/CJH2GBF7Y/p1690969553863649?thread_ts=1690961038.586229&cid=CJH2GBF7Y
For RelEng:
/lgtm
/approve

[mimowo]

Note that this issue also affects 1.26, but the fix cannot be cherry-picked there in this form, as it is based on the refactoring which as done in 1.27: #114768.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, mimowo, xmudrii

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/controller/job/OWNERS [alculquicondor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[xmudrii]

/retest

[k8s-ci-robot]

@mimowo: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-e2e-gce	ed0cdc9	link	unknown	/test pull-kubernetes-e2e-gce

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[mimowo]

/test pull-kubernetes-e2e-gce