Only declare job as finished after removing all finalizers

[alculquicondor]

What type of PR is this?

/kind bug

What this PR does / why we need it:

When reaching the limit of how many Pod UIDs we can keep in .status, we cannot declare the job as finished.

Side problem (seems to be a relic from before the times of tracking with finalizers):
Always count the failures towards the backoffLimit

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Only declare Job as finished after removing all Pod finalizers to avoid orphan Pods

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[alculquicondor]

/assign @mimowo

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/controller/job/OWNERS [alculquicondor]
• test/integration/job/OWNERS [alculquicondor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mimowo]

LGTM, but I'm wondering about splitting out the check for backoff limit to a separate PR to make the cherry-picks easier.

[alculquicondor]

/retest

[mimowo]

/lgtm for the proposed master fix. We can figure out the cherry-picking in the process.

[mimowo]

/lgtm
Add again as it wasn't picked by the automation before

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: b0b61b0a7dc7bfd2fd07ea40183d9d722d9b2d92

[k8s-triage-robot]

The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

This bot retests PRs for certain kubernetes repos according to the following rules:

• The PR does have any do-not-merge/* labels
• The PR does not have the needs-ok-to-test label
• The PR is mergeable (does not have a needs-rebase label)
• The PR is approved (has cncf-cla: yes, lgtm, approved labels)
• The PR is failing tests required for merge

You can:

• Review the full test history for this PR
• Prevent this bot from retesting with /lgtm cancel or /hold
• Help make our tests less flaky by following our Flaky Tests Guide

/retest