Fix handling of NoExecute taint when PodDisruptionConditions is enabled

[mimowo]

What type of PR is this?

/kind bug

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #112517

The issue is triggered by this GET (protected by the PodDisruptionConditions feature gate):

kubernetes/pkg/controller/nodelifecycle/scheduler/taint_manager.go

Line 124 in a4ad8f4

pod, err := c.CoreV1().Pods(ns).Get(ctx, name, metav1.GetOptions{})

What is more the error was not logged as it was propagated up to here:

kubernetes/pkg/controller/nodelifecycle/scheduler/timed_workers.go

Line 59 in a4ad8f4

timer := clock.AfterFunc(delay, func() { f(ctx, args) })

Thus, as a part of this PR I propose to make sure errors from the functions invoked by timed_workers are logged.

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Yes, it fix that pods running on nodes tainted with NoExecute continue to run when the PodDisruptionConditions feature gate is enabled

Fix that pods running on nodes tainted with NoExecute continue to run when the PodDisruptionConditions feature gate is enabled

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[k8s-ci-robot]

@mimowo: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mimowo]

Initial investigation, after adding a generic error message suggests this is a permissions issue:
I0916 16:04:35.577007 1 timed_workers.go:62] "Timed worker function failed. " err="pods "job-longrun-c7jcp" is forbidden: User "system:serviceaccount:kube-system:node-controller" cannot get resource "pods" in API group "" in the namespace "default""

[alculquicondor]

/lgtm

[mimowo]

/retest

[mimowo]

/retest

[mimowo]

@alculquicondor please renew LGTM, no changes in this PR, but I rebased as there were some unrelated tests failing - don't know why, but rebase helped.

[alculquicondor]

/lgtm

[mimowo]

/assign @liggitt
Should we also cherry-pick it for 1.25?

[alculquicondor]

low priority, because it's alpha, but we should.

[liggitt]

revert the changes to this test, I don't think we need/want a proliferation of fixtures for every feature gate

[mimowo]

Ok, reverted the unit test changes. Will rely on manual testing + e2e testing once in Beta.
@alculquicondor please renew the LGTM if you are ok with the change.

[liggitt]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, mimowo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/controller/nodelifecycle/OWNERS [liggitt]
• plugin/pkg/auth/authorizer/OWNERS [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[alculquicondor]

/lgtm