There seems to be at least one other similar report in the wild.

You can sort of get a feel for what's going on here by looking at these two code snippets:

This is what happens in the fast path when the node does not have any NoExecute taints:

	if len(taints) == 0 {
		klog.V(4).Infof("All taints were removed from the Node %v. Cancelling all evictions...", node.Name)
		for i := range pods {
			tc.cancelWorkWithEvent(types.NamespacedName{Namespace: pods[i].Namespace, Name: pods[i].Name})
		}
		return
	}

If you have any NoExecute taints on the node processPodOnNode gets called and this routine is really, really eager to evict pods. So this code path is followed:

	if minTolerationTime < 0 {
		klog.V(4).Infof("New tolerations for %v tolerate forever. Scheduled deletion won't be cancelled if already scheduled.", podNamespacedName.String())
		return
	}

We are currently working around this issue by changing over our NoExecute taints to NoSchedule.

/sig scheduling node

/cc @karan

/remove-sig scheduling

scheduler isn't involved in this case.

Which SIG owns the controller-manager or is the node lifecycle manager (and its NoExecuteTaintManager) owned by sig-node alone?

I think node lifecycle is jointly owned by sig-node and sig-cloud-provider

API machinery owns the mechanics of the controller manager (controller loop setup and management), but the individual controllers are SIG-specific

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Looks like we lost some momentum here and I'd like to avoid this falling into the abyss of bugs that never get fixed. When this bug is triggered the outcome is pretty terrible and once it does happen it is very difficult to dig into and understand what actually happened. Did we ever get this routed to the right owner?

adding sig-node leads for triage/routing. this seems like a clearly reproducible bug with significant correctness issues.

/assign @derekwaynecarr @dchen1107

@bobveznat @pires thanks for the detailed report and reproducer. to help accelerate this, do you have a form of your reproducer we could add in the e2e suite (even if its failing initially)?

do you have a form of your reproducer we could add in the e2e suite

custom taints are considered disruptive, so it couldn't run in the main suite

It's been a while since I wrote tests for the e2e suite but if you are available over slack in case guidance is needed, I'm willing to put the effort.

custom taints are considered disruptive, so it couldn't run in the main suite

Do all suites need to pass for a fix to be merged? I'd assume it is the case, otherwise how would a regression be prevented after the fix lands?

@pires i was more inquiring if there was a test that could automate the reproduction, so even just a linked commit may be helpful for someone that wants to pick this up. i am happy to help if you want to reach out over slack later this week.

looks like a unit test could reproduce the issue, opened a WIP fix in #93722

@derekwaynecarr our code is not publicly accessible, I'm sorry. PR linked by @liggitt seems to cover both the fix and the test that proves it.

/remove-lifecycle frozen

frozen keeps the bot from auto-closing

Ah, sorry then.