Fix a TaintBasedEviction integration test flake

[Huang-Wei]

What type of PR is this?

/kind flake

What this PR does / why we need it:

If all nodes of a cluster become NotReady, internally it enters a "fullyDisrupted" mode, and hence some functions are not honored (like applying taint, enforce Pods eviction, etc.)

In the TaintBasedEviction test, if a subtest can't finished in 5 seconds, it flakes with following pattern of messages:

I1101 17:05:16.059199  105553 node_lifecycle_controller.go:1060] node node-0 hasn't been updated for 5.000511225s. Last Ready is: &NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:0001-01-01 00:00:00 +0000 UTC,Reason:,Message:,}
...
I1101 17:05:16.078489  105553 node_lifecycle_controller.go:1132] Controller detected that all Nodes are not-Ready. Entering master disruption mode.
...

This PR ensures each Node has a goroutine reporting heartbeat info regularly.

Which issue(s) this PR fixes:

May fix #83321.

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

/sig scheduling
/cc @ahg-g @ravisantoshgudimetla @damemi
/priority important-soon

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Huang-Wei

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• test/integration/scheduler/OWNERS [Huang-Wei]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ahg-g]

why is this not a problem for other integration tests?

[Huang-Wei]

why is this not a problem for other integration tests?

It's because TaintBasedEviction feature is actually implemented by node lifecycle manager (inside controller-manager), not scheduler. And in terms of applying NoExecute taints, node lifecycle manager behaves differently on different modes (Normal, PartialDisrupted, FullyDisrupted). The flake occurs at the exact step of ensuring NoExecute taint is properly applied:

I1101 17:05:42.006204  105553 reflector.go:268] k8s.io/client-go/informers/factory.go:134: forcing resync
    --- FAIL: TestTaintBasedEvictions/Taint_based_evictions_for_NodeNotReady_with_no_pod_tolerations (34.97s)
        taint_test.go:770: Failed to taint node in test 1 <node-2>, err: timed out waiting for the condition

So from the log, we can see that the Node being set NotReady=true intentionally has been adding to taint queue:

I1101 17:05:16.078448  105553 node_lifecycle_controller.go:814] Node node-2 is NotReady as of 2019-11-01 17:05:16.078430562 +0000 UTC m=+268.763594109. Adding it to the Taint queue.

However, in fullyDisrupted mode, it's expected to suppress further actions like Pods eviction since evicted Pods won't have any room to be placed still, so the node is reset to Reachable statue:

kubernetes/pkg/controller/nodelifecycle/node_lifecycle_controller.go

Lines 1150 to 1161 in 590cbef

	// We're switching to full disruption mode
	if allAreFullyDisrupted {
	klog.V(0).Info("Controller detected that all Nodes are not-Ready. Entering master disruption mode.")
	for i := range nodes {
	if nc.useTaintBasedEvictions {
	_, err := nc.markNodeAsReachable(nodes[i])
	if err != nil {
	klog.Errorf("Failed to remove taints from Node %v", nodes[i].Name)
	}
	} else {
	nc.cancelPodEviction(nodes[i])
	}

[ahg-g]

/lgtm

[ahg-g]

/retest

[damemi]

Interesting, thank you for (hopefully!) solving this