Kubernetes scheduler spams cluster with pods in NodeAffinity status

[rtheis]

What happened:

A deployment is created that uses a nodeSelector after the cluster master is created but before the cluster worker nodes are fully initialized. During the initialization of the worker nodes, the nodes are temporarily available for scheduling without the necessary label to match the deployment's node selector. Depending on how long the nodes are available for scheduling without the necessary node labels for the deployment, the scheduler will start to spam with cluster with pods in NodeAffinity status. This spamming stops once the worker nodes are fully initialized and the pods are scheduled successful. The exact timing of all of this is TBD.

What you expected to happen:

Pods in NodeAffinity status are annoying and require manual cleanup. It would be preferred that the scheduler avoid this an simply issue warning events.

How to reproduce it (as minimally and precisely as possible):

See issue description.

Anything else we need to know?: No.

Environment:

• Kubernetes version (use kubectl version): 1.18.3
• Cloud provider or hardware configuration: IBM Cloud Kubernetes Service
• OS (e.g: cat /etc/os-release): Ubuntu 18.04.4 LTS
• Kernel (e.g. uname -a): 4.15.0-96-generic
• Install tools: Managed service install
• Network plugin and version (if this is a network-related bug): N/A
• Others: N/A

[rtheis]

Related to #43846 and #52902.

[rtheis]

/sig scheduling

[ahg-g]

by spamming are you referring to scheduler logs? also, why are the nodes not labelled from the get go?

[ingvagabund]

the nodes are temporarily available for scheduling without the necessary label to match the deployment's node selector.

Just to confirm, all the pods in question are still in Pending state, yes?

Pods in NodeAffinity status are annoying and require manual cleanup.

If the pods are in pending state, all of them will get scheduled eventually once the nodes get their labels.

[rtheis]

The pods are not in Pending state. They are actually in NodeAffinitiy state. The pods eventually get scheduled but we are left with anywhere from a few to dozens of these NodeAffinity state pods.

Also, we are looking at labeling the nodes via kubelet flags to see if that helps.

[ahg-g]

/remove-kind bug

[zhouya0]

After we add some retry logic and wait time in getNodeAnyway this problem can be solved:

func (kl *Kubelet) getNodeAnyWay() (*v1.Node, error) {
	if kl.kubeClient != nil {
		if n, err := kl.nodeLister.Get(string(kl.nodeName)); err == nil {
			return n, nil
		}
	}
	return kl.initialNode(context.TODO())
}

As you can see this function returns initial node as the first request failed.

[invidian]

I also get hit by this, when single controller node gets rebooted and scheduler starts as static container before kubelet becomes ready.

Can be cleaned up with: kubectl delete pods $(kubectl get pods | grep NodeAff | awk '{print $1}' | tr \\n ' ')

[sjenning]

/sig node

[derekwaynecarr]

the spam cycle is this loop: "pod create, pod schedule, worker reject, pod failed" -- we don't have a lot of protection in higher level workload controllers to handle pods that fail at kubelet admission time versus pods that just dont create due to quota/admission. in general, is the spam causing any other stability issues? is pod gc pruning the pods?

[derekwaynecarr]

I wonder if we can do a trick in node lister to mitigate any race condition here...

[derekwaynecarr]

we could maybe do something similar to https://github.com/kubernetes/kubernetes/pull/91500/files to ensure node lister has synced...

[derekwaynecarr]

hacking on option to ensure if kubelet has a node lister with a valid kube client, we wait for it to sync at least once here which should mitigate this issue. see: #94087

[rtheis]

We noticed the problem on the OLM addon-catalog-source pod which is managed by the OLM operator. Right or wrong, the operator does not like seeing the pod go to NodeAffinity status and no longer reconciles the pod. The end result is that manual intervention (i.e. pod deletion) is required to fix the problem.

[derekwaynecarr]

@rtheis sure... i would have thought the pod is failed in that instance and reached a terminal state. either way, i am looking at what i can do to mitigate this.

[igraecao]

The problem persists in our environment as well, and the workaround cited above will not do anymore. Is there something in the works for this one?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[rtheis]

/remove-lifecycle stale