Scheduler will run into race conditions on large scale clusters

[ahg-g]

What happened?

The scheduler has a 30s timeout for the bind operation to succeed; if we don't get a response within 30s, the in-memory assignment of pod to node in the scheduler cache expires.

A race condition will happen in the follow case:

1. pod1 is assigned to a node, scheduler cache is updated with the assignment, bind operation issued to apiserver.

2. if the apiserver is under huge pressure, bind takes more than 30s, scheduler expires the cached pod-to-node assignment.

3. bind eventually succeeds, but because the apiserver is under huge pressure, the pod update with the node name takes a long time to propagate to the scheduler.

4. because the pod update took a long time to propagate and the cache entry expired, the scheduler is not aware that the assignment actually happened, and so it had no problem assigning a second pod to the same node that would otherwise not fit if the scheduler was aware that the first pod was eventually assigned to the node.

On the scheduler side, what we need to do is make the 30s longer for large clusters, and ideally adaptable to cluster state.

/sig scheduling

What did you expect to happen?

No race conditions.

How can we reproduce it (as minimally and precisely as possible)?

Create a large scale cluster.

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
# paste output here

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

@ahg-g: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[alculquicondor]

IIRC, the 30s timeout only starts running after we receive a 200 response for the binding. But yes, if it takes longer than 30s to receive the update, we would invalidate the cache.

I don't know if there is particular reason to invalidate the cache at all. I suppose it was meant to guard against missing Pod deletion events. We need to improve that regardless.

I would be in favor of dropping the timeout altogether.

[ahg-g]

ok, so the deadline is effectively for receiving the update.

Perhaps making it longer, like 15min, is good first step before completely removing it just so we can easily revert in case something else goes wrong.

[ahg-g]

I sent #106412 to increase the timeout to 15min. We should backport this.

[wojtek-t]

I don't know if there is particular reason to invalidate the cache at all. I suppose it was meant to guard against missing Pod deletion events. We need to improve that regardless.

We should double check the history, but I think that in the past we were:

• asynchronously sending bind request
• updating the cache before sending the request
  and then the timeout was actually protecting us from the binding request actually not succeeding.

So I agree with Aldo - that we should double check if this timeout is at all needed currently.