Make volume binder resilient to races

[cofyc]

What type of PR is this?

Uncomment only one, leave it on its own line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

Make volume binder resilient to races between main schedule loop and async binding operation.

Which issue(s) this PR fixes:

Fixes #71928 #72013 #56236

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Make volume binder resilient to races between main schedule loop and async binding operation 

/sig storage
/sig scheduling

[cofyc]

/assign @msau42

Verified this can fix issue locally, but couldn't figure out how to write a test for it yet. Because this adds a new interface, please review first.

[msau42]

Hm another idea I had is to treat static binding and pending provisioning similarly in the predicate. Instead of failing the predicate if selectedNode is set, can we instead make the predicate return true only for the selectedNode, and false for all other nodes? AssumePodVolumes checks for if the PVCs are fully bound (which they are not) to determine whether or not to call BindPodVolumes.

This would be a bigger change, but I think would make our logic more resilient to these races.

[cofyc]

That makes sense and seems better! Always try to bind pod volumes if they are not fully bound.

[cofyc]

Instead of failing the predicate if selectedNode is set, can we instead make the predicate return true only for the selectedNode, and false for all other nodes? AssumePodVolumes checks for if the PVCs are fully bound (which they are not) to determine whether or not to call BindPodVolumes.

1. One thing is how to make API calls again when necessary. IMO there are two methods:

• Method 1: BindPodVolumes makes API calls whenever it find it is necessary to update API objects
• Method 2: BindPodVolumes makes API calls once for PVs and PVCs in binding cache, and fails with error when it find it is necessary to update API objects, let scheduler retry again

I'm using method 2.

2. Another thing is how to detect if it is necessary to update API objects

There is a special case, if there is an another instance of pod already in scheduling and find same bindings and selected node and assume same PVs and PVCs. API changes from apiserver will be overwritten.

At first, we make FindPodVolumes/AssumePodVolumes idempotent, do not fail with error (as you suggested). Then, BindPodVolumes detects periodically if it is necessary to update API objects:

• Method 1: simplest way is to fail whenever there is another schedule loop to find/assume pod volumes (make API calls for each new schedule, assuming new schedule means we need bind again
• Method 2: check PVs & PVCs against objects from apiserver, fail when they are out of date

I've tested method 1, but scheduler will retry pod repeatedly and make API calls every second (check period) currently, because there are two instances of pod in scheduler, each one will fail another instance of pod in binding operation.

I'm using method 2.

---

IMO, the simplest design is to avoid calling find/assume on pod when it is in binding, then bindings and assumed PVs/PVCs for assumed pod will not be changed in one scheduling cycle (if we think binding operation is part of scheduling cycle). Scheduling on assumed pod repeatedly is unnecessary, better to wait for pod is bound successfully or unassumed on failure.

Current scheduler does not support this. And we cannot do it in FindPodVolumes, because error in FindPodVolumes will make pod unscheduable. I'm investigating possibilities with new scheduler framework. Maybe we can achieve this in future.

[cofyc]

Scenarios

PVC selectedNode is rejected by provisioner

xref: #71928

BindPodVolumes will be retired by following cases:

• No pod is in scheduling. New PVC object from apiserver is added into pvc cache. BindPodVolumes find selectedNode does not exist, fail with an error on which scheduler will reschedule and make API calls again
• A pod is in scheduling. It will assume on new PVC object and update binding cache. However, BindPodVolumes will find PVs & PVCs in binding cache are out of date, fail with an error on which scheduler will reschedule and make API calls again

[cofyc]

@msau42 Squashed into logical commits (no content change):

• 13d87fb: Make volume binder resilient to races
• 8b94b96: Unit tests only
• cfc8ef5: Scheduler change
• 1a62f53: If provisioning PVC's PV is not found, check next time

[msau42]

/lgtm
/approve

[cofyc]

cc @k82cn @bsalamat
For approval on scheduler change

[cofyc]

/hold
for checking pod binding cache issues

[cofyc]

/hold cancel

xref: #72045 (comment)

[cofyc]

/priority important-soon

[bsalamat]

/approve

I do not know the volume binder logic very well, but based on your explanation that the volume binder cache needs to be kept only for unassigned pods, the changes in the scheduler code look good to me.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bsalamat, cofyc, msau42, xiaoxubeii

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/controller/volume/persistentvolume/OWNERS [msau42]
• pkg/scheduler/OWNERS [bsalamat]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[liggitt]

CI failures around PV binding sharply increased between 1/11 and 1/12:

• https://storage.googleapis.com/k8s-gubernator/triage/index.html#e927cd3b7abc588dea28
• https://storage.googleapis.com/k8s-gubernator/triage/index.html#2a1eabc27e9e628f3d83

can we determine if this PR contributed and consider rollback/rework if so?

[msau42]

With a quick glance, I don't think so. It seems like only containerd jobs with CSI hostpath or intree nfs drivers have spiked. Also some of the nfs tests that are failing don't use PVs. I more suspect an issue with containerd

[liggitt]

good observation. opened #72863

[bsalamat]

We are also seeing an increase in scheduling latency right after this PR. The scheduler predicate evaluation latency has increase by about 50%.

[cofyc]

hi, @bsalamat
Could you give me your benchmark commands? I'd like to debug and optimize FindPodVolumes which is called by predicate.

[cofyc]

I guess it is because new FindPodVolumes in this PR needs to clear old pod binding cache, and to achieve this I update pod cache even if there is no PVCs to bind/provision. There are some optimizations we can do. Sorry about increased scheduling latency. I'm working on a fix.

[bsalamat]

@cofyc You can go the our perf dashboard and then choose:
gce-100 nodes > Scheduler > Scheduling Latency > predicate evaluation
to see the results of latest runs.

[cofyc]

Thanks!