Considering PVC as a resource for balanced resource utilization in the scheduler

[abhgupta]

Is this a BUG REPORT or FEATURE REQUEST?:
/kind feature

What happened:
The scheduler currently does not have a priority function to properly spread out pods that request PVCs. In the absence of such a priority function, scheduling of pods with PVCs could be skewed resulting in:

• some nodes exhaust their allocatable PVC limits, while still having enough schedulable cpu/memory resources
• some nodes have enough available PVC mount points, but have exhausted their schedulable cpu/memory resources

If the above happens, we get into a state of inefficient node utilization.

What you expected to happen:
PVCs should be considered as a resource within the balanced resource utilization (BRU) priority function. Instead of the current algorithm in place within the BRU priority function, we could consider using standard deviation to allow more than 2 resources to be balanced across nodes. Input values for the standard deviation calculation could be the fractions/ratios of the scheduled->capacity for the resources (memory, cpu, and pvc).

Alternately, a separate priority function could be considered for just PVCs - whatever makes more sense.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:
The PVC resource should also be considered in the "least requested" priority function.

Environment:

• Kubernetes version (use kubectl version): Any
• Cloud provider or hardware configuration: AWS/Azure/GCP
• OS (e.g. from /etc/os-release): Any
• Kernel (e.g. uname -a): Any
• Install tools: N/A
• Others:

[abhgupta]

cc @derekwaynecarr @aveshagarwal

[abhgupta]

/sig scheduling

[resouer]

But PVC is not bound to node, not very sure about how the spreading here works @abhgupta Mind to explain a little bit?

[timothysc]

@abhgupta are you just looking for a count-leveling? So distributions of PVC's are not lumpy? If so, that seems pretty trivial.

[k82cn]

some nodes exhaust their allocatable PVC limits

@abhgupta , any more detail about PVC limits? My understanding is that PVC are NOT ALWAYS related to node.

[ravisantoshgudimetla]

@k82 - IIUC, what @abhgupta wants is - a node which has balanced CPU, memory and number of volumes attached.
I think
We can modify the balanced resource allocation priority to compute standard deviation or variance of (CPU Fraction, Memory Fraction, (volumes_mounted_till_now_on_node+requested)/max_no_of_vols_allowed) for every node. Whichever node has least standard deviation will have high priority.

[k82cn]

number of volumes attached.

Any impact if we have a bigger number, performance or max mount points ? If max mount points, that's interesting point that how many pods will run in each node :)

Another case in my mind is DFS (distributed file system) client issue : if so many Pod read/write data into dfs mount point, the connection maybe a bottleneck.

No objection, just would like to know the detail of case before we build a priorities for it :).

[k82cn]

/cc @bsalamat @kubernetes/sig-storage-feature-requests

[abhgupta]

@ravisantoshgudimetla Thanks! That exactly captures the gist of my proposal. We could ignore PVs (for storage classes) that do not have an upper limit. Such PVs (with no upper limits) could simply be dealt with the "least requested" priority function.

If I understand what @k82cn was highlighting
There are multiple scheduling issues with PVCs

1. Imbalance with just the number of PVs mounted (on pods) on each node

• This is the low hanging fruit that I wish to target

2. Imbalance with the network I/O for data read/writes from all pods on a node to PVs

• This is an important issue, but more complicated and I wouldn't want to club it in this issue.

[msau42]

This is also related to #24317. Bigger nodes can have higher limits on the max number of volumes attached. An even count spreading assumes that all nodes are the same size.

[ravisantoshgudimetla]

@k82cn

Any impact if we have a bigger number, performance or max mount points ? If max mount points, that's interesting point that how many pods will run in each node :)

I think as of now, we are limited by 39 for AWS and 16 for GCE which are hardcoded from scheduling perspective with users being able to specify through environment variable, so the numbers aren't that high. I can create a benchmark to test this.

@msau42 - Thanks for pointing to the issue where node sizes vary. I think as long as I rely on the variable that uses configMap(#53461 (comment)), we should be good to go.

Bigger nodes can have higher limits on the max number of volumes attached. An even count spreading assumes that all nodes are the same size.

I think this statement holds good for memory and CPU as well. AFAIU, a balanced node is one which has memory utilization(requested/capacity), CPU utilization close to each other meaning variance is less irrespective of size. Won't this be same be same for volumes attached? Or is your statement based on not using configmap which has information on cloud provider specific value.

[bsalamat]

PVCs should be considered as a resource within the balanced resource utilization (BRU) priority function.

Are you using PVCs and attached volumes interchangeably here? AFAIK, PVCs should first be processed and result in an attached volume before the Pod can be scheduled.

[msau42]

Right now the scheduler does not cache number of volumes attached per node. The predicate just walks through all Pods and calculates the volume counts per node. In general, there is also a lot of churn in this area:

• Volume count predicate is being redesigned to handle variable limits per node
• Most volume plugins are starting to get migrated to use the CSI plugin interface, which has a VolumeAttachement object that the scheduler may be able to leverage better. CSI may also be enhanced to better support reporting the node limits.

[k82cn]

CSI may also be enhanced to better support reporting the node limits.

+1, it's also better to enhance scheduler for general purpose, e.g. CSI, instead of hardcode :).

Overall, that's a reasonable feature to me :).