[proposal] CPUSet injections to container runtime for fine-grained CPU orchestration

[zwzhang0107]

What is your proposal:
Setting cgroup/cpuset parameters for containers by pod QoS.

Why is this needed:
fine-grained CPU orchestration is supported by koord-scheduler, koordlet should apply the scheduling result to cgroup paramters. See #96 for more details.

Is there a suggested solution, if so, please add it:
rules of cpuset:

• LSE/LSR: follow the scheduling result field cpuset from the annotation scheduler.koordinator.sh/alloc-status of pod.
• LS: follow the scheduling result field numaNode or numaSocketfrom the annotation scheduler.koordinator.sh/alloc-status of pod if specified. Or using all cpu share pool if not.
• BE: set as be-cpu-share-pool
• cgroup cpuset should be updated once the alloc-status or share-pool update.

cpuset injections should be implemented as a plugin of runtime hooks; here are two kinds of container injections.

• create: parse cpuset from alloc-status or share-pool, should be injected by the interceptor of plugin.
• update: alloc-status updated or share-pool info update, this will trigger the update of statesinformer, plugin will be executed as a callback. this always happens if LSR/LSE pod rebinds, other share-pool updates due to the pod created/destroyed.

using cri request to modify cpuset is better

for origin k8s pods(without koord-QoS):
Guaranteed(without static): treat as LS.
Guaranteed(with static): follow the rules from /var/lib/kubelet/cpu_manager_state if present. Or treat as LS if not present.
Burstable: treat as LS.
Besteffort: treat as BE.

• v0.5 [proposal] CPUSet injections for pod with koord QoS #265
• v0.6 [proposal] koordlet compatibility the existing K8s Guaranteed Pods with static policy #346

[zwzhang0107]

/assign zwzhang0107

[zwzhang0107]

/area koordlet

[jasonliu747]

/close

[koordinator-bot]

@jasonliu747: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.