[BUG] unexpected cfs throttles on cpuset pods if cfs_quota_us is set

[saintube]

What happened:

When pods are scheduled and started with static CPU binding policy and Linux cpuset controller, we expect that the pod/containers can use all the cpuset CPUs and not get throttled by Linux CFS Bandwidth Control. However, there are results which pods still have chance to get cfs throttled if cpu.cfs_quota_us is set (according to container's CPU limit).

e.g.

Pod A:

      resources:
        limits:
          cpu: "1"
          memory: 2Gi
        requests:
          cpu: "1"
          memory: 2Gi

$ # print pod-level/container-level cfs_quota, cfs_period
$ cat /sys/fs/cgroup/cpu/kubepods.slice/kubepods-podxxx/cpu.cfs_quota_us
100000
$ cat /sys/fs/cgroup/cpu/kubepods.slice/kubepods-podxxx/cpu.cfs_period_us
100000
$ # print pod-level/container-level cpuset
$ cat /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-podxxx/cpuset.cpus
2
$ # run commands inside the pod
$ echo $$ > /sys/fs/cgroup/cpu/kubepods.slice/kubepods-podxxx/cgroup.procs
$ echo $$ > /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-podxxx/cgroup.procs
$ while :; do :; done
$ # check cpu throttled count number (nr_throttled)
$ cat /sys/fs/cgroup/cpu/kubepods.slice/kubepods-podxxx/cpu.stat
nr_periods 3076
nr_throttled 34
throttled_time 12070000

What you expected to happen:

cpuset pods should not have cfs throttles. We may disable cpu.cfs_quota_us for the cpuset pods via the koordlet.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• App version: 0.6.x
• Kubernetes version (use kubectl version): 1.22
• Install details (e.g. helm install args):
• Others:
  • OS Version: Centos 7.9 (kernel 3.10) & Anolis OS 7 (kernel 4.19)

FYI:

• CFS quotas can lead to unnecessary throttling kubernetes/kubernetes#67577
• torvalds/linux@512ac99

[saintube]

/area koordlet

[jasonliu747]

/assign @saintube