vmm process was incorrectly added to kata_sandbox cgroup in cgroup v2

[yaoyinnan]

After #4397 was merged, Kata can run on systems using cgroup v2. When the system uses cgroup v2, the vmm process will be incorrectly added to kata_sandbox.

root@k8s-worker01:/opt/shell# ps -A
  34962 ?        00:00:00 containerd-shim
  34971 ?        00:00:00 virtiofsd
  34977 ?        00:00:03 qemu-system-x86
  34979 ?        00:00:00 virtiofsd
  34982 ?        00:00:00 vhost-34977
  34985 ?        00:00:00 kvm-pit/34977
  34986 ?        00:00:00 vhost-34977
  34997 pts/0    00:00:00 ps

root@k8s-worker01:/opt/shell# cat /proc/{34962,34977,34971,34979}/cgroup
0::/kata_overhead/84f329f0d35ce26872da8296b441f0474721f083e73e9c82decd971a6ce5f3da
0::/kubepods/besteffort/podf5199815-6dd7-4250-8f04-69b89527bad7/kata_84f329f0d35ce26872da8296b441f0474721f083e73e9c82decd971a6ce5f3da
0::/kata_overhead/84f329f0d35ce26872da8296b441f0474721f083e73e9c82decd971a6ce5f3da
0::/kata_overhead/84f329f0d35ce26872da8296b441f0474721f083e73e9c82decd971a6ce5f3da

root@k8s-worker01:/opt/shell# ps -eT | grep 34977
  34977   34977 ?        00:00:00 qemu-system-x86
  34977   34978 ?        00:00:00 qemu-system-x86
  34977   34983 ?        00:00:00 qemu-system-x86
  34977   34984 ?        00:00:27 qemu-system-x86
  34982   34982 ?        00:00:00 vhost-34977
  34985   34985 ?        00:00:00 kvm-pit/34977
  34986   34986 ?        00:00:00 vhost-34977

root@k8s-worker01:/opt/shell# cat /proc/{34977,34978,34983,34984}/cgroup
0::/kubepods/besteffort/podf5199815-6dd7-4250-8f04-69b89527bad7/kata_84f329f0d35ce26872da8296b441f0474721f083e73e9c82decd971a6ce5f3da
0::/kubepods/besteffort/podf5199815-6dd7-4250-8f04-69b89527bad7/kata_84f329f0d35ce26872da8296b441f0474721f083e73e9c82decd971a6ce5f3da
0::/kubepods/besteffort/podf5199815-6dd7-4250-8f04-69b89527bad7/kata_84f329f0d35ce26872da8296b441f0474721f083e73e9c82decd971a6ce5f3da
0::/kubepods/besteffort/podf5199815-6dd7-4250-8f04-69b89527bad7/kata_84f329f0d35ce26872da8296b441f0474721f083e73e9c82decd971a6ce5f3da

As expected, when using sandbox_cgroup_only = false, processes(vmm, containerd-shim, virtiofsd) other than vCPU threads should be moved to the kata_overhead cgroup for management. For detailed design content, please refer to https://github.com/kata-containers/kata-containers/blob/main/docs/design/host-cgroups.md#sandbox_cgroup_only--false-default-setting.

This problem does not exist in cgroup v1, because cgroup v1 allows processes and their child threads to be moved to different cgroups. In cgroup v2, because threads can only be added to the cgroup where the process is located or the sub-cgroup of the process, we can no longer move processes other than vCPU threads to kata_overhead according to the host cgroup design scheme.

In this case, since the vmm (qemu-system-x86) process occupies a lot of resources, especially memory resources, if they are placed in the kata_sandbox cgroup, it may cause oom.

We need to discuss how to adjust the scheme of host cgroup management. It may be discussed in the following ways:

• The vmm process is placed in kata_sandbox.
• The vmm process is placed in kata_overhead along with the vCPU process.
• Or is there any other way to make processes and threads under cgroup v2 placed under different cgroups.

@sameo @liubin @snir911 Please give me some suggestion to fix this problem, thanks.

[liubin]

One main change of cgroup v2 is the control granularity, v2 is mainly processes target but not threads. But CPU controller requires thread mode.

These are supported threaded controller:

• cpu
• cpuset
• perf_event
• pids

Cgroup v2 supports "Threaded subtree", so put overhead and sandbox cgroups into one threaded subtree may fix this issue:

[fidencio]

cc @egernst