Fix Kubelet and Containerd when using cgroupfs

[pasqualet]

What type of PR is this?

/kind bug

What this PR does / why we need it:

Fix the usage of cgroupfs with containerd

Which issue(s) this PR fixes:

Fixes #8122

Special notes for your reviewer:

You can test the PR with the following variables:

container_manager: containerd
containerd_use_systemd_cgroup: false

Does this PR introduce a user-facing change?:

[containerd] Fix the usage of cgroupfs with containerd and introduce cgroupsfs specific variables (⚠️ `containerd_runtimes` is now `containerd_additional_runtimes` )

[cristicalin]

Would you mind adding a CI job for this configuration and also adding a point to the docs about why/when this would be needed? I have in mind especially the discussion on #8068

[fungusakafungus]

We use

kubelet_runtime_cgroups: /system.slice/containerd.service
kubelet_kubelet_cgroups: /system.slice/kubelet.service

and systemCgroups: /system.slice in kubelet_config_extra_args
with that containerd configuration

container_manager: containerd
containerd_use_systemd_cgroup: false

[pasqualet]

@fungusakafungus Are you also setting cgroupRoot? As far I can see from the doc, this is a required field when systemCgroups is not empty.

[fungusakafungus]

@fungusakafungus Are you also setting cgroupRoot? As far I can see from the doc, this is a required field when systemCgroups is not empty.

ah, yes, we set it to cgroupRoot: /

[cristicalin]

I tested this on top of the Kata PR #8068 and I managed to successfully create pods with kata 2.2.2.

[cristicalin]

@pasqualet since you are replacing the user facing inventory variable containerd_runtimes with containerd_additional_runtimes I suggest you update our docs with the relevant description.

[pasqualet]

I've updated default values and the documentation.

[fungusakafungus]

This removes configurability for the default runc runtime, I was just going to use that...

[fungusakafungus]

Alternatively, we could set defaults depending on kubelet_cgroup_driver|default(kubelet_cgroup_driver_detected) in defaults.yml like

# Set runtime cgroups
kubelet_runtime_cgroups: >-
  {%- if kubelet_cgroup_driver|default(kubelet_cgroup_driver_detected) == "systemd" -%}
  /systemd/system.slice
  {%- else -%}
  /system.slice/containerd.service
  {%- endif -%}

[fungusakafungus]

Are you sure about the /systemd/ prefix? For us /system.slice/containerd.service works better(we have containerd metrics via cAdvisor from kubelet), I also just found that value here: https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/util.sh#L827

[pasqualet]

You are right, it was a mistake. Fixed.

[pasqualet]

This removes configurability for the default runc runtime, I was just going to use that...

I have created the variable named containerd_runc_runtime to cusotmize the runc runtime.

[cristicalin]

Nice work @pasqualet! Thanks for fixing this!

/lgtm

[floryut]

Thanks, reminder for release note and people dropping here
containerd_runtimes no longer exists and containerd_additional_runtimes should now be used.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: floryut, pasqualet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [floryut]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[floryut]

This also introduce 3 new variables when using cgroupfs

kubelet_runtime_cgroups_cgroupfs
kubelet_kubelet_cgroups_cgroupfs
kubelet_config_extra_args_cgroupfs

Will try to remember that for the release note.