Default cgroup driver to systemd from k8s 1.20

[bharath-123]

Currently, kOps uses cgroupfs cgroup driver for the kubelet and CRIs. This PR defaults
the cgroup driver to systemd for clusters created with k8s versions >= 1.20.

Using systemd as the cgroup-driver is the recommended way as per
https://kubernetes.io/docs/setup/production-environment/container-runtimes/

It would be really great if this can be tested by others :) !! I have tested this by bring up and tearing down a whole bunch of clusters :) .

I verified that systemd was being used by checking the memory cgroup hierarchy. I was able to see the top level kubepod.slice hierarchy. If cgroupfs driver is used then the directory would be named kubepod instead of kubepod.slice.

Please do let me know if I have missed anything major here. Getting containerd working was very tricky since we don't have a TOML parser in kOps.

Fixes: #10372

[k8s-ci-robot]

Hi @bharath-123. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bharath-123]

cc: @hakman @bmelbourne @olemarkus

[bharath-123]

as usual, i have forgotten to run make gofmt and static checks. Need to add them to my pre push commits :)

[bharath-123]

I think this can reviewed. Would love to get some feedback on this. Made this a WIP mainly so that we can test this out since this can break kOps clusters if not done right.

[rifelpet]

/ok-to-test

[hakman]

@bharath-123 Thanks for digging into this. It looks mostly good at first sight. Will need some more time to review it and not sure if I can get to it until next week.

One thought is that kOps can use at the moment containerd 1.2+ and this PR introduces 2 things that may be incompatible with older containerd versions:

• config version 2
• runtime version 2 (not sure when it was introduced, but was made default only in 1.4)

I am considering dropping support for 1.2, as it was never advertised as supported or used anywhere.

[bmelbourne]

@bharath-123 Thanks for digging into this. It looks mostly good at first sight. Will need some more time to review it and not sure if I can get to it until next week.

One thought is that kOps can use at the moment containerd 1.2+ and this PR introduces 2 things that may be incompatible with older containerd versions:

• config version 2
• runtime version 2 (not sure when it was introduced, but was made default only in 1.4)

I am considering dropping support for 1.2, as it was never advertised as supported or used anywhere.

@hakman @bharath-123
I've actually started working on a similar update for Kubespray where the containerd version is being bumped to 1.4.3, and could incorporate the containerd config v2 into PR #10370...just let me know your preference?

[bharath-123]

Seems I don't get emails anymore regarding github! Missed these convos!

@hakman I think we can drop support for containerd 1.2. It has reached EOL https://containerd.io/releases/

@bmelbourne will have a look at your PR!

[bmelbourne]

As you're switching to the default CRI plugin runtime io.containerd.runc.v2 for Containerd 1.4, the SystemdCgroup option can be removed as it's only relevant for runtime io.containerd.runtime.v1.linux.

https://github.com/containerd/containerd/blob/master/pkg/cri/config/config.go#L213-L216

[bharath-123]

got it! Thanks for this @bmelbourne !

[bmelbourne]

As you're switching to the default CRI plugin runtime io.containerd.runc.v2 for Containerd 1.4, the SystemdCgroup option can be removed as it's only relevant for runtime io.containerd.runtime.v1.linux.

https://github.com/containerd/containerd/blob/master/pkg/cri/config/config.go#L213-L216

[hakman]

@bharath-123 @bmelbourne sorry for not answering sooner. I spent some time during the holidays to simplify the code and remove the unsupported parts like kubenet and 1.2.x.
Also had some time to think about this and the settings. We should still keep io.containerd.runc.v2 to set this for 1.3.x.
@bharath-123 if you could update the PR, I can take a look at it and review it these days.

[bharath-123]

Sure @hakman. I ll update this PR by tommorow for a review.

[bharath-123]

I am not sure if this would break existing clusters running on runc runtime type v1 since containerd v < 1.4 run on runtime v1? I can't confirm that kubelet decides which cgroup-driver is used tbh. I have put a message in #sig-node and #containerd to confirm from the cri and containerd maintainers.

[bharath-123]

Based on discussions on slack, we will be defaulting to runtime v2 on containerd. Have pushed the latest code. Would love a round of review :) @bmelbourne @hakman @olemarkus

[hakman]

Time to see it in action. Thanks @bharath-123!
/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bharath-123, hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [hakman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hakman]

/lgtm

[bharath-123]

/retest