add systemd as cgroup driver #27943

akhilerm · 2022-11-09T13:45:53Z

The containerd test jobs are now using cos-stable image now; which has cgroupv2 enabled by default. Therefore we need to add systemd as cgroup driver for all the tests which makes use of cos-stable.

Ref : #27912 (comment)

Signed-off-by: Akhil Mohan makhil@vmware.com

k8s-ci-robot · 2022-11-09T13:46:03Z

Hi @akhilerm. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

akhilerm · 2022-11-09T16:37:52Z

@endocrimes @adisky Can you take a look at this PR. Its related to setting cgroup driver to systemd, for all jobs for which image has been updated to cos-stable.

SergeyKanzhelev · 2022-11-09T18:19:49Z

add systemd as cgroup driver for all containerd jobs that make use of cos-stable image.

Can you please add more details on the motivation for this change.

SergeyKanzhelev · 2022-11-09T18:20:46Z

/assign @bobbypage

bobbypage · 2022-11-09T19:58:35Z

config/jobs/kubernetes/sig-node/containerd.yaml

@@ -17,7 +17,7 @@ presets:
  - name: NON_MASQUERADE_CIDR
    value: 0.0.0.0/0
  - name: KUBELET_TEST_ARGS
-    value: --runtime-cgroups=/system.slice/containerd.service
+    value: "--runtime-cgroups=/system.slice/containerd.service --cgroup-driver=systemd"


This assumes that all the jobs with preset-e2e-containerd should use systemd cgroup driver, I don't think this is accuarete.

For example the job below ci-containerd-e2e-ubuntu-gce uses the preset preset-e2e-containerd, but it's running on ubuntu-2004-lts which does not enable cgroupv2, so we shouldn't switch to use systemd cgroup driver.

We should not need this -- for cluster e2e tests, the GCE node bootstrap scripts already configure --cgroup-driver=systemd on cgroupv2 OS images (see https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/configure-helper.sh#L1630-L1633). So we only need to configure node e2e tests, not cluster e2e tests.

Should we have a separate label preset for those jobs then?

Just thinking out loud. . We can have a new preset for tests that use cos image and preset-e2e-containerd will be as is without any changes. Will it impact any other tests? Or that would be a huge change to do.

Should we have a separate label preset for those jobs then?

That's one option, we don't have too many cluster e2e tests in config/jobs/kubernetes/sig-node/containerd.yaml, so I think we can just also just update each them or setup a present and annotate with preset, up to you.

The important part is we only update the ones for systemd cgroup driver on the tests which are using cgroupv2 images.

Except these 4 jobs, other have either systemd driver set or is using an older cos image

ci-cos-containerd-node-e2e

ci-cos-containerd-node-e2e-features

ci-kubernetes-node-kubelet-lock-contention

ci-kubernetes-node-kubelet-credential-provider

The above jobs have multiple images (ubuntu and cos stable) specified in the image config. So I am not sure how to update the --cgroup-driver value for these jobs.

I think the general rule we can follow is:

If the job is pointing to an image config file that is using something older than COS M97 (or != cos-stable), then let's keep it as is for now

Otherwise, for jobs pointing to image config file with cos-stable, let's also update it to ubuntu 22.04 and also update cgroup driver to systemd.

config/jobs/kubernetes/sig-node/containerd.yaml

SergeyKanzhelev · 2022-11-15T00:24:53Z

/ok-to-test

SergeyKanzhelev · 2022-11-16T21:21:40Z

config/jobs/kubernetes/sig-release/release-branch-jobs/1.24.yaml

@@ -998,7 +998,7 @@ presubmits:
        - --gcp-zone=us-west1-b
        - '--node-test-args=--container-runtime-endpoint=unix:///run/containerd/containerd.sock
          --container-runtime-process-name=/usr/bin/containerd --container-runtime-pid-file=
-          --kubelet-flags="--cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/containerd.service"
+          --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/containerd.service"


changing previous release jobs sounds wrong. Let's have a separate copy of image config file for those locked on non cgroupv2 images

I made the change to the release jobs since the referring image file had changed.
But it seems like the jobs itself on those branches are not failing after the update to cos-stable. So I think this change wont be needed.

Also a bit confused that, ideally if the image is using cos-stable, and we are not setting cgroup driver to systemd (on both kubelet as well as containerd), it shouldnt be working correctly right?

cc: @bobbypage

Signed-off-by: Akhil Mohan <makhil@vmware.com>

update VM images to use cos-stable for testing containerd branches 1.5 and 1.6 Signed-off-by: Akhil Mohan <makhil@vmware.com>

Signed-off-by: Akhil Mohan <makhil@vmware.com>

akhilerm · 2022-12-01T06:16:44Z

/cc @bobbypage @SergeyKanzhelev

bobbypage · 2022-12-06T02:09:53Z

/lgtm

bobbypage · 2022-12-06T02:10:00Z

Thanks for the updates!

bobbypage · 2022-12-06T02:10:51Z

/assign @dims

dims · 2022-12-06T02:17:38Z

/approve

k8s-ci-robot · 2022-12-06T02:17:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: akhilerm, dims

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~config/jobs/kubernetes/sig-node/OWNERS~~ [dims]
~~jobs/e2e_node/OWNERS~~ [dims]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2022-12-06T02:34:44Z

@akhilerm: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

key containerd.yaml using file config/jobs/kubernetes/sig-node/containerd.yaml
key sig-node-presubmit.yaml using file config/jobs/kubernetes/sig-node/sig-node-presubmit.yaml

In response to this:

The containerd test jobs are now using cos-stable image now; which has cgroupv2 enabled by default. Therefore we need to add systemd as cgroup driver for all the tests which makes use of cos-stable.

Ref : #27912 (comment)

Signed-off-by: Akhil Mohan makhil@vmware.com

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

liggitt · 2022-12-06T17:26:02Z

it looks like https://testgrid.k8s.io/sig-node-containerd#pull-node-e2e went red right around when these two PRs merged:

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 9, 2022

k8s-ci-robot added area/config Issues or PRs related to code in /config area/jobs sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Nov 9, 2022

k8s-ci-robot requested a review from karan November 9, 2022 13:46

k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Nov 9, 2022

k8s-ci-robot requested a review from SergeyKanzhelev November 9, 2022 13:46

SergeyKanzhelev added this to Triage in SIG Node CI/Test Board Nov 9, 2022

SergeyKanzhelev added this to Triage in SIG Node PR Triage Nov 9, 2022

k8s-ci-robot assigned bobbypage Nov 9, 2022

SergeyKanzhelev moved this from Triage to PRs - Needs Reviewer in SIG Node CI/Test Board Nov 9, 2022

bobbypage mentioned this pull request Nov 9, 2022

ci: separarte pull-containerd-node-e2e for 1.5 branch #27912

Merged

bobbypage reviewed Nov 9, 2022

View reviewed changes

config/jobs/kubernetes/sig-node/containerd.yaml Outdated Show resolved Hide resolved

bobbypage reviewed Nov 9, 2022

View reviewed changes

config/jobs/kubernetes/sig-node/containerd.yaml Show resolved Hide resolved

akhilerm marked this pull request as ready for review November 11, 2022 11:43

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 11, 2022

k8s-ci-robot requested a review from bart0sh November 11, 2022 11:43

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 15, 2022

akhilerm removed request for karan and SergeyKanzhelev November 15, 2022 05:06

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. area/release-eng Issues or PRs related to the Release Engineering subproject sig/release Categorizes an issue or PR as relevant to SIG Release. labels Nov 16, 2022

SergeyKanzhelev reviewed Nov 16, 2022

View reviewed changes

akhilerm mentioned this pull request Nov 21, 2022

fix tests in sig-node related to CRIv1 update #28071

Merged

akhilerm added 4 commits November 22, 2022 13:40

add systemd as cgroup driver

5862522

Signed-off-by: Akhil Mohan <makhil@vmware.com>

update to cos-stable for containerd 1.6 and 1.5

4b739df

update VM images to use cos-stable for testing containerd branches 1.5 and 1.6 Signed-off-by: Akhil Mohan <makhil@vmware.com>

revert setting cgroup driver on preset-e2e-containerd

075e0dd

Signed-off-by: Akhil Mohan <makhil@vmware.com>

update systemd for all jobs using cos-stable

98f2595

Signed-off-by: Akhil Mohan <makhil@vmware.com>

akhilerm force-pushed the containerd-upgrade-systemd-cgroup branch from adc1455 to 98f2595 Compare November 24, 2022 06:26

revert release branch job updates

90088f0

Signed-off-by: Akhil Mohan <makhil@vmware.com>

k8s-ci-robot requested review from bobbypage and SergeyKanzhelev December 1, 2022 06:16

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 6, 2022

k8s-ci-robot assigned dims Dec 6, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 6, 2022

k8s-ci-robot merged commit afa6a41 into kubernetes:master Dec 6, 2022

SIG Node CI/Test Board automation moved this from PRs - Needs Reviewer to Done Dec 6, 2022

SIG Node PR Triage automation moved this from Triage to Done Dec 6, 2022

akhilerm deleted the containerd-upgrade-systemd-cgroup branch December 6, 2022 03:25

This was referenced Dec 6, 2022

Revert "add systemd as cgroup driver" #28189

Merged

REQUEST: New membership for akhilerm kubernetes/org#3889

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add systemd as cgroup driver #27943

add systemd as cgroup driver #27943

akhilerm commented Nov 9, 2022 •

edited

k8s-ci-robot commented Nov 9, 2022

akhilerm commented Nov 9, 2022

SergeyKanzhelev commented Nov 9, 2022

SergeyKanzhelev commented Nov 9, 2022

bobbypage Nov 9, 2022 •

edited

bobbypage Nov 9, 2022

akhilerm Nov 10, 2022

bobbypage Nov 11, 2022

akhilerm Nov 11, 2022 •

edited

bobbypage Nov 15, 2022 •

edited

SergeyKanzhelev commented Nov 15, 2022

SergeyKanzhelev Nov 16, 2022

akhilerm Nov 22, 2022

akhilerm commented Dec 1, 2022

bobbypage commented Dec 6, 2022

bobbypage commented Dec 6, 2022

bobbypage commented Dec 6, 2022

dims commented Dec 6, 2022

k8s-ci-robot commented Dec 6, 2022

k8s-ci-robot commented Dec 6, 2022

liggitt commented Dec 6, 2022

add systemd as cgroup driver #27943

add systemd as cgroup driver #27943

Conversation

akhilerm commented Nov 9, 2022 • edited

k8s-ci-robot commented Nov 9, 2022

akhilerm commented Nov 9, 2022

SergeyKanzhelev commented Nov 9, 2022

SergeyKanzhelev commented Nov 9, 2022

bobbypage Nov 9, 2022 • edited

Choose a reason for hiding this comment

bobbypage Nov 9, 2022

Choose a reason for hiding this comment

akhilerm Nov 10, 2022

Choose a reason for hiding this comment

bobbypage Nov 11, 2022

Choose a reason for hiding this comment

akhilerm Nov 11, 2022 • edited

Choose a reason for hiding this comment

bobbypage Nov 15, 2022 • edited

Choose a reason for hiding this comment

SergeyKanzhelev commented Nov 15, 2022

SergeyKanzhelev Nov 16, 2022

Choose a reason for hiding this comment

akhilerm Nov 22, 2022

Choose a reason for hiding this comment

akhilerm commented Dec 1, 2022

bobbypage commented Dec 6, 2022

bobbypage commented Dec 6, 2022

bobbypage commented Dec 6, 2022

dims commented Dec 6, 2022

k8s-ci-robot commented Dec 6, 2022

k8s-ci-robot commented Dec 6, 2022

liggitt commented Dec 6, 2022

akhilerm commented Nov 9, 2022 •

edited

bobbypage Nov 9, 2022 •

edited

akhilerm Nov 11, 2022 •

edited

bobbypage Nov 15, 2022 •

edited