Raspberry Pi 5 - Fails to Enable Memory cgroup despite correct kernel parameters.

[Borillion]

Describe the bug

Despite correctly setting the cgroup_enable=memory kernel parameter to enable the memory cgroup subsystem for k3s installation, the system does not recognize or apply this setting. As a result, the memory cgroup is marked as disabled in /proc/cgroups, preventing k3s from starting.

Steps to reproduce the behaviour

Edit the /boot/firmware/cmdline.txt file to include the kernel boot parameters: cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
Reboot the system and attempt to start k3s, it does not start and fails with errors.

Device (s)

Raspberry Pi 5

System

Operating System: Raspberry Pi OS Lite 64 bit ( BookWorm )
Kernel Version: Linux pi-01 6.1.74-v8-16k+ #1725 SMP PREEMPT Mon Jan 22 13:38:39 GMT 2024 aarch64 GNU/Linux

Logs

Contents of file: /boot/firmware/cmdline.txt

console=serial0,115200 console=tty1 root=PARTUUID=147263d9-02 rootfstype=ext4 fsck.repair=yes rootwait cfg80211.ieee80211_regdom=US cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
File: /proc/cgroups

#subsys_name	hierarchy	num_cgroups	enabled
cpuset	0	39	1
cpu	0	39	1
cpuacct	0	39	1
blkio	0	39	1
memory	0	39	0
devices	0	39	1
freezer	0	39	1
net_cls	0	39	1
perf_event	0	39	1
net_prio	0	39	1
pids	0	39	1

journalctl output:

░░ The job identifier is 30165 and the job result is done.
Feb 07 00:17:24 pi-01 systemd[1]: Starting k3s.service - Lightweight Kubernetes...
░░ Subject: A start job for unit k3s.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit k3s.service has begun execution.
░░ 
░░ The job identifier is 30165.
Feb 07 00:17:24 pi-01 sh[8038]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Feb 07 00:17:24 pi-01 k3s[8042]: time="2024-02-07T00:17:24-08:00" level=info msg="Starting k3s v1.28.6+k3s2 (c9f49a3b)"
Feb 07 00:17:24 pi-01 k3s[8042]: time="2024-02-07T00:17:24-08:00" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Feb 07 00:17:24 pi-01 k3s[8042]: time="2024-02-07T00:17:24-08:00" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Feb 07 00:17:24 pi-01 k3s[8042]: time="2024-02-07T00:17:24-08:00" level=info msg="Database tables and indexes are up to date"
Feb 07 00:17:24 pi-01 k3s[8042]: time="2024-02-07T00:17:24-08:00" level=info msg="Kine available at unix://kine.sock"
Feb 07 00:17:24 pi-01 k3s[8042]: time="2024-02-07T00:17:24-08:00" level=info msg="Reconciling bootstrap data between datastore and disk"
Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorizat>
Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/sche>
Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Waiting for API server to become available"
Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=>
Feb 07 00:17:25 pi-01 k3s[8042]: I0207 00:17:25.209491    8042 options.go:220] external host was not specified, using 192.168.86.40
Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kube>
Feb 07 00:17:25 pi-01 k3s[8042]: I0207 00:17:25.210496    8042 server.go:156] Version: v1.28.6+k3s2
Feb 07 00:17:25 pi-01 k3s[8042]: I0207 00:17:25.210529    8042 server.go:158] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Server node token is available at /var/lib/rancher/k3s/server/token"
Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="To join server node to cluster: k3s server -s https://192.168.86.40:6443 -t ${SERVER_NODE_TOKEN}"
Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Agent node token is available at /var/lib/rancher/k3s/server/agent-token"
Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="To join agent node to cluster: k3s agent -s https://192.168.86.40:6443 -t ${AGENT_NODE_TOKEN}"
Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Run: k3s kubectl"
Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=fatal msg="failed to find memory cgroup (v2)"
Feb 07 00:17:25 pi-01 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd

Additional context

Updated the pi5's eeprom using rpi-update command, and rebooted, it had no effect.

sudo rpi-update
*** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
*** Performing self-update
*** Relaunching after update
*** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
FW_REV:d86b5843d68b9972a5430a6d3da1b271cfc83521
BOOTLOADER_REV:c987375f1d9633ea7d33d30f68a024055b76b9a7
*** We're running for the first time
*** Backing up files (this will take a few minutes)
*** Remove old firmware backup
*** Backing up firmware
*** Remove old modules backup
*** Backing up modules 6.1.0-rpi8-rpi-2712
WANT_32BIT:0 WANT_64BIT:1 WANT_PI4:1 WANT_PI5:1
#############################################################
WARNING: This update bumps to rpi-6.1.y linux tree
See: https://forums.raspberrypi.com/viewtopic.php?t=344246

'rpi-update' should only be used if there is a specific
reason to do so - for example, a request by a Raspberry Pi
engineer or if you want to help the testing effort
and are comfortable with restoring if there are regressions.

DO NOT use 'rpi-update' as part of a regular update process.
##############################################################
Would you like to proceed? (y/N)
Downloading bootloader tools
Downloading bootloader images
*** Downloading specific firmware revision (this will take a few minutes)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 144M 100 144M 0 0 21.2M 0 0:00:06 0:00:06 --:--:-- 25.9M
*** PREPARING EEPROM UPDATES ***

BOOTLOADER: update available
CURRENT: Fri 5 Jan 15:57:40 UTC 2024 (1704470260)
LATEST: Mon 5 Feb 14:38:34 UTC 2024 (1707143914)
RELEASE: latest (/lib/firmware/raspberrypi/bootloader-2712/latest)
Use raspi-config to change the release.
CURRENT: Fri 5 Jan 15:57:40 UTC 2024 (1704470260)
UPDATE: Mon 5 Feb 14:38:34 UTC 2024 (1707143914)
BOOTFS: /boot/firmware
'/tmp/tmp.tKmXhxVki6' -> '/boot/firmware/pieeprom.upd'
Copying recovery.bin to /boot/firmware for EEPROM update

EEPROM updates pending. Please reboot to apply the update.
To cancel a pending update run "sudo rpi-eeprom-update -r".
*** Updating firmware
*** Updating kernel modules
*** depmod 6.1.74-v8-16k+
*** depmod 6.1.74-v8+
*** Updating VideoCore libraries
*** Using SoftFP libraries
*** Updating SDK
*** Running ldconfig
*** Storing current firmware revision
*** Deleting downloaded files
*** Syncing changes to disk
*** If no errors appeared, your firmware was successfully updated to d86b5843d68b9972a5430a6d3da1b271cfc83521
*** A reboot is needed to activate the new firmware

All packages are up to date:
$ sudo apt update
Hit:1 http://deb.debian.org/debian bookworm InRelease
Hit:2 http://deb.debian.org/debian-security bookworm-security InRelease
Hit:3 http://deb.debian.org/debian bookworm-updates InRelease
Hit:4 http://archive.raspberrypi.com/debian bookworm InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.

[zhengyi13]

You're not the only one to notice this:

• CGROUPS not found on Raspberry Pi k3s-io/k3s#8890
• https://forums.raspberrypi.com/viewtopic.php?t=359827

Looks like the underlying issue is a change in the location for cmdline.txt.

ETA: well, I misread your initial report; my bad. If you've got that file in /boot/firmware as documented in https://www.raspberrypi.com/documentation/computers/configuration.html#the-boot-folder, I wonder what's broken?

[mrsauravsahu]

I was facing the same issue, when installing k3s on Rpi5

Updating my /boot/cmdline.txt to the snippet below fixed the issue, the pi is part of my cluster with k3s running.

console=serial0,115200 console=tty1 root=PARTUUID=ff577c67-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cgroup_memory=1 cgroup_enable=memory

[Borillion]

I was facing the same issue, when installing k3s on Rpi5

Updating my /boot/cmdline.txt to the snippet below fixed the issue, the pi is part of my cluster with k3s running.

console=serial0,115200 console=tty1 root=PARTUUID=ff577c67-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cgroup_memory=1 cgroup_enable=memory

Do you mean /boot/firmware/cmdline on Raspberry Pi OS ?

My /boot/cmdline.txt says not to edit the file and that the contents have moved.

Did you install any other packages?

[mrsauravsahu]

/boot/cmdline.txt

Yes looks like the file is actually a symlink to /boot/firmware/cmdline.txt/, and it is write protected. Edit it using root.

sudo vim /boot/cmdline.txt

I had also run these two commands but they didn't fix the issue.

sudo rpi-update
sudo rpi-eeprom-update

[Borillion]

😅 @mrsauravsahu , very strange. I was using sudo and it did not work. Mine are not symlinks.

I was able to have the kernel cmdline take effect with one important change in the method I used.
Edit the file directly on the microsd mounted on a different device. Using sudo to edit the file while the Pi is running the OS does nothing, the obvious hint in hindsight was the output from cat /proc/cmdline not having the options listed.

[lukfd]

I would like to clarify that the k3s documentation specify that it is a requirement to append cgroup_memory=1 cgroup_enable=memory to the /boot/cmdline.txt file. I had to make the cmdline.txt modification right after downloading the OS in the SD card.

[boshkash1151]

😅 @mrsauravsahu , very strange. I was using sudo and it did not work. Mine are not symlinks.

I was able to have the kernel cmdline take effect with one important change in the method I used. Edit the file directly on the microsd mounted on a different device. Using sudo to edit the file while the Pi is running the OS does nothing, the obvious hint in hindsight was the output from cat /proc/cmdline not having the options listed.

I confirm that this worked for me