k3s on pi error - cgroup_memory=1 cgroup_enable=memory #2067

plsnotracking · 2020-07-26T05:47:01Z

Have been trying to run k3s on Raspberry Pi 3 Model B Rev 1.2 (BCM2835) but have been unsuccessful.

My OS:

Raspbian GNU/Linux 10 (buster)
Linux kube-master 5.4.51-v7+ #1327 SMP Thu Jul 23 10:58:46 BST 2020 armv7l GNU/Linux

level=info msg="Run: k3s kubectl"
Jul 26 06:34:05 kube-master k3s[943]: time="2020-07-26T06:34:05.256984297+01:00" level=info msg="k3s is up and running"
Jul 26 06:34:05 kube-master k3s[943]: time="2020-07-26T06:34:05.257725574+01:00" level=error msg="Failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"
Jul 26 06:34:05 kube-master k3s[943]: time="2020-07-26T06:34:05.257859589+01:00" level=fatal msg="failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"
Jul 26 06:34:05 kube-master systemd[1]: Started Lightweight Kubernetes.
Jul 26 06:34:05 kube-master systemd[1]: Started Lightweight Kubernetes.
Jul 26 06:34:05 kube-master systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Jul 26 06:34:05 kube-master systemd[1]: k3s.service: Failed with result 'exit-code'.
Jul 26 06:34:10 kube-master systemd[1]: k3s.service: Service RestartSec=5s expired, scheduling restart.
Jul 26 06:34:10 kube-master systemd[1]: k3s.service: Scheduled restart job, restart counter is at 1.
Jul 26 06:34:10 kube-master systemd[1]: Stopped Lightweight Kubernetes.
Jul 26 06:34:10 kube-master systemd[1]: Starting Lightweight Kubernetes...

cat /boot/cmdline.txt
console=serial0,115200 console=tty1 root=PARTUUID=5b7d0535-02 rootfstype=ext4 cgroup_memory=1 cgroup_enable=memory elevator=deadline fsck.repair=yes rootwait

I've tried putting the at cgroup_memory=1 cgroup_enable=memory at the end of the file too, has changed nothing for me. I've rebooted the pi several times as well.

It'd be great if someone could point me in the right direction, thank you.

The text was updated successfully, but these errors were encountered:

mkenne11 · 2020-07-26T22:29:27Z

@plsnotracking I had the same issue on my Raspberry Pi 4 Model Bs after a recent eeprom & kernel update. Btw I'm also running k3s on Raspbian GNU/Linux 10 (buster).

My k3s instances resolved the issue after adding cgroup_memory=1 cgroup_enable=memory to the end of my cat /boot/cmdline.txt file.

klevermonicker · 2020-07-26T23:14:09Z

So this is happening to me specifically on the pi3's i had in my k3s cluster. My pi4's are perfectly fine with it, and i have this in /boot/cmdline.txt
console=serial0,115200 console=tty1 root=PARTUUID=738a4d67-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
it would appear that the order may work on pi4 but not pi3. i have yet to find a fix :( Granted, i am replacing my 3's with 4's for other reasons but that's a bit disheartening

plsnotracking · 2020-07-27T00:07:19Z

The latest kernel updates broke the ‘cgroup’ changes - https://www.raspberrypi.org/forums/viewtopic.php?t=203128

I eventually ended up installing the beta 64bit image raspian os.

11jwolfe2 · 2020-08-04T20:44:06Z

Same issue here! Tried to re install my Rpi3s and no luck, had an old pi 3 running previous release as soon as I did an update/upgrade. It no longer works. Going to try and use the 64bit and see how it goes.

brandond · 2020-08-04T20:50:03Z

The 64bit kernel does appear to resolve the issue, but I have heard anecdotally that performance on older devices can be a bit lacking. You might be better off sticking with an older 32bit OS on your Pi until they fix the cgroups.

brandond · 2020-12-04T23:24:44Z

I believe that all the Pi kernels now support the correct cgroups.

Bjohnson131 · 2021-03-17T02:47:40Z

I believe that all the Pi kernels now support the correct cgroups.

I'm having this issue on ubuntu 20.04

Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04

uname -a:

Linux ubuntu 5.4.0-1030-raspi #33-Ubuntu SMP PREEMPT Wed Feb 24 11:20:11 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

pomcho555 · 2021-03-18T07:53:23Z

I am having this issue on Ubuntu 20.04.

uname -a:
Linux pi4-node2 5.4.0-1030-raspi #33-Ubuntu SMP PREEMPT Wed Feb 24 11:20:11 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

And I SOLVED it with the following process.

sudo vim /boot/firmware/cmdline.txt

Add cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 into end of the file.

jazoom · 2021-04-03T04:32:06Z

I just started getting this error on Solus. I don't know enough about it to suggest why that might be the case, but the fix for Ubuntu did not work.

alanbchristie · 2021-07-10T17:31:22Z

Running v1.21.2+k3s1 on a RPi 3 Model B V1.2 and latest RPi OS Linux 5.10.17-v7+ #1421 SMP Thu May 27 13:59:01 BST 2021 armv7l GNU/Linux and also encountered this error. As @pomcho555 pointed out it's fixed by editing cmdline.txt. To be clear /boot/cmdline.txt on the device is a single line of text and editing it to the following, and rebooting, fixed it for me...

console=serial0,115200 console=tty1 root=PARTUUID=b9075e12-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

nodesocket · 2022-05-25T02:16:53Z

I'm also having issues related to cgroups with Kubernetes running on a raspberry pi. kubectl top pods does not work for example. Also, /var/log/syslog is filled with warnings and notices about cgroups.

pi@kube-worker-kw0me:~ $ sudo cat /boot/cmdline.txt
console=serial0,115200 console=tty1 root=PARTUUID=59ff6e86-02 rootfstype=ext4 fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

pi@kube-master:~ $ uname -a
Linux kube-master 5.15.32-v8+ #1538 SMP PREEMPT Thu Mar 31 19:40:39 BST 2022 aarch64 GNU/Linux

pi@kube-master:~ $ kubectl get nodes -o wide
NAME                STATUS   ROLES                  AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
kube-master         Ready    control-plane,master   177d   v1.23.3   10.0.0.6      <none>        Debian GNU/Linux 11 (bullseye)   5.15.32-v8+      docker://20.10.16
kube-worker-5elgu   Ready    <none>                 177d   v1.23.3   10.0.0.8      <none>        Debian GNU/Linux 11 (bullseye)   5.15.32-v8+      docker://20.10.16
kube-worker-bjcfq   Ready    <none>                 177d   v1.23.3   10.0.0.9      <none>        Debian GNU/Linux 11 (bullseye)   5.15.32-v8+      docker://20.10.16
kube-worker-kw0me   Ready    <none>                 177d   v1.23.3   10.0.0.7      <none>        Debian GNU/Linux 11 (bullseye)   5.15.32-v8+      docker://20.10.16

pi@kube-worker-kw0me:~ $ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)

Server:
 Containers: 45
  Running: 22
  Paused: 0
  Stopped: 23
 Images: 19
 Server Version: 20.10.16
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
 runc version: v1.1.1-0-g52de29d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.32-v8+
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.678GiB
 Name: kube-worker-kw0me
 ID: L2BN:JM3F:573K:PIVH:NNIE:JZVA:U5LG:7YKR:ZV3Q:DYKD:KRLK:UNHF
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

brandond · 2022-05-25T02:43:14Z

@nodesocket you don't appear to be using k3s?

Cesarsk · 2022-08-27T22:15:25Z

In my case, raspberry pi4, raspberry pi os lite 64 bit, k3s, after adding the options suggested inside /boot/cmdline.txt and rebooting the cluster, it worked.

driversti · 2023-08-25T11:41:59Z

For those using DietPi on RPI4:
add cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory at the end of /boot/cmdline.txt

plsnotracking mentioned this issue Jul 29, 2020

Installation on Raspberry Pi 3 Model B fails #2071

Closed

brandond closed this as completed Dec 4, 2020

louis030195 mentioned this issue Feb 14, 2021

Job for k3s.service failed because the control process exited with error code. #1011

Closed

brandond mentioned this issue May 6, 2021

Install fails on Raspberry Pi 4 2021-03-04-raspios-buster-armhf-lite #3283

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

k3s on pi error - cgroup_memory=1 cgroup_enable=memory #2067

k3s on pi error - cgroup_memory=1 cgroup_enable=memory #2067

plsnotracking commented Jul 26, 2020

mkenne11 commented Jul 26, 2020 •

edited

klevermonicker commented Jul 26, 2020

plsnotracking commented Jul 27, 2020

11jwolfe2 commented Aug 4, 2020

brandond commented Aug 4, 2020

brandond commented Dec 4, 2020

Bjohnson131 commented Mar 17, 2021 •

edited

pomcho555 commented Mar 18, 2021

jazoom commented Apr 3, 2021 •

edited

alanbchristie commented Jul 10, 2021

nodesocket commented May 25, 2022 •

edited

brandond commented May 25, 2022

Cesarsk commented Aug 27, 2022

driversti commented Aug 25, 2023 •

edited

k3s on pi error - cgroup_memory=1 cgroup_enable=memory #2067

k3s on pi error - cgroup_memory=1 cgroup_enable=memory #2067

Comments

plsnotracking commented Jul 26, 2020

mkenne11 commented Jul 26, 2020 • edited

klevermonicker commented Jul 26, 2020

plsnotracking commented Jul 27, 2020

11jwolfe2 commented Aug 4, 2020

brandond commented Aug 4, 2020

brandond commented Dec 4, 2020

Bjohnson131 commented Mar 17, 2021 • edited

pomcho555 commented Mar 18, 2021

jazoom commented Apr 3, 2021 • edited

alanbchristie commented Jul 10, 2021

nodesocket commented May 25, 2022 • edited

brandond commented May 25, 2022

Cesarsk commented Aug 27, 2022

driversti commented Aug 25, 2023 • edited

mkenne11 commented Jul 26, 2020 •

edited

Bjohnson131 commented Mar 17, 2021 •

edited

jazoom commented Apr 3, 2021 •

edited

nodesocket commented May 25, 2022 •

edited

driversti commented Aug 25, 2023 •

edited