[error] OCI runtime create failed: setting cgroup config for procHooks process caused: load program: invalid argument #2959

paulo-erichsen · 2021-05-24T16:04:06Z

Hello, I just upgraded Arch Linux ARM yesterday and after rebooting I can no longer run docker containers and I'm wondering if anyone can help

error

$ sudo docker run hello-world
docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: load program: invalid argument: unknown.
ERRO[0000] error waiting for container: context canceled

journalctl logs for docker.service when the above command is run

May 20 19:26:32 odroid dockerd[7086]: time="2021-05-20T19:26:32.350386235Z" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/a875a104734247b216a8da518b8e6ecd12a8776588b4b86b210f52bbfb80287e pid=7924
May 20 19:26:32 odroid dockerd[7086]: time="2021-05-20T19:26:32.520339498Z" level=info msg="shim disconnected" id=a875a104734247b216a8da518b8e6ecd12a8776588b4b86b210f52bbfb80287e
May 20 19:26:32 odroid dockerd[7086]: time="2021-05-20T19:26:32.520569462Z" level=warning msg="cleaning up after shim disconnected" id=a875a104734247b216a8da518b8e6ecd12a8776588b4b86b210f52bbfb80287e namespace=moby
May 20 19:26:32 odroid dockerd[7086]: time="2021-05-20T19:26:32.520627755Z" level=info msg="cleaning up dead shim"
May 20 19:26:32 odroid dockerd[7086]: time="2021-05-20T19:26:32.549147915Z" level=warning msg="cleanup warnings time=\"2021-05-20T19:26:32Z\" level=info msg=\"starting signal loop\" namespace=moby pid=7976\n"
May 20 19:26:32 odroid dockerd[7086]: time="2021-05-20T19:26:32.550299904Z" level=error msg="copy shim log" error="read /proc/self/fd/13: file already closed"
May 20 19:26:32 odroid dockerd[7054]: time="2021-05-20T19:26:32.551576187Z" level=error msg="stream copy error: reading from a closed fifo"
May 20 19:26:32 odroid dockerd[7054]: time="2021-05-20T19:26:32.551826860Z" level=error msg="stream copy error: reading from a closed fifo"
May 20 19:26:32 odroid dockerd[7054]: time="2021-05-20T19:26:32.719594566Z" level=error msg="a875a104734247b216a8da518b8e6ecd12a8776588b4b86b210f52bbfb80287e cleanup: failed to delete container from containerd: no such container"
May 20 19:26:32 odroid dockerd[7054]: time="2021-05-20T19:26:32.719712902Z" level=error msg="Handler for POST /v1.41/containers/a875a104734247b216a8da518b8e6ecd12a8776588b4b86b210f52bbfb80287e/start returned error: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: load program: invalid argument: unknown"

docker version

$ sudo docker version
Client:
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.16.3
 Git commit:        370c28948e
 Built:             Mon Apr 12 17:03:09 2021
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.3
  Git commit:       8728dd246c
  Built:            Mon Apr 12 17:02:33 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          v1.5.1
  GitCommit:        12dca9790f4cb6b18a6a7a027ce420145cb98ee7.m
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

sudo docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-tp-docker)

Server:
 Containers: 12
  Running: 0
  Paused: 0
  Stopped: 12
 Images: 18
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 12dca9790f4cb6b18a6a7a027ce420145cb98ee7.m
 runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 4.9.219-1-ARCH
 Operating System: Arch Linux ARM
 OSType: linux
 Architecture: aarch64
 CPUs: 6
 Total Memory: 3.623GiB
 Name: odroid
 ID: HCSN:DECD:NVM3:7KPL:W36N:IKH3:HL3F:GGER:XQCD:UZGV:Y65O:3QX6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support

runc --debug

$ sudo runc --debug --systemd-cgroup run test
DEBU[0000]github.com/opencontainers/runc/libcontainer/cgroups/fscommon/open.go:37 github.com/opencontainers/runc/libcontainer/cgroups/fscommon.prepareOpenat2.func1() openat2 not available, falling back to securejoin 
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec[19194]: => nsexec container setup     
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[19194]: ~> nsexec stage-0           
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[19194]: spawn stage-1               
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[19194]: -> stage-1 synchronisation loop 
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-1[19206]: ~> nsexec stage-1           
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-1[19206]: unshare remaining namespace (except cgroupns) 
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-1[19206]: spawn stage-2               
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-1[19206]: request stage-0 to forward stage-2 pid (19217) 
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[19194]: stage-1 requested pid to be forwarded 
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[19194]: forward stage-1 (19206) and stage-2 (19217) pids to runc 
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-1[19206]: signal completion to stage-0 
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[19194]: stage-1 complete            
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[19194]: <- stage-1 synchronisation loop 
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[19194]: -> stage-2 synchronisation loop 
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[19194]: signalling stage-2 to run   
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-1[19206]: <~ nsexec stage-1           
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-2[1]: ~> nsexec stage-2               
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-2[1]: unshare cgroup namespace        
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-2[1]: signal completion to stage-0    
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[19194]: stage-2 complete            
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[19194]: <- stage-2 synchronisation loop 
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[19194]: <~ nsexec stage-0           
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-2[1]: <= nsexec container setup       
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-2[1]: booting up go runtime ...       
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() child process in init()                      
INFO[0000]github.com/opencontainers/runc/libcontainer/cgroups/systemd/v2.go:440 github.com/opencontainers/runc/libcontainer/cgroups/systemd.(*unifiedManager).Set() freeze container before SetUnitProperties failed: freezer not supported: open /sys/fs/cgroup/system.slice/runc-test.scope/cgroup.freeze: no such file or directory 
ERRO[0000]github.com/opencontainers/runc/utils.go:57 main.fatal() container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: load program: invalid argument 
DEBU[0000]github.com/opencontainers/runc/utils.go:59 main.fatal() container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: load program: invalid argument

I also posted this question on the Arch Linux ARM forums

The text was updated successfully, but these errors were encountered:

AkihiroSuda · 2021-05-24T18:24:18Z

Do you know whether this is specific to ARM?

cyphar · 2021-05-25T04:25:11Z

I wonder if the issue is that cgroupv2 bpf rules aren't supported with the Arch Linux ARM kernel builds? Their kernel is missing cgroup.freeze which means it's either quite old or has a bunch of features disabled.

EDIT: Nope, their kernel config has CONFIG_CGROUP_BPF=y. But I'm then quite confused why cgroup.freeze is missing -- their config also has CONFIG_CGROUP_FREEZER=y...

EDIT: Oh, your kernel version is 4.9.x not the latest one. @paulohefagundes are you using a custom config or custom kernel?

AkihiroSuda · 2021-05-25T04:29:28Z

Kernel needs be 4.15 at least for cgroup v2
https://github.com/opencontainers/runc/blob/master/docs/cgroup-v2.md
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ecf8fecb7828648cba0e42de7464a7e600c93459

cyphar · 2021-05-25T05:12:33Z

Yeah we don't do hybrid mode cgroups -- so we require having all the necessary features for cgroupv2 or just use cgroupv1.

paulo-erichsen · 2021-05-25T14:28:51Z

thank you guys for helping out. Thanks for your info I figured I could workaround and downgrade to cgroup v1 by adding systemd.unified_cgroup_hierarchy=0 to the kernel parameters and that got me to get docker working again

3nprob · 2021-12-08T03:24:50Z

I am having the same issue, except running on a newer kernel (5.16.0) and setting systemd.unified_cgroup_hierarchy=0 does not seem to work.

$ sudo docker run --rm -it hello-world
docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented: unknown.

$ ls /sys/fs/cgroup/cgroup.controllers -la
-r--r--r-- 1 root root 0 Dec  8 03:10 /sys/fs/cgroup/cgroup.controllers

journalctl

Dec 08 03:13:17 localhost dockerd[3434]: time="2021-12-08T03:13:17.010539224Z" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/d3a22a23f42955c04dd1c3ae271a2d5bbf22bf787747976c20f059d620c93bbe pid=5798
Dec 08 03:13:17 localhost dockerd[3434]: time="2021-12-08T03:13:17.226684801Z" level=info msg="shim disconnected" id=d3a22a23f42955c04dd1c3ae271a2d5bbf22bf787747976c20f059d620c93bbe
Dec 08 03:13:17 localhost dockerd[3434]: time="2021-12-08T03:13:17.226990407Z" level=warning msg="cleaning up after shim disconnected" id=d3a22a23f42955c04dd1c3ae271a2d5bbf22bf787747976c20f059d620c93bbe namespace=moby
Dec 08 03:13:17 localhost dockerd[3434]: time="2021-12-08T03:13:17.227068693Z" level=info msg="cleaning up dead shim"
Dec 08 03:13:17 localhost dockerd[3434]: time="2021-12-08T03:13:17.282291867Z" level=warning msg="cleanup warnings time=\"2021-12-08T03:13:17Z\" level=info msg=\"starting signal loop\" namespace=moby pid=5827\n"
Dec 08 03:13:17 localhost dockerd[3434]: time="2021-12-08T03:13:17.283880641Z" level=error msg="copy shim log" error="read /proc/self/fd/13: file already closed"
Dec 08 03:13:17 localhost dockerd[3306]: time="2021-12-08T03:13:17.288216367Z" level=error msg="stream copy error: reading from a closed fifo"
Dec 08 03:13:17 localhost dockerd[3306]: time="2021-12-08T03:13:17.601330810Z" level=error msg="d3a22a23f42955c04dd1c3ae271a2d5bbf22bf787747976c20f059d620c93bbe cleanup: failed to delete container from containerd: no such container"
Dec 08 03:13:18 localhost dockerd[3306]: time="2021-12-08T03:13:18.130567189Z" level=error msg="Handler for POST /v1.41/containers/d3a22a23f42955c04dd1c3ae271a2d5bbf22bf787747976c20f059d620c93bbe/start returned error: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented: unknown"

docker version

Client:
 Version:           20.10.11
 API version:       1.41
 Go version:        go1.17.3
 Git commit:        dea9396e18
 Built:             Sat Nov 20 14:08:33 2021
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.11
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.3
  Git commit:       847da184ad
  Built:            Sat Nov 20 14:08:02 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          v1.5.8
  GitCommit:        1e5ef943eb76627a6d3b6de8cd1ef6537f393a71.m
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:                                                                                                                                                                                                                                                                                                                                                                                 [22/71]
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 4
 Server Version: 20.10.11
 Storage Driver: devicemapper
  Backing Filesystem: ext4
  Udev Sync Supported: true
  Data file: /dev/loop0
  Metadata file: /dev/loop1
  Data loop file: /var/lib/docker/devicemapper/devicemapper/data
  Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
  Deferred Removal Enabled: true
  Deferred Deletion Enabled: true
  Deferred Deleted Device Count: 0
  Library Version: 1.02.181 (2021-10-20)
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1e5ef943eb76627a6d3b6de8cd1ef6537f393a71.m
 runc version: v1.0.3-0-gf46b6ba2
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.16.0-rc2-1
 Operating System: Arch Linux ARM
 OSType: linux
 Architecture: aarch64

runc debug

$  sudo runc --debug --systemd-cgroup run test
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec[10891]: => nsexec container setup
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[10891]: ~> nsexec stage-0
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[10891]: spawn stage-1
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[10891]: -> stage-1 synchronisation loop
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-1[10893]: ~> nsexec stage-1
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-1[10893]: unshare remaining namespace (except cgroupns)
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-1[10893]: spawn stage-2
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-1[10893]: request stage-0 to forward stage-2 pid (10896)
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[10891]: stage-1 requested pid to be forwarded
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-2[1]: ~> nsexec stage-2
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[10891]: forward stage-1 (10893) and stage-2 (10896) pids to runc
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-1[10893]: signal completion to stage-0
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-1[10893]: <~ nsexec stage-1
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[10891]: stage-1 complete
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[10891]: <- stage-1 synchronisation loop
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[10891]: -> stage-2 synchronisation loop
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[10891]: signalling stage-2 to run
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-2[1]: unshare cgroup namespace
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-2[1]: signal completion to stage-0
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-2[1]: <= nsexec container setup
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-2[1]: booting up go runtime ...
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[10891]: stage-2 complete
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[10891]: <- stage-2 synchronisation loop
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() nsexec-0[10891]: <~ nsexec stage-0
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() child process in init()
ERRO[0000]github.com/opencontainers/runc/utils.go:57 main.fatal() container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented
DEBU[0000]github.com/opencontainers/runc/utils.go:59 main.fatal() container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented

AkihiroSuda · 2021-12-08T03:41:29Z

@3nprob

bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented: unknown"

This error is different from the OP. Please make sure your kernel is built with CGROUP_BPF.
https://github.com/torvalds/linux/blob/8ab774587903771821b59471cc723bba6d893942/init/Kconfig#L1140

3nprob · 2021-12-08T03:48:23Z

@AkihiroSuda The odd thing is, it is:

$ zcat /proc/config.gz   | grep -E 'CONFIG_BPF|CONFIG_FREEZER'
CONFIG_BPF=y
# CONFIG_BPF_SYSCALL is not set
CONFIG_BPF_JIT=y
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_FREEZER=y
# CONFIG_BPFILTER is not set

cyphar · 2021-12-08T03:48:56Z

You need CONFIG_BPF_SYSCALL=y.

3nprob · 2021-12-08T03:49:37Z

You need CONFIG_BPF_SYSCALL=y.

Aha! Thanks, will try that

Our BuildKite host's kernel is old, but runc somehow uses cgroup v2. While we should fix runc, we'd like to workaround the issue to unblock us. opencontainers/runc#2959 Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>

Force cgroup v1 usage since some runc apps like docker 20.10+ cannot run on rockchip with cgroup v2 enabled. (It generates BPF related issues, see opencontainers/runc#2959) This is a workaround until a proper fix is submitted. Tested on Asus Tinkerboard with current stable kernel.

AkihiroSuda added area/cgroupv2 area/ebpf platform/arm ARM labels May 24, 2021

AkihiroSuda added the kind/question label May 25, 2021

AkihiroSuda closed this as completed May 25, 2021

AkihiroSuda removed the platform/arm ARM label May 25, 2021

Joulinar mentioned this issue Sep 3, 2021

Odroid XU4 | Force legacy cgroup v1 usage MichaIng/DietPi#4705

Closed

3nprob mentioned this issue Dec 8, 2021

add CONFIG_BPF_SYSCALL=y dreemurrs-embedded/Pine64-Arch#252

Closed

BaochengSu mentioned this issue Jan 6, 2022

modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/5.10.64 siemens/meta-iot2050#238

Closed

paralin mentioned this issue Mar 13, 2022

What about the 5.10.x kernels? OE4T/linux-tegra-4.9#21

Closed

robster7674 mentioned this issue Apr 4, 2022

AppImage for ARM Linux spesmilo/electrum#5159

Open

theblackhole mentioned this issue May 27, 2022

[bootenv/rockchip] Force cgroup v1 usage armbian/build#3830

Merged

8 tasks

cmanfre4 mentioned this issue Sep 14, 2022

Image v0.21.1 crashloop on EKS fargate nodes with kubernetes version 1.23 aws-observability/aws-otel-collector#1487

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[error] OCI runtime create failed: setting cgroup config for procHooks process caused: load program: invalid argument #2959

[error] OCI runtime create failed: setting cgroup config for procHooks process caused: load program: invalid argument #2959

paulo-erichsen commented May 24, 2021

AkihiroSuda commented May 24, 2021

cyphar commented May 25, 2021 •

edited

Loading

AkihiroSuda commented May 25, 2021

cyphar commented May 25, 2021

paulo-erichsen commented May 25, 2021

3nprob commented Dec 8, 2021 •

edited

Loading

AkihiroSuda commented Dec 8, 2021

3nprob commented Dec 8, 2021

cyphar commented Dec 8, 2021

3nprob commented Dec 8, 2021

[error] OCI runtime create failed: setting cgroup config for procHooks process caused: load program: invalid argument #2959

[error] OCI runtime create failed: setting cgroup config for procHooks process caused: load program: invalid argument #2959

Comments

paulo-erichsen commented May 24, 2021

AkihiroSuda commented May 24, 2021

cyphar commented May 25, 2021 • edited Loading

AkihiroSuda commented May 25, 2021

cyphar commented May 25, 2021

paulo-erichsen commented May 25, 2021

3nprob commented Dec 8, 2021 • edited Loading

journalctl

docker version

docker info

runc debug

AkihiroSuda commented Dec 8, 2021

3nprob commented Dec 8, 2021

cyphar commented Dec 8, 2021

3nprob commented Dec 8, 2021

cyphar commented May 25, 2021 •

edited

Loading

3nprob commented Dec 8, 2021 •

edited

Loading