bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented: unknown.

[Marietto2008]

Hello to everyone.

I’ve just installed ubuntu 22.04 on my jetson nano. Docker does not work on 22.04,but it works on ubuntu 18.04 and 20.04. The kernel that I use is always the same

Linux marietto-nano 4.9.299+ #0 SMP PREEMPT Wed Mar 29 14:22:17 CEST 2023 aarch64 aarch64 aarch64 GNU/Linux

,so it’s not its fault if it does not work. I suppose there is some incompatibility between some component present only on ubuntu 22.04 and not on ubuntu 18 and 20. The error is the following :

# docker images

REPOSITORY    TAG       IMAGE ID       CREATED         SIZE
hello-world   latest    46331d942d63   13 months ago   9.14kB

# docker run hello-world

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented: unknown.
ERRO[0004] error waiting for container: context canceled

I don't know what to do. I even tried to upgrade the docker container files following this guide :

https://www.server-world.info/en/note?os=Ubuntu_22.04&p=nvidia&f=2

so,this is what I did :

# curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | apt-key add -
OK

# curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu22.04/nvidia-docker.list > /etc/apt/sources.list.d/nvidia-docker.list

# apt upgrade

Before the upgrade I had these versions :

nvidia-docker2/stable,now 2.8.0-1 all
nvidia-container-toolkit/stable,now 1.7.0-1 arm64

after :

nvidia-docker2/bionic 2.13.0-1 all
nvidia-container-toolkit/bionic 1.13.1-1 arm64
nvidia-container-toolkit-base/bionic 1.13.1-1 arm64

they have been upgraded,but I still see that those packages come from bionic,but I've used the repos of jammy

# curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu22.04/nvidia-docker.list > /etc/apt/sources.list.d/nvidia-docker.list

And anyway,even with the packages upgraded,the error hasn't been fixed. What can I do to make work Docker on ubuntu 22.04 on the jetson nano ?

[elezar]

@Marietto2008 some questions:

• How is docker configured on your system (what are the contents of /etc/docker/daemon.json)? The command you show doesn't seem to invoke the NVIDIA Container Runtime at all.
• Are you using a Jetpack-based distribution?

With regards to the references to 18.04 in the repository list, this is expected. The packages are forward-compatible and we effectively duplicate the .list file server-side to avoid more complex redirects.

[Marietto2008]

/etc/docker/daemon.json

{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

I'm using the JetPack 4.6.3 / L4T 32.7.3.

[elezar]

Thanks for the clarification. That means that the nvidia-container-runtime (or any part of the NVIDIA Container Stack) is not being invoked when you run docker run hello-world and that this error is coming from runc directly.

Note that Ubuntu22.04 switched to using cgroupv2 by default and it could be that this is not compatible with your platform -- or with your runc version. Could you update docker and runc and confirm that the errors still exist.

[Marietto2008]

How can I update docker and runc ?

this is the version I'm using ;

# runc --version

runc version 1.1.4-0ubuntu1~22.04.1
spec: 1.0.2-dev
go: go1.18.1
libseccomp: 2.5.3

# docker --version
Docker version 20.10.21, build 20.10.21-0ubuntu1~22.04.3

they are already upgraded !

[Marietto2008]

I've installed,removed and reinstalled these packages : cgroup-tools_0.41-10_arm64.deb and cgroup-tools_2.0-2_arm64.deb and I've restarted docker. Nothing. the error is still there.

[Marietto2008]

Hello,are you there ? I'm freezed here and I'm waiting for you,for some additional help. thanks.

[elezar]

@Marietto2008 as already discussed, this is not specific to the NVIDIA container stack and is probaly a result of your kernel being too old to properly support cgroupv2.

The behaviour seems similar to docker/cli#3481, although I'm not sure what the process is to update the kernel on the Jetson Nano.

One option would be to disable cgroupv2. Considering https://github.com/opencontainers/runc/blob/main/docs/cgroup-v2.md#cgroup-v2 this should be acheived by adding systemd.unified_cgroup_hierarchy=0 to boot.ini -- although the exact steps may depend on your distribution and system.

the issue is with the

[Marietto2008]

can you give a look at this post ? It may help to understand how to fix the problem :

docker/cli#4238

thanks bro'.

[Marietto2008]

solution :

APPEND ${cbootargs} root=PARTUUID=5ac80d7c-40fb-4796-bd56-4110e389819b rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 systemd.unified_cgroup_hierarchy=0

[elezar]

I am closing this issue. The switch to cgroupv1 seems to have addressed the original problem.