Looking for help figuring out why i can't start rootless containers that use my nvidia gpu

[Matty666]

I am using nerdctl rootless and when i try the example command to run nvidia-smi in a container it fails with:
❯ nerdctl run -it --rm --gpus all nvidia/cuda:12.3.1-base-ubuntu20.04 nvidia-smi FATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running createRuntime hook #0: exit status 1, stdout: , stderr: nvidia-container-cli: mount error: failed to add device rules: unable to find any existing device filters attached to the cgroup: bpf_prog_query(BPF_CGROUP_DEVICE) failed: operation not permitted

i tried editing the /etc/nvidia-container-runtime/config.toml file and set no-cgroups = true but it makes no difference. I enabled the debug logs in that file too, but the files are not created.

Not sure where to look next, any help will be much appreciated.

[AkihiroSuda]

cc @ktock

[ningmingxiao]

how about nerdctl run --privileged @Matty666

[Matty666]

@ningmingxiao no luck

nerdctl run -it --privileged --rm --gpus all nvidia/cuda:12.3.1-base-ubuntu20.04 nvidia-smi
FATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running createRuntime hook #0: exit status 1, stdout: , stderr: nvidia-container-cli: mount error: failed to add device rules: unable to find any existing device filters attached to the cgroup: bpf_prog_query(BPF_CGROUP_DEVICE) failed: operation not permitted