System Update in Arch Linux, broked NVIDIA container. Open Source drivers

[waifucon]

docker run -d -p 3000:8080 --gpus all --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:cuda
3483b5dc42f3fe0e9b2c237316872cb649ecea029944bd00c15933e96341c4c1
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: ldcache error: process /sbin/ldconfig terminated with signal 9

Run 'docker run --help' for more information

System Info:
nvidia-smi
Fri Aug 15 15:37:13 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.76.05 Driver Version: 580.76.05 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3070 ... Off | 00000000:01:00.0 On | N/A |
| N/A 56C P5 20W / 125W | 632MiB / 8192MiB | 22% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 873 G /usr/lib/Xorg 21MiB |
| 0 N/A N/A 1241 G /usr/bin/ksecretd 3MiB |
| 0 N/A N/A 1242 G Hyprland 95MiB |
| 0 N/A N/A 1317 G qs 119MiB |
| 0 N/A N/A 1324 G ...it-kde-authentication-agent-1 3MiB |
| 0 N/A N/A 1403 G Xwayland 3MiB |
| 0 N/A N/A 1456 G /usr/lib/xdg-desktop-portal-kde 3MiB |
| 0 N/A N/A 1880 G /usr/lib/firefox/firefox 198MiB |
| 0 N/A N/A 8240 G /usr/bin/kitty 36MiB |
+-----------------------------------------------------------------------------------------+

uname -a:
Linux archlinux 6.16.0-arch2-1 #1 SMP PREEMPT_DYNAMIC Wed, 13 Aug 2025 23:38:48 +0000 x86_64 GNU/Linux

[biuniun]

Having the same problem, nvidia-container-cli -k -d /dev/tty info shows no error message. Rolling back to nvidia-open:575 might be able to resolve the problem for now. Seems like a problem appeared during the update of the driver.

[waifucon]

Having the same problem, nvidia-container-cli -k -d /dev/tty info shows no error message. Rolling back to nvidia-open:575 might be able to resolve the problem for now. Seems like a problem appeared during the update of the driver.

Thanks. for the solution. I deleted docker. I won't look back. But your comment might help others.

[waifucon]

Is affecting other people too. is not only us:
https://bbs.archlinux.org/viewtopic.php?id=307596

[mattrienz]

Having the same issue. Rolled back my mirror list temporarily.

[biuniun]

Interestingly enough, nvidia-container-toolkit works for podman without any problem. The error with ldconfig only appears with Docker.

Trying to debug, but feeling kind of not making sense. Any instance of this issue on other distributions with the 580 driver available?

[davison]

Had the same issue. Since I only run a single container on this machine, I installed podman and ran it with that. Works perfectly using the same docker image. GPU usage is fine. Will probably remove docker completely and stick with this.

[waifucon]

Had the same issue. Since I only run a single container on this machine, I installed podman and ran it with that. Works perfectly using the same docker image. GPU usage is fine. Will probably remove docker completely and stick with this.

Same.

[biuniun]

After further looking into this issue, I realized the output mentioned the legacy mode of using Docker runtime. The bug is associated with ldcache, which is used in runtime instead of CDI mode. The pre-start hook runs an update-ldcache step (calls /sbin/ldconfig inside the container). That’s where the failure occurs. I stopped there for now, but I will dig deeper into it if I have time.

You can set Docker to use CDI for runtime instead of the legacy hook. These require sudo; review before running.
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
sudo nvidia-ctk config --in-place --set nvidia-container-runtime.mode=cdi && systemctl restart docker

With the problem resolved, I am unsure if Nvidia still wants to maintain the legacy mode of using hooks for runc (since it's low-level and partially about the driver). Some installations of the newest version of Docker and the container toolkit have already used CDI runtime as the default. If not, I hope Nvidia can issue the timeline for deprecation.

Edited: fixed some typos.

[waifucon]

After further looking into this issue, I realized the output mentioned the legacy mode of using Docker runtime. The bug is associated with ldcache, which is used in runtime instead of CDI mode. The pre-start hook runs an update-ldcache step (calls /sbin/ldconfig inside the container). That’s where the failure occurs. I stopped there for now, but I will dig deeper into it if I have time.

You can set Docker to use CDI for runtime instead of the legacy hook. These require sudo; review before running. sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml sudo nvidia-ctk config --in-place --set nvidia-container-runtime.mode=cdi && systemctl restart docker

With the problem resolved, I am unsure if Nvidia still wants to maintain the legacy mode of using hooks for runc (since it's low-level and partially about the driver). Some installations of the newest version of Docker and the container toolkit have already used CDI runtime as the default. If not, I hope Nvidia can issue the timeline for depreciation.

Thanks for debugging!

[waifucon]

After further looking into this issue, I realized the output mentioned the legacy mode of using Docker runtime. The bug is associated with ldcache, which is used in runtime instead of CDI mode. The pre-start hook runs an update-ldcache step (calls /sbin/ldconfig inside the container). That’s where the failure occurs. I stopped there for now, but I will dig deeper into it if I have time.
You can set Docker to use CDI for runtime instead of the legacy hook. These require sudo; review before running. sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml sudo nvidia-ctk config --in-place --set nvidia-container-runtime.mode=cdi && systemctl restart docker
With the problem resolved, I am unsure if Nvidia still wants to maintain the legacy mode of using hooks for runc (since it's low-level and partially about the driver). Some installations of the newest version of Docker and the container toolkit have already used CDI runtime as the default. If not, I hope Nvidia can issue the timeline for depreciation.

Thanks for debugging!

But I'll stick with Podman. I'll keep this issue open. To see if NVIDIA could fix it.

[C0rn3j]

@biuniun it does have the negative side effect of causing this

 % docker start whisper-cpp
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Using requested mode 'cdi'
invoking the NVIDIA Container Runtime Hook directly (e.g. specifying the docker --gpus flag) is not supported. Please use the NVIDIA Container Runtime (e.g. specify the --runtime=nvidia flag) instead.
Error: failed to start containers: whisper-cpp

I had to drop my deploy blocks and replace them with a single runtime line:

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

>>>

    runtime: nvidia

My Incus containers with Nvidia forwarded to it are however still broken.

[waifucon]

@biuniun it does have the negative side effect of causing this

 % docker start whisper-cpp
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Using requested mode 'cdi'
invoking the NVIDIA Container Runtime Hook directly (e.g. specifying the docker --gpus flag) is not supported. Please use the NVIDIA Container Runtime (e.g. specify the --runtime=nvidia flag) instead.
Error: failed to start containers: whisper-cpp

I had to drop my deploy blocks and replace them with a single runtime line:

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: all
          capabilities: [gpu]

runtime: nvidia

My Incus containers with Nvidia forwarded to it are however still broken.

Podman is better. NGL

[biuniun]

@biuniun it does have the negative side effect of causing this

 % docker start whisper-cpp
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Using requested mode 'cdi'
invoking the NVIDIA Container Runtime Hook directly (e.g. specifying the docker --gpus flag) is not supported. Please use the NVIDIA Container Runtime (e.g. specify the --runtime=nvidia flag) instead.
Error: failed to start containers: whisper-cpp

I had to drop my deploy blocks and replace them with a single runtime line:

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: all
          capabilities: [gpu]

runtime: nvidia

My Incus containers with Nvidia forwarded to it are however still broken.

Yes, I assume it is a driver-related issue. LXC uses device passthrough and does not support CDI yet, which still requires memory caching. Can you take a look at the log inside the container? Maybe there is some information that can help.

Also, since it does change the runtime/connection setting, the changes for Docker Compose are expected.