Skip to content
This repository has been archived by the owner on Jan 22, 2024. It is now read-only.

No nvidia gpu, docker: Error response from daemon: failed to create shim task: OCI runtime create failed #1786

Closed
Godeta opened this issue Oct 10, 2023 · 1 comment

Comments

@Godeta
Copy link

Godeta commented Oct 10, 2023

1. Issue or feature description

I'm trying to build a container but I get the error Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy' nvidia-container-cli: initialization error: WSL environment detected but no adapters were found: unknown

More details of the execution :
[2023-10-10T18:21:21.939Z] Start: Run: docker events --format {{json .}} --filter event=start [2023-10-10T18:21:21.943Z] PersistedPath=/tmp/devcontainercli-godeta, ContainerHasLabels=true [2023-10-10T18:21:21.943Z] Container was created with these config files: /home/godeta/rob2023/start-creating-robots-email/.devcontainer/docker-compose.yml,/tmp/devcontainercli-godeta/docker-compose/docker-compose.devcontainer.containerFeatures-1696960110032.yml [2023-10-10T18:21:21.943Z] Expected to find a docker-compose file prefixed with docker-compose.devcontainer.build, but did not. [2023-10-10T18:21:21.944Z] Restoring /tmp/devcontainercli-godeta/docker-compose/docker-compose.devcontainer.containerFeatures-1696960110032.yml from persisted storage [2023-10-10T18:21:21.944Z] Start: Run: docker-compose --project-name start-creating-robots-email_devcontainer -f /home/godeta/rob2023/start-creating-robots-email/.devcontainer/docker-compose.yml -f /tmp/devcontainercli-godeta/docker-compose/docker-compose.devcontainer.containerFeatures-1696960110032.yml up -d --no-recreate [2023-10-10T18:21:22.239Z] [+] Building 0.0s (0/0) docker:default [2023-10-10T18:21:22.273Z] [+] Running 0/0 ⠋ Container start-creating-robots-email_devcontainer-humble-desktop-full-1 Starting0.0s [2023-10-10T18:21:22.373Z] [+] Running 0/1 ⠙ Container start-creating-robots-email_devcontainer-humble-desktop-full-1 Starting0.1s [2023-10-10T18:21:22.473Z] [+] Running 0/1 [2023-10-10T18:21:22.474Z] ⠹ Container start-creating-robots-email_devcontainer-humble-desktop-full-1 Starting0.2s [2023-10-10T18:21:22.573Z] [+] Running 0/1 ⠸ Container start-creating-robots-email_devcontainer-humble-desktop-full-1 Starting0.3s [2023-10-10T18:21:22.651Z] [+] Running 0/1 ⠼ Container start-creating-robots-email_devcontainer-humble-desktop-full-1 Starting0.4s Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy' nvidia-container-cli: initialization error: WSL environment detected but no adapters were found: unknown [2023-10-10T18:21:22.658Z] Stop (714 ms): Run: docker-compose --project-name start-creating-robots-email_devcontainer -f /home/godeta/rob2023/start-creating-robots-email/.devcontainer/docker-compose.yml -f /tmp/devcontainercli-godeta/docker-compose/docker-compose.devcontainer.containerFeatures-1696960110032.yml up -d --no-recreate [2023-10-10T18:21:22.662Z] Error: Command failed: docker-compose --project-name start-creating-robots-email_devcontainer -f /home/godeta/rob2023/start-creating-robots-email/.devcontainer/docker-compose.yml -f /tmp/devcontainercli-godeta/docker-compose/docker-compose.devcontainer.containerFeatures-1696960110032.yml up -d --no-recreate [2023-10-10T18:21:22.664Z] at iAA (/home/godeta/.vscode-remote-containers/dist/dev-containers-cli-0.315.1/dist/spec-node/devContainersSpecCLI.js:427:3165) [2023-10-10T18:21:22.665Z] at process.processTicksAndRejections (node:internal/process/task_queues:95:5) [2023-10-10T18:21:22.666Z] at async tAA (/home/godeta/.vscode-remote-containers/dist/dev-containers-cli-0.315.1/dist/spec-node/devContainersSpecCLI.js:409:3169) [2023-10-10T18:21:22.666Z] at async SAA (/home/godeta/.vscode-remote-containers/dist/dev-containers-cli-0.315.1/dist/spec-node/devContainersSpecCLI.js:479:3833) [2023-10-10T18:21:22.667Z] at async GC (/home/godeta/.vscode-remote-containers/dist/dev-containers-cli-0.315.1/dist/spec-node/devContainersSpecCLI.js:479:4775) [2023-10-10T18:21:22.668Z] at async ZeA (/home/godeta/.vscode-remote-containers/dist/dev-containers-cli-0.315.1/dist/spec-node/devContainersSpecCLI.js:611:12251) [2023-10-10T18:21:22.669Z] at async VeA (/home/godeta/.vscode-remote-containers/dist/dev-containers-cli-0.315.1/dist/spec-node/devContainersSpecCLI.js:611:11992) [2023-10-10T18:21:22.675Z] Stop (1966 ms): Run in Host: /home/godeta/.vscode-remote-containers/bin/abd2f3db4bdb28f9e95536dfa84d8479f1eb312d/node /home/godeta/.vscode-remote-containers/dist/dev-containers-cli-0.315.1/dist/spec-node/devContainersSpecCLI.js up --container-session-data-folder /tmp/devcontainers-9afe62a3-caf2-493c-a13c-0a5ba7df52821696962077996 --workspace-folder /home/godeta/rob2023/start-creating-robots-email --workspace-mount-consistency cached --id-label devcontainer.local_folder=\\wsl.localhost\Ubuntu-22.04\home\godeta\rob2023\start-creating-robots-email --id-label devcontainer.config_file=/home/godeta/rob2023/start-creating-robots-email/.devcontainer/devcontainer.json --log-level debug --log-format json --config /home/godeta/rob2023/start-creating-robots-email/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --mount type=bind,source=/run/user/1000/wayland-0,target=/tmp/vscode-wayland-d435890b-d12d-41af-a3bb-a7fdcf2a68c3.sock --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root [2023-10-10T18:21:22.676Z] Exit code 1

  • Docker works properly, I receive the message "your installation appears to be working correctly" with sudo docker run hello-world

  • Some nvidia-container information: nvidia-container-cli -k -d /dev/tty info

`godeta@LAPTOP-27IDE9KB:/etc/cdi$ nvidia-container-cli -k -d /dev/tty info

I1010 18:04:36.527738 8259 nvc.c:376] initializing library context (version=1.14.2, build=1eb5a30a6ad0415550a9df632ac8832bf7e2bbba)
I1010 18:04:36.527792 8259 nvc.c:350] using root /
I1010 18:04:36.527796 8259 nvc.c:351] using ldcache /etc/ld.so.cache
I1010 18:04:36.527798 8259 nvc.c:352] using unprivileged user 1000:1000
I1010 18:04:36.527812 8259 nvc.c:393] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I1010 18:04:36.529504 8259 dxcore.c:228] Creating a new WDDM Adapter for hAdapter:40000000 luid:220097
I1010 18:04:36.530285 8259 dxcore.c:211] Core Nvidia component libcuda.so.1.1 not found in /usr/lib/wsl/drivers/u0377383.inf_amd64_30501df11e64fe9a
I1010 18:04:36.530551 8259 dxcore.c:211] Core Nvidia component libcuda_loader.so not found in /usr/lib/wsl/drivers/u0377383.inf_amd64_30501df11e64fe9a
I1010 18:04:36.530792 8259 dxcore.c:211] Core Nvidia component libnvidia-ptxjitcompiler.so.1 not found in /usr/lib/wsl/drivers/u0377383.inf_amd64_30501df11e64fe9a
I1010 18:04:36.531028 8259 dxcore.c:211] Core Nvidia component libnvidia-ml.so.1 not found in /usr/lib/wsl/drivers/u0377383.inf_amd64_30501df11e64fe9a
I1010 18:04:36.531244 8259 dxcore.c:211] Core Nvidia component libnvidia-ml_loader.so not found in /usr/lib/wsl/drivers/u0377383.inf_amd64_30501df11e64fe9a
I1010 18:04:36.531494 8259 dxcore.c:211] Core Nvidia component nvidia-smi not found in /usr/lib/wsl/drivers/u0377383.inf_amd64_30501df11e64fe9a
I1010 18:04:36.531705 8259 dxcore.c:211] Core Nvidia component nvcubins.bin not found in /usr/lib/wsl/drivers/u0377383.inf_amd64_30501df11e64fe9a
I1010 18:04:36.531731 8259 dxcore.c:216] No Nvidia component found in /usr/lib/wsl/drivers/u0377383.inf_amd64_30501df11e64fe9a
E1010 18:04:36.531735 8259 dxcore.c:262] Failed to query the core Nvidia libraries for the adapter. Skipping it.
I1010 18:04:36.531738 8259 dxcore.c:326] dxcore layer initialized successfully
E1010 18:04:36.532117 8259 nvc.c:398] dxcore initialization succeeded but no adapters were found
nvidia-container-cli: initialization error: WSL environment detected but no adapters were found
I1010 18:04:36.532177 8259 nvc.c:434] shutting down library context
godeta@LAPTOP-27IDE9KB:/etc/cdi$`

  • This command I found on some suggestion on another post successfully created the file but didn't change anything for me : sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

  • Kernel version from uname -a
    Linux LAPTOP-27IDE9KB 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

  • Any relevant kernel output lines from dmesg

I1010 18:04:36.527738 8259 nvc.c:376] initializing library context (version=1.14.2, build=1eb5a30a6ad0415550a9df632ac8832bf7e2bbba) I1010 18:04:36.527792 8259 nvc.c:350] using root / I1010 18:04:36.527796 8259 nvc.c:351] using ldcache /etc/ld.so.cache I1010 18:04:36.527798 8259 nvc.c:352] using unprivileged user 1000:1000 I1010 18:04:36.527812 8259 nvc.c:393] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL) I1010 18:04:36.529504 8259 dxcore.c:228] Creating a new WDDM Adapter for hAdapter:40000000 luid:220097 I1010 18:04:36.530285 8259 dxcore.c:211] Core Nvidia component libcuda.so.1.1 not found in /usr/lib/wsl/drivers/u0377383.inf_amd64_30501df11e64fe9a I1010 18:04:36.530551 8259 dxcore.c:211] Core Nvidia component libcuda_loader.so not found in /usr/lib/wsl/drivers/u0377383.inf_amd64_30501df11e64fe9a I1010 18:04:36.530792 8259 dxcore.c:211] Core Nvidia component libnvidia-ptxjitcompiler.so.1 not found in /usr/lib/wsl/drivers/u0377383.inf_amd64_30501df11e64fe9a I1010 18:04:36.531028 8259 dxcore.c:211] Core Nvidia component libnvidia-ml.so.1 not found in /usr/lib/wsl/drivers/u0377383.inf_amd64_30501df11e64fe9a I1010 18:04:36.531244 8259 dxcore.c:211] Core Nvidia component libnvidia-ml_loader.so not found in /usr/lib/wsl/drivers/u0377383.inf_amd64_30501df11e64fe9a I1010 18:04:36.531494 8259 dxcore.c:211] Core Nvidia component nvidia-smi not found in /usr/lib/wsl/drivers/u0377383.inf_amd64_30501df11e64fe9a I1010 18:04:36.531705 8259 dxcore.c:211] Core Nvidia component nvcubins.bin not found in /usr/lib/wsl/drivers/u0377383.inf_amd64_30501df11e64fe9a I1010 18:04:36.531731 8259 dxcore.c:216] No Nvidia component found in /usr/lib/wsl/drivers/u0377383.inf_amd64_30501df11e64fe9a E1010 18:04:36.531735 8259 dxcore.c:262] Failed to query the core Nvidia libraries for the adapter. Skipping it. I1010 18:04:36.531738 8259 dxcore.c:326] dxcore layer initialized successfully E1010 18:04:36.532117 8259 nvc.c:398] dxcore initialization succeeded but no adapters were found nvidia-container-cli: initialization error: WSL environment detected but no adapters were found I1010 18:04:36.532177 8259 nvc.c:434] shutting down library context Linux LAPTOP-27IDE9KB 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux godeta@LAPTOP-27IDE9KB:/etc/cdi$ dmesg [ 0.000000] Linux version 5.15.90.1-microsoft-standard-WSL2 (oe-user@oe-host) (x86_64-msft-linux-gcc (GCC) 9.3.0, GNU ld (GNU Binutils) 2.34.0.20200220) #1 SMP Fri Jan 27 02:56:13 UTC 2023 [ 0.000000] Command line: initrd=\initrd.img WSL_ROOT_INIT=1 panic=-1 nr_cpus=8 bonding.max_bonds=0 dummy.numdummies=0 fb_tunnels=none swiotlb=force console=hvc0 debug pty.legacy_count=0

  • Driver information from nvidia-smi -a but nvidia-smi doesn't work :
    godeta@LAPTOP-27IDE9KB:/etc/cdi$ nvidia-smi NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

  • Docker version from docker version

`godeta@LAPTOP-27IDE9KB:/etc/cdi$ docker version
Client: Docker Engine - Community
Cloud integration: v1.0.35+desktop.5
Version: 24.0.6
API version: 1.43
Go version: go1.20.7
Git commit: ed223bc
Built: Mon Sep 4 12:32:16 2023
OS/Arch: linux/amd64
Context: default

Server: Docker Desktop
Engine:
Version: 24.0.6
API version: 1.43 (minimum version 1.12)
Go version: go1.20.7
Git commit: 1a79695
Built: Mon Sep 4 12:32:16 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.22
GitCommit: 8165feabfdfe38c65b599c4993d227328c231fca
runc:
Version: 1.1.8
GitCommit: v1.1.8-0-g82f18fe
docker-init:
Version: 0.19.0
GitCommit: de40ad0`

  • NVIDIA container library version from nvidia-container-cli -V

godeta@LAPTOP-27IDE9KB:/etc/cdi$ nvidia-container-cli -V cli-version: 1.14.2 lib-version: 1.14.2 build date: 2023-09-25T10:10+00:00 build revision: 1eb5a30a6ad0415550a9df632ac8832bf7e2bbba build compiler: x86_64-linux-gnu-gcc-7 7.5.0 build platform: x86_64 build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fplan9-extensions -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections

2. Steps to reproduce the issue

Using a computer without nvidia gpu for example "dxdiag" shows :
image

image

Using docker, wsl2, ubuntu 22 downloaded from microsoft store, vs code and the docker container from : https://github.com/johnny555/start-creating-robots-email

Opening the folder start-creating-robots-email from ubuntu22 in vs code and then "Open Folder In Dev Container".

3. Additionnal

I have seen the issue #1648 but couldn't resolve my problem as my situation is different (I don't have an nvidia gpu at all). I have also tried a lot of different things looking at similar issues on internet but as my understanding is limited I find myself struck.
Thank you for reading me, I apologize for my poor english and lack of knowledge I don't know much about docker and linux.

@Godeta
Copy link
Author

Godeta commented Oct 11, 2023

Thanks to the help of John Vial the author of the github repo that I was trying to run I finally managed to solve my problem.
The solution : modifying the docker-compose.yml file to get rid of the nvidia devices settings


version: '3'
services:
 humble-desktop-full:
   image: johnny555/stc:v1
   user: ros
   volumes:
     - ../:/workspace:cached
     - /tmp/.X11-unix:/tmp/.X11-unix
     - /mnt/wslg:/mnt/wslg
     - /usr/lib/wsl:/usr/lib/wsl

   devices:
     - /dev/dxg:/dev/dxg

   deploy:
     resources:
       reservations:
         devices:
           - driver: nvidia
             count: 1
             capabilities: [ gpu ]

   environment:
     - DISPLAY=:0
     - WAYLAND_DISPLAY
     - XDG_RUNTIME_DIR
     - PULSE_SERVER

   command: /bin/sh -c "while sleep 1000; do :; done" 

deleting lines 12 to 21 (the devices and deploy parts) does the trick for me.

I apologize If I misplaced this github issue It's the first time I write one.

@Godeta Godeta closed this as completed Oct 11, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant