Skip to content
This repository has been archived by the owner on Jan 22, 2024. It is now read-only.

nvidia-smi executable file not found in $PATH #1668

Closed
6 tasks
carlitosgs opened this issue Aug 24, 2022 · 8 comments
Closed
6 tasks

nvidia-smi executable file not found in $PATH #1668

carlitosgs opened this issue Aug 24, 2022 · 8 comments

Comments

@carlitosgs
Copy link

Hi,

1. Issue or feature description

I'm unable to reproduce the Nvidia Container Toolkit installation guide. I have the Nvidia driver installed and nvidia-smi works in my host computer, but it doesn't work inside the nvidia/cuda container.

2. Steps to reproduce the issue

I followed all the installation steps but when I run sudo docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi it throws the following error:
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "nvidia-smi": executable file not found in $PATH: unknown.

3. Information to attach (optional if deemed irrelevant)

  • Some nvidia-container information: nvidia-container-cli -k -d /dev/tty info
 -- WARNING, the following logs are for debugging purposes only --

I0824 11:46:06.136313 12009 nvc.c:376] initializing library context (version=1.10.0, build=395fd41701117121f1fd04ada01e1d7e006a37ae)
I0824 11:46:06.136364 12009 nvc.c:350] using root /
I0824 11:46:06.136372 12009 nvc.c:351] using ldcache /etc/ld.so.cache
I0824 11:46:06.136386 12009 nvc.c:352] using unprivileged user 1000:1000
I0824 11:46:06.136414 12009 nvc.c:393] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I0824 11:46:06.136881 12009 nvc.c:395] dxcore initialization failed, continuing assuming a non-WSL environment
W0824 11:46:06.138360 12010 nvc.c:273] failed to set inheritable capabilities
W0824 11:46:06.138407 12010 nvc.c:274] skipping kernel modules load due to failure
I0824 11:46:06.138655 12011 rpc.c:71] starting driver rpc service
I0824 11:46:06.146875 12014 rpc.c:71] starting nvcgo rpc service
I0824 11:46:06.147719 12009 nvc_info.c:766] requesting driver information with ''
I0824 11:46:06.148864 12009 nvc_info.c:173] selecting /usr/lib64/vdpau/libvdpau_nvidia.so.510.73.08
I0824 11:46:06.149043 12009 nvc_info.c:173] selecting /usr/lib64/libnvoptix.so.510.73.08
I0824 11:46:06.149124 12009 nvc_info.c:173] selecting /usr/lib64/libnvidia-tls.so.510.73.08
I0824 11:46:06.149179 12009 nvc_info.c:173] selecting /usr/lib64/libnvidia-rtcore.so.510.73.08
I0824 11:46:06.149237 12009 nvc_info.c:173] selecting /usr/lib64/libnvidia-ptxjitcompiler.so.510.73.08
I0824 11:46:06.149319 12009 nvc_info.c:173] selecting /usr/lib64/libnvidia-opticalflow.so.510.73.08
I0824 11:46:06.149396 12009 nvc_info.c:173] selecting /usr/lib64/libnvidia-opencl.so.510.73.08
I0824 11:46:06.149456 12009 nvc_info.c:173] selecting /usr/lib64/libnvidia-ngx.so.510.73.08
I0824 11:46:06.149513 12009 nvc_info.c:173] selecting /usr/lib64/libnvidia-ml.so.510.73.08
I0824 11:46:06.149596 12009 nvc_info.c:173] selecting /usr/lib64/libnvidia-glvkspirv.so.510.73.08
I0824 11:46:06.149652 12009 nvc_info.c:173] selecting /usr/lib64/libnvidia-glsi.so.510.73.08
I0824 11:46:06.149712 12009 nvc_info.c:173] selecting /usr/lib64/libnvidia-glcore.so.510.73.08
I0824 11:46:06.149769 12009 nvc_info.c:173] selecting /usr/lib64/libnvidia-fbc.so.510.73.08
I0824 11:46:06.149852 12009 nvc_info.c:173] selecting /usr/lib64/libnvidia-encode.so.510.73.08
I0824 11:46:06.149929 12009 nvc_info.c:173] selecting /usr/lib64/libnvidia-eglcore.so.510.73.08
I0824 11:46:06.149990 12009 nvc_info.c:173] selecting /usr/lib64/libnvidia-compiler.so.510.73.08
I0824 11:46:06.150047 12009 nvc_info.c:173] selecting /usr/lib64/libnvidia-cfg.so.510.73.08
I0824 11:46:06.150127 12009 nvc_info.c:173] selecting /usr/lib64/libnvidia-allocator.so.510.73.08
I0824 11:46:06.150207 12009 nvc_info.c:173] selecting /usr/lib64/libnvcuvid.so.510.73.08
I0824 11:46:06.150569 12009 nvc_info.c:173] selecting /usr/lib64/libcuda.so.510.73.08
I0824 11:46:06.150713 12009 nvc_info.c:173] selecting /usr/lib64/libGLX_nvidia.so.510.73.08
I0824 11:46:06.150777 12009 nvc_info.c:173] selecting /usr/lib64/libGLESv2_nvidia.so.510.73.08
I0824 11:46:06.150833 12009 nvc_info.c:173] selecting /usr/lib64/libGLESv1_CM_nvidia.so.510.73.08
I0824 11:46:06.150895 12009 nvc_info.c:173] selecting /usr/lib64/libEGL_nvidia.so.510.73.08
I0824 11:46:06.150962 12009 nvc_info.c:173] selecting /usr/lib/vdpau/libvdpau_nvidia.so.510.73.08
I0824 11:46:06.151029 12009 nvc_info.c:173] selecting /usr/lib/libnvidia-tls.so.510.73.08
I0824 11:46:06.151087 12009 nvc_info.c:173] selecting /usr/lib/libnvidia-ptxjitcompiler.so.510.73.08
I0824 11:46:06.151166 12009 nvc_info.c:173] selecting /usr/lib/libnvidia-opticalflow.so.510.73.08
I0824 11:46:06.151244 12009 nvc_info.c:173] selecting /usr/lib/libnvidia-opencl.so.510.73.08
I0824 11:46:06.151317 12009 nvc_info.c:173] selecting /usr/lib/libnvidia-ml.so.510.73.08
I0824 11:46:06.151391 12009 nvc_info.c:173] selecting /usr/lib/libnvidia-glvkspirv.so.510.73.08
I0824 11:46:06.151446 12009 nvc_info.c:173] selecting /usr/lib/libnvidia-glsi.so.510.73.08
I0824 11:46:06.151503 12009 nvc_info.c:173] selecting /usr/lib/libnvidia-glcore.so.510.73.08
I0824 11:46:06.151562 12009 nvc_info.c:173] selecting /usr/lib/libnvidia-fbc.so.510.73.08
I0824 11:46:06.151641 12009 nvc_info.c:173] selecting /usr/lib/libnvidia-encode.so.510.73.08
I0824 11:46:06.151722 12009 nvc_info.c:173] selecting /usr/lib/libnvidia-eglcore.so.510.73.08
I0824 11:46:06.151775 12009 nvc_info.c:173] selecting /usr/lib/libnvidia-compiler.so.510.73.08
I0824 11:46:06.151835 12009 nvc_info.c:173] selecting /usr/lib/libnvidia-allocator.so.510.73.08
I0824 11:46:06.151914 12009 nvc_info.c:173] selecting /usr/lib/libnvcuvid.so.510.73.08
I0824 11:46:06.151995 12009 nvc_info.c:173] selecting /usr/lib/libcuda.so.510.73.08
I0824 11:46:06.152075 12009 nvc_info.c:173] selecting /usr/lib/libGLX_nvidia.so.510.73.08
I0824 11:46:06.152134 12009 nvc_info.c:173] selecting /usr/lib/libGLESv2_nvidia.so.510.73.08
I0824 11:46:06.152189 12009 nvc_info.c:173] selecting /usr/lib/libGLESv1_CM_nvidia.so.510.73.08
I0824 11:46:06.152248 12009 nvc_info.c:173] selecting /usr/lib/libEGL_nvidia.so.510.73.08
W0824 11:46:06.152282 12009 nvc_info.c:399] missing library libnvidia-nscq.so
W0824 11:46:06.152297 12009 nvc_info.c:399] missing library libcudadebugger.so
W0824 11:46:06.152311 12009 nvc_info.c:399] missing library libnvidia-fatbinaryloader.so
W0824 11:46:06.152324 12009 nvc_info.c:399] missing library libnvidia-pkcs11.so
W0824 11:46:06.152331 12009 nvc_info.c:399] missing library libnvidia-ifr.so
W0824 11:46:06.152341 12009 nvc_info.c:399] missing library libnvidia-cbl.so
W0824 11:46:06.152352 12009 nvc_info.c:403] missing compat32 library libnvidia-cfg.so
W0824 11:46:06.152362 12009 nvc_info.c:403] missing compat32 library libnvidia-nscq.so
W0824 11:46:06.152376 12009 nvc_info.c:403] missing compat32 library libcudadebugger.so
W0824 11:46:06.152388 12009 nvc_info.c:403] missing compat32 library libnvidia-fatbinaryloader.so
W0824 11:46:06.152395 12009 nvc_info.c:403] missing compat32 library libnvidia-pkcs11.so
W0824 11:46:06.152406 12009 nvc_info.c:403] missing compat32 library libnvidia-ngx.so
W0824 11:46:06.152418 12009 nvc_info.c:403] missing compat32 library libnvidia-ifr.so
W0824 11:46:06.152427 12009 nvc_info.c:403] missing compat32 library libnvidia-rtcore.so
W0824 11:46:06.152435 12009 nvc_info.c:403] missing compat32 library libnvoptix.so
W0824 11:46:06.152449 12009 nvc_info.c:403] missing compat32 library libnvidia-cbl.so
I0824 11:46:06.154184 12009 nvc_info.c:299] selecting /usr/bin/nvidia-smi
I0824 11:46:06.154218 12009 nvc_info.c:299] selecting /usr/bin/nvidia-debugdump
I0824 11:46:06.154252 12009 nvc_info.c:299] selecting /usr/bin/nvidia-persistenced
I0824 11:46:06.154286 12009 nvc_info.c:299] selecting /usr/bin/nv-fabricmanager
I0824 11:46:06.154320 12009 nvc_info.c:299] selecting /usr/bin/nvidia-cuda-mps-control
I0824 11:46:06.154353 12009 nvc_info.c:299] selecting /usr/bin/nvidia-cuda-mps-server
I0824 11:46:06.154404 12009 nvc_info.c:343] listing firmware path /usr/lib/firmware/nvidia/510.73.08/gsp.bin
W0824 11:46:06.154435 12009 nvc_info.c:323] missing device /dev/nvidia-uvm
W0824 11:46:06.154460 12009 nvc_info.c:323] missing device /dev/nvidia-uvm-tools
I0824 11:46:06.154468 12009 nvc_info.c:529] listing device /dev/nvidiactl
I0824 11:46:06.154480 12009 nvc_info.c:529] listing device /dev/nvidia-modeset
W0824 11:46:06.154527 12009 nvc_info.c:349] missing ipc path /var/run/nvidia-persistenced/socket
W0824 11:46:06.154566 12009 nvc_info.c:349] missing ipc path /var/run/nvidia-fabricmanager/socket
W0824 11:46:06.154593 12009 nvc_info.c:349] missing ipc path /tmp/nvidia-mps
I0824 11:46:06.154607 12009 nvc_info.c:822] requesting device information with ''
I0824 11:46:06.160372 12009 nvc_info.c:713] listing device /dev/nvidia0 (GPU-57046403-af31-513e-b7e1-cc3ec3ca8dbc at 00000000:00:1e.0)
NVRM version:   510.73.08
CUDA version:   11.6

Device Index:   0
Device Minor:   0
Model:          Tesla T4
Brand:          QuadroRTX
GPU UUID:       GPU-57046403-af31-513e-b7e1-cc3ec3ca8dbc
Bus Location:   00000000:00:1e.0
Architecture:   7.5
I0824 11:46:06.160425 12009 nvc.c:434] shutting down library context
I0824 11:46:06.160461 12014 rpc.c:95] terminating nvcgo rpc service
I0824 11:46:06.160863 12009 rpc.c:135] nvcgo rpc service terminated successfully
I0824 11:46:06.162923 12011 rpc.c:95] terminating driver rpc service
I0824 11:46:06.163076 12009 rpc.c:135] driver rpc service terminated successfully
  • Kernel version from uname -a
    Linux ip-10-20-11-131 5.10.112-108.499.amzn2.x86_64 #1 SMP Wed Apr 27 23:39:40 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
  • Driver information from nvidia-smi -a
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.08    Driver Version: 510.73.08    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
| N/A   32C    P8    16W /  70W |      0MiB / 15360MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
  • Docker version from docker version
    Docker version 20.10.17, build 100c701
  • NVIDIA container library version from nvidia-container-cli -V
cli-version: 1.11.0~rc.2
lib-version: 1.11.0~rc.2
build date: 2022-07-27T11:40+0000
build revision: ab4ac25ea4752ec8a01afef6c994754cf67a0796
build compiler: gcc 7.3.1 20180712 (Red Hat 7.3.1-15)
build platform: x86_64
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fplan9-extensions -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections
  • Docker command, image and tag used
    docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi

Thanks in advance!
@elezar
Copy link
Member

elezar commented Aug 24, 2022

Hi @carlos-garcia-returnly. Your nvidia-container-cli -V outptu shows version 1.11.0~rc.2 whereas your nvidia-container-cli -k -d /dev/tty info output shows version 1.10.0. Were thes generated by the same CLI?

Would it be possible to do the following:

  1. Enable debug logging for the nvidia-container-cli by uncommenting the #debug = lines in the /etc/nvidia-container-runtime/config.toml file.
  2. Repeat the command that is not working as expected
  3. Attach the /var/log/nvidia-container-toolkit.log file to the issue.

Another thing to try:

  1. Check whether your /etc/docker/daemon.json file contains a reference to the NVIDIA Container Runtime:
{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}
  1. Restart docker: sudo systemctl restart docker
  2. Run the container using:
sudo docker run --rm --gpus all --runtime=nvidia nvidia/cuda:11.0.3-base-ubuntu20.04

Note that this would append to the log(s) (nvidia-container-toolkit.log and nvidia-container-runtime.log) and these could also be attached.

@carlitosgs
Copy link
Author

carlitosgs commented Aug 24, 2022 •
edited
Loading

@elezar as you said those logs were from different cli, sorry. I reinstalled everything and still have the same error. Here are all the information for debugging purposes:

  • Some nvidia-container information: nvidia-container-cli -k -d /dev/tty info
-- WARNING, the following logs are for debugging purposes only --

I0824 16:46:47.451459 25813 nvc.c:376] initializing library context (version=1.11.0~rc.2, build=ab4ac25ea4752ec8a01afef6c994754cf67a0796)
I0824 16:46:47.451503 25813 nvc.c:350] using root /
I0824 16:46:47.451513 25813 nvc.c:351] using ldcache /etc/ld.so.cache
I0824 16:46:47.451520 25813 nvc.c:352] using unprivileged user 1000:1000
I0824 16:46:47.451545 25813 nvc.c:393] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I0824 16:46:47.451990 25813 nvc.c:395] dxcore initialization failed, continuing assuming a non-WSL environment
W0824 16:46:47.453490 25814 nvc.c:273] failed to set inheritable capabilities
W0824 16:46:47.453529 25814 nvc.c:274] skipping kernel modules load due to failure
I0824 16:46:47.453778 25815 rpc.c:71] starting driver rpc service
I0824 16:46:47.984622 25830 rpc.c:71] starting nvcgo rpc service
I0824 16:46:47.985521 25813 nvc_info.c:766] requesting driver information with ''
I0824 16:46:47.986671 25813 nvc_info.c:173] selecting /usr/lib64/vdpau/libvdpau_nvidia.so.515.48.07
I0824 16:46:47.986854 25813 nvc_info.c:173] selecting /usr/lib64/libnvoptix.so.515.48.07
I0824 16:46:47.986936 25813 nvc_info.c:173] selecting /usr/lib64/libnvidia-tls.so.515.48.07
I0824 16:46:47.986995 25813 nvc_info.c:173] selecting /usr/lib64/libnvidia-rtcore.so.515.48.07
I0824 16:46:47.987048 25813 nvc_info.c:173] selecting /usr/lib64/libnvidia-ptxjitcompiler.so.515.48.07
I0824 16:46:47.987119 25813 nvc_info.c:173] selecting /usr/lib64/libnvidia-opticalflow.so.515.48.07
I0824 16:46:47.987191 25813 nvc_info.c:173] selecting /usr/lib64/libnvidia-opencl.so.515.48.07
I0824 16:46:47.987244 25813 nvc_info.c:173] selecting /usr/lib64/libnvidia-ngx.so.515.48.07
I0824 16:46:47.987294 25813 nvc_info.c:173] selecting /usr/lib64/libnvidia-ml.so.515.48.07
I0824 16:46:47.987367 25813 nvc_info.c:173] selecting /usr/lib64/libnvidia-glvkspirv.so.515.48.07
I0824 16:46:47.987426 25813 nvc_info.c:173] selecting /usr/lib64/libnvidia-glsi.so.515.48.07
I0824 16:46:47.987485 25813 nvc_info.c:173] selecting /usr/lib64/libnvidia-glcore.so.515.48.07
I0824 16:46:47.987544 25813 nvc_info.c:173] selecting /usr/lib64/libnvidia-fbc.so.515.48.07
I0824 16:46:47.987624 25813 nvc_info.c:173] selecting /usr/lib64/libnvidia-encode.so.515.48.07
I0824 16:46:47.987703 25813 nvc_info.c:173] selecting /usr/lib64/libnvidia-eglcore.so.515.48.07
I0824 16:46:47.987762 25813 nvc_info.c:173] selecting /usr/lib64/libnvidia-compiler.so.515.48.07
I0824 16:46:47.987821 25813 nvc_info.c:173] selecting /usr/lib64/libnvidia-cfg.so.515.48.07
I0824 16:46:47.987898 25813 nvc_info.c:173] selecting /usr/lib64/libnvidia-allocator.so.515.48.07
I0824 16:46:47.987978 25813 nvc_info.c:173] selecting /usr/lib64/libnvcuvid.so.515.48.07
I0824 16:46:47.988336 25813 nvc_info.c:173] selecting /usr/lib64/libcuda.so.515.48.07
I0824 16:46:47.988480 25813 nvc_info.c:173] selecting /usr/lib64/libGLX_nvidia.so.515.48.07
I0824 16:46:47.988538 25813 nvc_info.c:173] selecting /usr/lib64/libGLESv2_nvidia.so.515.48.07
I0824 16:46:47.988593 25813 nvc_info.c:173] selecting /usr/lib64/libGLESv1_CM_nvidia.so.515.48.07
I0824 16:46:47.988654 25813 nvc_info.c:173] selecting /usr/lib64/libEGL_nvidia.so.515.48.07
I0824 16:46:47.988722 25813 nvc_info.c:173] selecting /usr/lib/vdpau/libvdpau_nvidia.so.515.48.07
I0824 16:46:47.988789 25813 nvc_info.c:173] selecting /usr/lib/libnvidia-tls.so.515.48.07
I0824 16:46:47.988847 25813 nvc_info.c:173] selecting /usr/lib/libnvidia-ptxjitcompiler.so.515.48.07
I0824 16:46:47.988928 25813 nvc_info.c:173] selecting /usr/lib/libnvidia-opticalflow.so.515.48.07
I0824 16:46:47.989008 25813 nvc_info.c:173] selecting /usr/lib/libnvidia-opencl.so.515.48.07
I0824 16:46:47.989067 25813 nvc_info.c:173] selecting /usr/lib/libnvidia-ml.so.515.48.07
I0824 16:46:47.989147 25813 nvc_info.c:173] selecting /usr/lib/libnvidia-glvkspirv.so.515.48.07
I0824 16:46:47.989204 25813 nvc_info.c:173] selecting /usr/lib/libnvidia-glsi.so.515.48.07
I0824 16:46:47.989262 25813 nvc_info.c:173] selecting /usr/lib/libnvidia-glcore.so.515.48.07
I0824 16:46:47.989322 25813 nvc_info.c:173] selecting /usr/lib/libnvidia-fbc.so.515.48.07
I0824 16:46:47.989400 25813 nvc_info.c:173] selecting /usr/lib/libnvidia-encode.so.515.48.07
I0824 16:46:47.989478 25813 nvc_info.c:173] selecting /usr/lib/libnvidia-eglcore.so.515.48.07
I0824 16:46:47.989533 25813 nvc_info.c:173] selecting /usr/lib/libnvidia-compiler.so.515.48.07
I0824 16:46:47.989593 25813 nvc_info.c:173] selecting /usr/lib/libnvidia-allocator.so.515.48.07
I0824 16:46:47.989671 25813 nvc_info.c:173] selecting /usr/lib/libnvcuvid.so.515.48.07
I0824 16:46:47.989750 25813 nvc_info.c:173] selecting /usr/lib/libcuda.so.515.48.07
I0824 16:46:47.989828 25813 nvc_info.c:173] selecting /usr/lib/libGLX_nvidia.so.515.48.07
I0824 16:46:47.989888 25813 nvc_info.c:173] selecting /usr/lib/libGLESv2_nvidia.so.515.48.07
I0824 16:46:47.989947 25813 nvc_info.c:173] selecting /usr/lib/libGLESv1_CM_nvidia.so.515.48.07
I0824 16:46:47.990006 25813 nvc_info.c:173] selecting /usr/lib/libEGL_nvidia.so.515.48.07
W0824 16:46:47.990057 25813 nvc_info.c:399] missing library libnvidia-nscq.so
W0824 16:46:47.990073 25813 nvc_info.c:399] missing library libcudadebugger.so
W0824 16:46:47.990082 25813 nvc_info.c:399] missing library libnvidia-fatbinaryloader.so
W0824 16:46:47.990094 25813 nvc_info.c:399] missing library libnvidia-pkcs11.so
W0824 16:46:47.990110 25813 nvc_info.c:399] missing library libnvidia-ifr.so
W0824 16:46:47.990121 25813 nvc_info.c:399] missing library libnvidia-cbl.so
W0824 16:46:47.990134 25813 nvc_info.c:403] missing compat32 library libnvidia-cfg.so
W0824 16:46:47.990146 25813 nvc_info.c:403] missing compat32 library libnvidia-nscq.so
W0824 16:46:47.990157 25813 nvc_info.c:403] missing compat32 library libcudadebugger.so
W0824 16:46:47.990169 25813 nvc_info.c:403] missing compat32 library libnvidia-fatbinaryloader.so
W0824 16:46:47.990180 25813 nvc_info.c:403] missing compat32 library libnvidia-pkcs11.so
W0824 16:46:47.990192 25813 nvc_info.c:403] missing compat32 library libnvidia-ngx.so
W0824 16:46:47.990203 25813 nvc_info.c:403] missing compat32 library libnvidia-ifr.so
W0824 16:46:47.990215 25813 nvc_info.c:403] missing compat32 library libnvidia-rtcore.so
W0824 16:46:47.990226 25813 nvc_info.c:403] missing compat32 library libnvoptix.so
W0824 16:46:47.990238 25813 nvc_info.c:403] missing compat32 library libnvidia-cbl.so
I0824 16:46:47.991177 25813 nvc_info.c:299] selecting /usr/bin/nvidia-smi
I0824 16:46:47.991215 25813 nvc_info.c:299] selecting /usr/bin/nvidia-debugdump
I0824 16:46:47.991251 25813 nvc_info.c:299] selecting /usr/bin/nvidia-persistenced
I0824 16:46:47.991291 25813 nvc_info.c:299] selecting /usr/bin/nv-fabricmanager
I0824 16:46:47.991328 25813 nvc_info.c:299] selecting /usr/bin/nvidia-cuda-mps-control
I0824 16:46:47.991367 25813 nvc_info.c:299] selecting /usr/bin/nvidia-cuda-mps-server
I0824 16:46:47.991418 25813 nvc_info.c:343] listing firmware path /usr/lib/firmware/nvidia/515.48.07/gsp.bin
W0824 16:46:47.991453 25813 nvc_info.c:323] missing device /dev/nvidia-uvm
W0824 16:46:47.991482 25813 nvc_info.c:323] missing device /dev/nvidia-uvm-tools
I0824 16:46:47.991495 25813 nvc_info.c:529] listing device /dev/nvidiactl
I0824 16:46:47.991512 25813 nvc_info.c:529] listing device /dev/nvidia-modeset
W0824 16:46:47.991552 25813 nvc_info.c:349] missing ipc path /var/run/nvidia-persistenced/socket
W0824 16:46:47.991592 25813 nvc_info.c:349] missing ipc path /var/run/nvidia-fabricmanager/socket
W0824 16:46:47.991624 25813 nvc_info.c:349] missing ipc path /tmp/nvidia-mps
I0824 16:46:47.991638 25813 nvc_info.c:822] requesting device information with ''
I0824 16:46:47.997350 25813 nvc_info.c:713] listing device /dev/nvidia0 (GPU-1feeec3e-32f4-430a-a05a-2f67d37b5697 at 00000000:00:1e.0)
NVRM version:   515.48.07
CUDA version:   11.7

Device Index:   0
Device Minor:   0
Model:          Tesla T4
Brand:          Nvidia
GPU UUID:       GPU-1feeec3e-32f4-430a-a05a-2f67d37b5697
Bus Location:   00000000:00:1e.0
Architecture:   7.5
I0824 16:46:47.997421 25813 nvc.c:434] shutting down library context
I0824 16:46:47.997466 25830 rpc.c:95] terminating nvcgo rpc service
I0824 16:46:47.997900 25813 rpc.c:135] nvcgo rpc service terminated successfully
I0824 16:46:48.067915 25815 rpc.c:95] terminating driver rpc service
I0824 16:46:48.068091 25813 rpc.c:135] driver rpc service terminated successfully
  • Driver information from nvidia-smi -a
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.48.07    Driver Version: 515.48.07    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   42C    P0    27W /  70W |      0MiB / 15360MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
  • Docker version from docker version
 Client:
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.18.3
 Git commit:        100c701
 Built:             Thu Jun 16 20:08:47 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.3
  Git commit:       a89b842
  Built:            Thu Jun 16 20:09:24 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.3
  GitCommit:        1e7bb5b773162b57333d57f612fd72e3f8612d94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
  • NVIDIA container library version from nvidia-container-cli -V
 cli-version: 1.11.0~rc.2
lib-version: 1.11.0~rc.2
build date: 2022-07-27T11:40+0000
build revision: ab4ac25ea4752ec8a01afef6c994754cf67a0796
build compiler: gcc 7.3.1 20180712 (Red Hat 7.3.1-15)
build platform: x86_64
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fplan9-extensions -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections

  • Docker command, image and tag used
    docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi

  • Debugging log cat /var/log/nvidia-container-toolkit.log

-- WARNING, the following logs are for debugging purposes only --

I0824 16:55:18.849978 29690 nvc.c:376] initializing library context (version=1.11.0~rc.2, build=ab4ac25ea4752ec8a01afef6c994754cf67a0796)
I0824 16:55:18.850053 29690 nvc.c:350] using root /
I0824 16:55:18.850062 29690 nvc.c:351] using ldcache /etc/ld.so.cache
I0824 16:55:18.850069 29690 nvc.c:352] using unprivileged user 65534:65534
I0824 16:55:18.850100 29690 nvc.c:393] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I0824 16:55:18.850273 29690 nvc.c:395] dxcore initialization failed, continuing assuming a non-WSL environment
I0824 16:55:18.851734 29697 nvc.c:278] loading kernel module nvidia
I0824 16:55:18.851832 29697 nvc.c:282] running mknod for /dev/nvidiactl
I0824 16:55:18.851865 29697 nvc.c:286] running mknod for /dev/nvidia0
I0824 16:55:18.851888 29697 nvc.c:290] running mknod for all nvcaps in /dev/nvidia-caps
I0824 16:55:18.857046 29697 nvc.c:218] running mknod for /dev/nvidia-caps/nvidia-cap1 from /proc/driver/nvidia/capabilities/mig/config
I0824 16:55:18.857171 29697 nvc.c:218] running mknod for /dev/nvidia-caps/nvidia-cap2 from /proc/driver/nvidia/capabilities/mig/monitor
I0824 16:55:18.858997 29697 nvc.c:296] loading kernel module nvidia_uvm
E0824 16:55:18.860325 29697 nvc.c:298] could not load kernel module nvidia_uvm
I0824 16:55:18.860334 29697 nvc.c:305] loading kernel module nvidia_modeset
I0824 16:55:18.860357 29697 nvc.c:309] running mknod for /dev/nvidia-modeset
I0824 16:55:18.860614 29698 rpc.c:71] starting driver rpc service
I0824 16:55:19.387355 29713 rpc.c:71] starting nvcgo rpc service
I0824 16:55:19.388341 29690 nvc_container.c:240] configuring container with 'compute utility supervised'
I0824 16:55:19.388596 29690 nvc_container.c:88] selecting /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/usr/local/cuda-11.0/compat/libcuda.so.450.191.01
I0824 16:55:19.388667 29690 nvc_container.c:88] selecting /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/usr/local/cuda-11.0/compat/libnvidia-ptxjitcompiler.so.450.191.01
I0824 16:55:19.389914 29690 nvc_container.c:262] setting pid to 29684
I0824 16:55:19.389924 29690 nvc_container.c:263] setting rootfs to /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged
I0824 16:55:19.389934 29690 nvc_container.c:264] setting owner to 0:0
I0824 16:55:19.389941 29690 nvc_container.c:265] setting bins directory to /usr/bin
I0824 16:55:19.389948 29690 nvc_container.c:266] setting libs directory to /usr/lib/x86_64-linux-gnu
I0824 16:55:19.389954 29690 nvc_container.c:267] setting libs32 directory to /usr/lib/i386-linux-gnu
I0824 16:55:19.389960 29690 nvc_container.c:268] setting cudart directory to /usr/local/cuda
I0824 16:55:19.389966 29690 nvc_container.c:269] setting ldconfig to @/sbin/ldconfig (host relative)
I0824 16:55:19.389972 29690 nvc_container.c:270] setting mount namespace to /proc/29684/ns/mnt
I0824 16:55:19.389978 29690 nvc_container.c:272] detected cgroupv1
I0824 16:55:19.389984 29690 nvc_container.c:273] setting devices cgroup to /sys/fs/cgroup/devices/docker/baab649e8a46f56555291be5e0faaddef0ca1c4e0af3476fafa030551e73a790
I0824 16:55:19.390002 29690 nvc_info.c:766] requesting driver information with ''
I0824 16:55:19.391158 29690 nvc_info.c:173] selecting /usr/lib64/vdpau/libvdpau_nvidia.so.515.48.07
I0824 16:55:19.391338 29690 nvc_info.c:173] selecting /usr/lib64/libnvoptix.so.515.48.07
I0824 16:55:19.391414 29690 nvc_info.c:173] selecting /usr/lib64/libnvidia-tls.so.515.48.07
I0824 16:55:19.391472 29690 nvc_info.c:173] selecting /usr/lib64/libnvidia-rtcore.so.515.48.07
I0824 16:55:19.391522 29690 nvc_info.c:173] selecting /usr/lib64/libnvidia-ptxjitcompiler.so.515.48.07
I0824 16:55:19.391591 29690 nvc_info.c:173] selecting /usr/lib64/libnvidia-opticalflow.so.515.48.07
I0824 16:55:19.391661 29690 nvc_info.c:173] selecting /usr/lib64/libnvidia-opencl.so.515.48.07
I0824 16:55:19.391712 29690 nvc_info.c:173] selecting /usr/lib64/libnvidia-ngx.so.515.48.07
I0824 16:55:19.391760 29690 nvc_info.c:173] selecting /usr/lib64/libnvidia-ml.so.515.48.07
I0824 16:55:19.391830 29690 nvc_info.c:173] selecting /usr/lib64/libnvidia-glvkspirv.so.515.48.07
I0824 16:55:19.391876 29690 nvc_info.c:173] selecting /usr/lib64/libnvidia-glsi.so.515.48.07
I0824 16:55:19.391924 29690 nvc_info.c:173] selecting /usr/lib64/libnvidia-glcore.so.515.48.07
I0824 16:55:19.391972 29690 nvc_info.c:173] selecting /usr/lib64/libnvidia-fbc.so.515.48.07
I0824 16:55:19.392041 29690 nvc_info.c:173] selecting /usr/lib64/libnvidia-encode.so.515.48.07
I0824 16:55:19.392108 29690 nvc_info.c:173] selecting /usr/lib64/libnvidia-eglcore.so.515.48.07
I0824 16:55:19.392158 29690 nvc_info.c:173] selecting /usr/lib64/libnvidia-compiler.so.515.48.07
I0824 16:55:19.392206 29690 nvc_info.c:173] selecting /usr/lib64/libnvidia-cfg.so.515.48.07
I0824 16:55:19.392275 29690 nvc_info.c:173] selecting /usr/lib64/libnvidia-allocator.so.515.48.07
I0824 16:55:19.392344 29690 nvc_info.c:173] selecting /usr/lib64/libnvcuvid.so.515.48.07
I0824 16:55:19.392688 29690 nvc_info.c:173] selecting /usr/lib64/libcuda.so.515.48.07
I0824 16:55:19.392830 29690 nvc_info.c:173] selecting /usr/lib64/libGLX_nvidia.so.515.48.07
I0824 16:55:19.392880 29690 nvc_info.c:173] selecting /usr/lib64/libGLESv2_nvidia.so.515.48.07
I0824 16:55:19.392928 29690 nvc_info.c:173] selecting /usr/lib64/libGLESv1_CM_nvidia.so.515.48.07
I0824 16:55:19.392978 29690 nvc_info.c:173] selecting /usr/lib64/libEGL_nvidia.so.515.48.07
I0824 16:55:19.393036 29690 nvc_info.c:173] selecting /usr/lib/vdpau/libvdpau_nvidia.so.515.48.07
I0824 16:55:19.393088 29690 nvc_info.c:173] selecting /usr/lib/libnvidia-tls.so.515.48.07
I0824 16:55:19.393134 29690 nvc_info.c:173] selecting /usr/lib/libnvidia-ptxjitcompiler.so.515.48.07
I0824 16:55:19.393202 29690 nvc_info.c:173] selecting /usr/lib/libnvidia-opticalflow.so.515.48.07
I0824 16:55:19.393272 29690 nvc_info.c:173] selecting /usr/lib/libnvidia-opencl.so.515.48.07
I0824 16:55:19.393320 29690 nvc_info.c:173] selecting /usr/lib/libnvidia-ml.so.515.48.07
I0824 16:55:19.393389 29690 nvc_info.c:173] selecting /usr/lib/libnvidia-glvkspirv.so.515.48.07
I0824 16:55:19.393435 29690 nvc_info.c:173] selecting /usr/lib/libnvidia-glsi.so.515.48.07
I0824 16:55:19.393482 29690 nvc_info.c:173] selecting /usr/lib/libnvidia-glcore.so.515.48.07
I0824 16:55:19.393530 29690 nvc_info.c:173] selecting /usr/lib/libnvidia-fbc.so.515.48.07
I0824 16:55:19.393600 29690 nvc_info.c:173] selecting /usr/lib/libnvidia-encode.so.515.48.07
I0824 16:55:19.393666 29690 nvc_info.c:173] selecting /usr/lib/libnvidia-eglcore.so.515.48.07
I0824 16:55:19.393713 29690 nvc_info.c:173] selecting /usr/lib/libnvidia-compiler.so.515.48.07
I0824 16:55:19.393761 29690 nvc_info.c:173] selecting /usr/lib/libnvidia-allocator.so.515.48.07
I0824 16:55:19.393831 29690 nvc_info.c:173] selecting /usr/lib/libnvcuvid.so.515.48.07
I0824 16:55:19.393901 29690 nvc_info.c:173] selecting /usr/lib/libcuda.so.515.48.07
I0824 16:55:19.393971 29690 nvc_info.c:173] selecting /usr/lib/libGLX_nvidia.so.515.48.07
I0824 16:55:19.394020 29690 nvc_info.c:173] selecting /usr/lib/libGLESv2_nvidia.so.515.48.07
I0824 16:55:19.394085 29690 nvc_info.c:173] selecting /usr/lib/libGLESv1_CM_nvidia.so.515.48.07
I0824 16:55:19.394136 29690 nvc_info.c:173] selecting /usr/lib/libEGL_nvidia.so.515.48.07
W0824 16:55:19.394161 29690 nvc_info.c:399] missing library libnvidia-nscq.so
W0824 16:55:19.394168 29690 nvc_info.c:399] missing library libcudadebugger.so
W0824 16:55:19.394174 29690 nvc_info.c:399] missing library libnvidia-fatbinaryloader.so
W0824 16:55:19.394182 29690 nvc_info.c:399] missing library libnvidia-pkcs11.so
W0824 16:55:19.394188 29690 nvc_info.c:399] missing library libnvidia-ifr.so
W0824 16:55:19.394195 29690 nvc_info.c:399] missing library libnvidia-cbl.so
W0824 16:55:19.394201 29690 nvc_info.c:403] missing compat32 library libnvidia-cfg.so
W0824 16:55:19.394207 29690 nvc_info.c:403] missing compat32 library libnvidia-nscq.so
W0824 16:55:19.394213 29690 nvc_info.c:403] missing compat32 library libcudadebugger.so
W0824 16:55:19.394219 29690 nvc_info.c:403] missing compat32 library libnvidia-fatbinaryloader.so
W0824 16:55:19.394225 29690 nvc_info.c:403] missing compat32 library libnvidia-pkcs11.so
W0824 16:55:19.394231 29690 nvc_info.c:403] missing compat32 library libnvidia-ngx.so
W0824 16:55:19.394237 29690 nvc_info.c:403] missing compat32 library libnvidia-ifr.so
W0824 16:55:19.394243 29690 nvc_info.c:403] missing compat32 library libnvidia-rtcore.so
W0824 16:55:19.394249 29690 nvc_info.c:403] missing compat32 library libnvoptix.so
W0824 16:55:19.394255 29690 nvc_info.c:403] missing compat32 library libnvidia-cbl.so
W0824 16:55:19.394903 29690 nvc_info.c:425] missing binary nvidia-smi
W0824 16:55:19.394910 29690 nvc_info.c:425] missing binary nvidia-debugdump
W0824 16:55:19.394916 29690 nvc_info.c:425] missing binary nvidia-persistenced
W0824 16:55:19.394922 29690 nvc_info.c:425] missing binary nv-fabricmanager
W0824 16:55:19.394928 29690 nvc_info.c:425] missing binary nvidia-cuda-mps-control
W0824 16:55:19.394934 29690 nvc_info.c:425] missing binary nvidia-cuda-mps-server
I0824 16:55:19.394978 29690 nvc_info.c:343] listing firmware path /usr/lib/firmware/nvidia/515.48.07/gsp.bin
W0824 16:55:19.395001 29690 nvc_info.c:323] missing device /dev/nvidia-uvm
W0824 16:55:19.395019 29690 nvc_info.c:323] missing device /dev/nvidia-uvm-tools
I0824 16:55:19.395026 29690 nvc_info.c:529] listing device /dev/nvidiactl
I0824 16:55:19.395032 29690 nvc_info.c:529] listing device /dev/nvidia-modeset
W0824 16:55:19.395064 29690 nvc_info.c:349] missing ipc path /var/run/nvidia-persistenced/socket
W0824 16:55:19.395093 29690 nvc_info.c:349] missing ipc path /var/run/nvidia-fabricmanager/socket
W0824 16:55:19.395114 29690 nvc_info.c:349] missing ipc path /tmp/nvidia-mps
I0824 16:55:19.395121 29690 nvc_info.c:822] requesting device information with ''
I0824 16:55:19.400852 29690 nvc_info.c:713] listing device /dev/nvidia0 (GPU-1feeec3e-32f4-430a-a05a-2f67d37b5697 at 00000000:00:1e.0)
I0824 16:55:19.400947 29690 nvc_mount.c:366] mounting tmpfs at /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/proc/driver/nvidia
I0824 16:55:19.401573 29690 nvc_mount.c:134] mounting /usr/lib64/libnvidia-ml.so.515.48.07 at /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.515.48.07
I0824 16:55:19.401687 29690 nvc_mount.c:134] mounting /usr/lib64/libnvidia-cfg.so.515.48.07 at /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.515.48.07
I0824 16:55:19.401787 29690 nvc_mount.c:134] mounting /usr/lib64/libcuda.so.515.48.07 at /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/usr/lib/x86_64-linux-gnu/libcuda.so.515.48.07
I0824 16:55:19.401886 29690 nvc_mount.c:134] mounting /usr/lib64/libnvidia-opencl.so.515.48.07 at /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.515.48.07
I0824 16:55:19.401987 29690 nvc_mount.c:134] mounting /usr/lib64/libnvidia-ptxjitcompiler.so.515.48.07 at /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.515.48.07
I0824 16:55:19.402112 29690 nvc_mount.c:134] mounting /usr/lib64/libnvidia-allocator.so.515.48.07 at /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.515.48.07
I0824 16:55:19.402216 29690 nvc_mount.c:134] mounting /usr/lib64/libnvidia-compiler.so.515.48.07 at /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.515.48.07
I0824 16:55:19.402251 29690 nvc_mount.c:527] creating symlink /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/usr/lib/x86_64-linux-gnu/libcuda.so -> libcuda.so.1
I0824 16:55:19.402491 29690 nvc_mount.c:134] mounting /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/usr/local/cuda-11.0/compat/libcuda.so.450.191.01 at /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/usr/lib/x86_64-linux-gnu/libcuda.so.450.191.01
I0824 16:55:19.402600 29690 nvc_mount.c:134] mounting /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/usr/local/cuda-11.0/compat/libnvidia-ptxjitcompiler.so.450.191.01 at /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.450.191.01
I0824 16:55:19.402831 29690 nvc_mount.c:85] mounting /usr/lib/firmware/nvidia/515.48.07/gsp.bin at /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/lib/firmware/nvidia/515.48.07/gsp.bin with flags 0x7
I0824 16:55:19.402933 29690 nvc_mount.c:230] mounting /dev/nvidiactl at /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/dev/nvidiactl
I0824 16:55:19.403145 29690 nvc_mount.c:230] mounting /dev/nvidia0 at /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/dev/nvidia0
I0824 16:55:19.403279 29690 nvc_mount.c:440] mounting /proc/driver/nvidia/gpus/0000:00:1e.0 at /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged/proc/driver/nvidia/gpus/0000:00:1e.0
I0824 16:55:19.403404 29690 nvc_ldcache.c:380] executing /sbin/ldconfig from host at /var/lib/docker/overlay2/c8f8860f10a7da7a5db0133979187e94a2cb84b5ba421b25465b2af8ddbedb2a/merged
I0824 16:55:19.416078 29690 nvc.c:434] shutting down library context
I0824 16:55:19.416131 29713 rpc.c:95] terminating nvcgo rpc service
I0824 16:55:19.416552 29690 rpc.c:135] nvcgo rpc service terminated successfully
I0824 16:55:19.487023 29698 rpc.c:95] terminating driver rpc service
I0824 16:55:19.487209 29690 rpc.c:135] driver rpc service terminated successfully
  • Debugging log /var/log/nvidia-container-toolkit.log and /var/log/nvidia-container-runtime.log after running sudo docker run --rm --gpus all --runtime=nvidia nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi
-- WARNING, the following logs are for debugging purposes only --

I0824 16:57:15.173690 30590 nvc.c:376] initializing library context (version=1.11.0~rc.2, build=ab4ac25ea4752ec8a01afef6c994754cf67a0796)
I0824 16:57:15.173747 30590 nvc.c:350] using root /
I0824 16:57:15.173754 30590 nvc.c:351] using ldcache /etc/ld.so.cache
I0824 16:57:15.173761 30590 nvc.c:352] using unprivileged user 65534:65534
I0824 16:57:15.173782 30590 nvc.c:393] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I0824 16:57:15.173987 30590 nvc.c:395] dxcore initialization failed, continuing assuming a non-WSL environment
I0824 16:57:15.175400 30597 nvc.c:278] loading kernel module nvidia
I0824 16:57:15.175500 30597 nvc.c:282] running mknod for /dev/nvidiactl
I0824 16:57:15.175533 30597 nvc.c:286] running mknod for /dev/nvidia0
I0824 16:57:15.175556 30597 nvc.c:290] running mknod for all nvcaps in /dev/nvidia-caps
I0824 16:57:15.180735 30597 nvc.c:218] running mknod for /dev/nvidia-caps/nvidia-cap1 from /proc/driver/nvidia/capabilities/mig/config
I0824 16:57:15.180863 30597 nvc.c:218] running mknod for /dev/nvidia-caps/nvidia-cap2 from /proc/driver/nvidia/capabilities/mig/monitor
I0824 16:57:15.182680 30597 nvc.c:296] loading kernel module nvidia_uvm
E0824 16:57:15.184026 30597 nvc.c:298] could not load kernel module nvidia_uvm
I0824 16:57:15.184035 30597 nvc.c:305] loading kernel module nvidia_modeset
I0824 16:57:15.184059 30597 nvc.c:309] running mknod for /dev/nvidia-modeset
I0824 16:57:15.184316 30598 rpc.c:71] starting driver rpc service
I0824 16:57:15.728741 30607 rpc.c:71] starting nvcgo rpc service
I0824 16:57:15.729690 30590 nvc_container.c:240] configuring container with 'compute utility supervised'
I0824 16:57:15.729931 30590 nvc_container.c:88] selecting /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/usr/local/cuda-11.0/compat/libcuda.so.450.191.01
I0824 16:57:15.730000 30590 nvc_container.c:88] selecting /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/usr/local/cuda-11.0/compat/libnvidia-ptxjitcompiler.so.450.191.01
I0824 16:57:15.731238 30590 nvc_container.c:262] setting pid to 30584
I0824 16:57:15.731249 30590 nvc_container.c:263] setting rootfs to /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged
I0824 16:57:15.731255 30590 nvc_container.c:264] setting owner to 0:0
I0824 16:57:15.731262 30590 nvc_container.c:265] setting bins directory to /usr/bin
I0824 16:57:15.731268 30590 nvc_container.c:266] setting libs directory to /usr/lib/x86_64-linux-gnu
I0824 16:57:15.731275 30590 nvc_container.c:267] setting libs32 directory to /usr/lib/i386-linux-gnu
I0824 16:57:15.731281 30590 nvc_container.c:268] setting cudart directory to /usr/local/cuda
I0824 16:57:15.731287 30590 nvc_container.c:269] setting ldconfig to @/sbin/ldconfig (host relative)
I0824 16:57:15.731294 30590 nvc_container.c:270] setting mount namespace to /proc/30584/ns/mnt
I0824 16:57:15.731300 30590 nvc_container.c:272] detected cgroupv1
I0824 16:57:15.731306 30590 nvc_container.c:273] setting devices cgroup to /sys/fs/cgroup/devices/docker/afd0a652ee94d4c550039315ba9f0cb603e7e425e24632b614b3aab56e195632
I0824 16:57:15.731324 30590 nvc_info.c:766] requesting driver information with ''
I0824 16:57:15.732414 30590 nvc_info.c:173] selecting /usr/lib64/vdpau/libvdpau_nvidia.so.515.48.07
I0824 16:57:15.732588 30590 nvc_info.c:173] selecting /usr/lib64/libnvoptix.so.515.48.07
I0824 16:57:15.732662 30590 nvc_info.c:173] selecting /usr/lib64/libnvidia-tls.so.515.48.07
I0824 16:57:15.732709 30590 nvc_info.c:173] selecting /usr/lib64/libnvidia-rtcore.so.515.48.07
I0824 16:57:15.732760 30590 nvc_info.c:173] selecting /usr/lib64/libnvidia-ptxjitcompiler.so.515.48.07
I0824 16:57:15.732828 30590 nvc_info.c:173] selecting /usr/lib64/libnvidia-opticalflow.so.515.48.07
I0824 16:57:15.732899 30590 nvc_info.c:173] selecting /usr/lib64/libnvidia-opencl.so.515.48.07
I0824 16:57:15.732949 30590 nvc_info.c:173] selecting /usr/lib64/libnvidia-ngx.so.515.48.07
I0824 16:57:15.732998 30590 nvc_info.c:173] selecting /usr/lib64/libnvidia-ml.so.515.48.07
I0824 16:57:15.733068 30590 nvc_info.c:173] selecting /usr/lib64/libnvidia-glvkspirv.so.515.48.07
I0824 16:57:15.733115 30590 nvc_info.c:173] selecting /usr/lib64/libnvidia-glsi.so.515.48.07
I0824 16:57:15.733162 30590 nvc_info.c:173] selecting /usr/lib64/libnvidia-glcore.so.515.48.07
I0824 16:57:15.733210 30590 nvc_info.c:173] selecting /usr/lib64/libnvidia-fbc.so.515.48.07
I0824 16:57:15.733280 30590 nvc_info.c:173] selecting /usr/lib64/libnvidia-encode.so.515.48.07
I0824 16:57:15.733347 30590 nvc_info.c:173] selecting /usr/lib64/libnvidia-eglcore.so.515.48.07
I0824 16:57:15.733398 30590 nvc_info.c:173] selecting /usr/lib64/libnvidia-compiler.so.515.48.07
I0824 16:57:15.733445 30590 nvc_info.c:173] selecting /usr/lib64/libnvidia-cfg.so.515.48.07
I0824 16:57:15.733515 30590 nvc_info.c:173] selecting /usr/lib64/libnvidia-allocator.so.515.48.07
I0824 16:57:15.733585 30590 nvc_info.c:173] selecting /usr/lib64/libnvcuvid.so.515.48.07
I0824 16:57:15.733947 30590 nvc_info.c:173] selecting /usr/lib64/libcuda.so.515.48.07
I0824 16:57:15.734108 30590 nvc_info.c:173] selecting /usr/lib64/libGLX_nvidia.so.515.48.07
I0824 16:57:15.734162 30590 nvc_info.c:173] selecting /usr/lib64/libGLESv2_nvidia.so.515.48.07
I0824 16:57:15.734210 30590 nvc_info.c:173] selecting /usr/lib64/libGLESv1_CM_nvidia.so.515.48.07
I0824 16:57:15.734263 30590 nvc_info.c:173] selecting /usr/lib64/libEGL_nvidia.so.515.48.07
I0824 16:57:15.734320 30590 nvc_info.c:173] selecting /usr/lib/vdpau/libvdpau_nvidia.so.515.48.07
I0824 16:57:15.734373 30590 nvc_info.c:173] selecting /usr/lib/libnvidia-tls.so.515.48.07
I0824 16:57:15.734419 30590 nvc_info.c:173] selecting /usr/lib/libnvidia-ptxjitcompiler.so.515.48.07
I0824 16:57:15.734496 30590 nvc_info.c:173] selecting /usr/lib/libnvidia-opticalflow.so.515.48.07
I0824 16:57:15.734566 30590 nvc_info.c:173] selecting /usr/lib/libnvidia-opencl.so.515.48.07
I0824 16:57:15.734615 30590 nvc_info.c:173] selecting /usr/lib/libnvidia-ml.so.515.48.07
I0824 16:57:15.734683 30590 nvc_info.c:173] selecting /usr/lib/libnvidia-glvkspirv.so.515.48.07
I0824 16:57:15.734729 30590 nvc_info.c:173] selecting /usr/lib/libnvidia-glsi.so.515.48.07
I0824 16:57:15.734777 30590 nvc_info.c:173] selecting /usr/lib/libnvidia-glcore.so.515.48.07
I0824 16:57:15.734824 30590 nvc_info.c:173] selecting /usr/lib/libnvidia-fbc.so.515.48.07
I0824 16:57:15.734893 30590 nvc_info.c:173] selecting /usr/lib/libnvidia-encode.so.515.48.07
I0824 16:57:15.734959 30590 nvc_info.c:173] selecting /usr/lib/libnvidia-eglcore.so.515.48.07
I0824 16:57:15.735005 30590 nvc_info.c:173] selecting /usr/lib/libnvidia-compiler.so.515.48.07
I0824 16:57:15.735053 30590 nvc_info.c:173] selecting /usr/lib/libnvidia-allocator.so.515.48.07
I0824 16:57:15.735122 30590 nvc_info.c:173] selecting /usr/lib/libnvcuvid.so.515.48.07
I0824 16:57:15.735192 30590 nvc_info.c:173] selecting /usr/lib/libcuda.so.515.48.07
I0824 16:57:15.735262 30590 nvc_info.c:173] selecting /usr/lib/libGLX_nvidia.so.515.48.07
I0824 16:57:15.735311 30590 nvc_info.c:173] selecting /usr/lib/libGLESv2_nvidia.so.515.48.07
I0824 16:57:15.735358 30590 nvc_info.c:173] selecting /usr/lib/libGLESv1_CM_nvidia.so.515.48.07
I0824 16:57:15.735405 30590 nvc_info.c:173] selecting /usr/lib/libEGL_nvidia.so.515.48.07
W0824 16:57:15.735430 30590 nvc_info.c:399] missing library libnvidia-nscq.so
W0824 16:57:15.735437 30590 nvc_info.c:399] missing library libcudadebugger.so
W0824 16:57:15.735443 30590 nvc_info.c:399] missing library libnvidia-fatbinaryloader.so
W0824 16:57:15.735449 30590 nvc_info.c:399] missing library libnvidia-pkcs11.so
W0824 16:57:15.735455 30590 nvc_info.c:399] missing library libnvidia-ifr.so
W0824 16:57:15.735461 30590 nvc_info.c:399] missing library libnvidia-cbl.so
W0824 16:57:15.735467 30590 nvc_info.c:403] missing compat32 library libnvidia-cfg.so
W0824 16:57:15.735474 30590 nvc_info.c:403] missing compat32 library libnvidia-nscq.so
W0824 16:57:15.735480 30590 nvc_info.c:403] missing compat32 library libcudadebugger.so
W0824 16:57:15.735486 30590 nvc_info.c:403] missing compat32 library libnvidia-fatbinaryloader.so
W0824 16:57:15.735492 30590 nvc_info.c:403] missing compat32 library libnvidia-pkcs11.so
W0824 16:57:15.735498 30590 nvc_info.c:403] missing compat32 library libnvidia-ngx.so
W0824 16:57:15.735504 30590 nvc_info.c:403] missing compat32 library libnvidia-ifr.so
W0824 16:57:15.735510 30590 nvc_info.c:403] missing compat32 library libnvidia-rtcore.so
W0824 16:57:15.735516 30590 nvc_info.c:403] missing compat32 library libnvoptix.so
W0824 16:57:15.735522 30590 nvc_info.c:403] missing compat32 library libnvidia-cbl.so
W0824 16:57:15.736158 30590 nvc_info.c:425] missing binary nvidia-smi
W0824 16:57:15.736164 30590 nvc_info.c:425] missing binary nvidia-debugdump
W0824 16:57:15.736170 30590 nvc_info.c:425] missing binary nvidia-persistenced
W0824 16:57:15.736177 30590 nvc_info.c:425] missing binary nv-fabricmanager
W0824 16:57:15.736183 30590 nvc_info.c:425] missing binary nvidia-cuda-mps-control
W0824 16:57:15.736189 30590 nvc_info.c:425] missing binary nvidia-cuda-mps-server
I0824 16:57:15.736233 30590 nvc_info.c:343] listing firmware path /usr/lib/firmware/nvidia/515.48.07/gsp.bin
W0824 16:57:15.736255 30590 nvc_info.c:323] missing device /dev/nvidia-uvm
W0824 16:57:15.736273 30590 nvc_info.c:323] missing device /dev/nvidia-uvm-tools
I0824 16:57:15.736280 30590 nvc_info.c:529] listing device /dev/nvidiactl
I0824 16:57:15.736289 30590 nvc_info.c:529] listing device /dev/nvidia-modeset
W0824 16:57:15.736320 30590 nvc_info.c:349] missing ipc path /var/run/nvidia-persistenced/socket
W0824 16:57:15.736349 30590 nvc_info.c:349] missing ipc path /var/run/nvidia-fabricmanager/socket
W0824 16:57:15.736369 30590 nvc_info.c:349] missing ipc path /tmp/nvidia-mps
I0824 16:57:15.736376 30590 nvc_info.c:822] requesting device information with ''
I0824 16:57:15.742082 30590 nvc_info.c:713] listing device /dev/nvidia0 (GPU-1feeec3e-32f4-430a-a05a-2f67d37b5697 at 00000000:00:1e.0)
I0824 16:57:15.742166 30590 nvc_mount.c:366] mounting tmpfs at /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/proc/driver/nvidia
I0824 16:57:15.742727 30590 nvc_mount.c:134] mounting /usr/lib64/libnvidia-ml.so.515.48.07 at /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.515.48.07
I0824 16:57:15.742820 30590 nvc_mount.c:134] mounting /usr/lib64/libnvidia-cfg.so.515.48.07 at /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.515.48.07
I0824 16:57:15.742902 30590 nvc_mount.c:134] mounting /usr/lib64/libcuda.so.515.48.07 at /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/usr/lib/x86_64-linux-gnu/libcuda.so.515.48.07
I0824 16:57:15.742983 30590 nvc_mount.c:134] mounting /usr/lib64/libnvidia-opencl.so.515.48.07 at /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.515.48.07
I0824 16:57:15.743066 30590 nvc_mount.c:134] mounting /usr/lib64/libnvidia-ptxjitcompiler.so.515.48.07 at /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.515.48.07
I0824 16:57:15.743148 30590 nvc_mount.c:134] mounting /usr/lib64/libnvidia-allocator.so.515.48.07 at /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.515.48.07
I0824 16:57:15.743229 30590 nvc_mount.c:134] mounting /usr/lib64/libnvidia-compiler.so.515.48.07 at /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.515.48.07
I0824 16:57:15.743257 30590 nvc_mount.c:527] creating symlink /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/usr/lib/x86_64-linux-gnu/libcuda.so -> libcuda.so.1
I0824 16:57:15.743447 30590 nvc_mount.c:134] mounting /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/usr/local/cuda-11.0/compat/libcuda.so.450.191.01 at /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/usr/lib/x86_64-linux-gnu/libcuda.so.450.191.01
I0824 16:57:15.743539 30590 nvc_mount.c:134] mounting /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/usr/local/cuda-11.0/compat/libnvidia-ptxjitcompiler.so.450.191.01 at /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.450.191.01
I0824 16:57:15.743772 30590 nvc_mount.c:85] mounting /usr/lib/firmware/nvidia/515.48.07/gsp.bin at /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/lib/firmware/nvidia/515.48.07/gsp.bin with flags 0x7
I0824 16:57:15.743859 30590 nvc_mount.c:230] mounting /dev/nvidiactl at /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/dev/nvidiactl
I0824 16:57:15.744062 30590 nvc_mount.c:230] mounting /dev/nvidia0 at /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/dev/nvidia0
I0824 16:57:15.744180 30590 nvc_mount.c:440] mounting /proc/driver/nvidia/gpus/0000:00:1e.0 at /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged/proc/driver/nvidia/gpus/0000:00:1e.0
I0824 16:57:15.744303 30590 nvc_ldcache.c:380] executing /sbin/ldconfig from host at /var/lib/docker/overlay2/0f54b2575d587db0aa3c1c3243d5a42bde0b68db076a60f6450bfe847518162b/merged
I0824 16:57:15.757312 30590 nvc.c:434] shutting down library context
I0824 16:57:15.757364 30607 rpc.c:95] terminating nvcgo rpc service
I0824 16:57:15.757760 30590 rpc.c:135] nvcgo rpc service terminated successfully
I0824 16:57:15.828666 30598 rpc.c:95] terminating driver rpc service
I0824 16:57:15.828840 30590 rpc.c:135] driver rpc service terminated successfully

AND

{"level":"debug","msg":"Command line arguments: [nvidia-container-runtime --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v2.task/moby/4a13088f28ffc64d8e83f5dd1a0df77841462236943acce8cce6ea2ac6ccc219/log.json --log-format json create --bundle /run/containerd/io.containerd.runtime.v2.task/moby/4a13088f28ffc64d8e83f5dd1a0df77841462236943acce8cce6ea2ac6ccc219 --pid-file /run/containerd/io.containerd.runtime.v2.task/moby/4a13088f28ffc64d8e83f5dd1a0df77841462236943acce8cce6ea2ac6ccc219/init.pid 4a13088f28ffc64d8e83f5dd1a0df77841462236943acce8cce6ea2ac6ccc219]","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Looking for runtime binary 'docker-runc'","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Checking candidate '/opt/containerd/bin/docker-runc'","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Candidate '/opt/containerd/bin/docker-runc' does not meet requirements: error getting info for /opt/containerd/bin/docker-runc: stat /opt/containerd/bin/docker-runc: no such file or directory","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Checking candidate '/usr/local/sbin/docker-runc'","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Candidate '/usr/local/sbin/docker-runc' does not meet requirements: error getting info for /usr/local/sbin/docker-runc: stat /usr/local/sbin/docker-runc: no such file or directory","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Checking candidate '/usr/local/bin/docker-runc'","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Candidate '/usr/local/bin/docker-runc' does not meet requirements: error getting info for /usr/local/bin/docker-runc: stat /usr/local/bin/docker-runc: no such file or directory","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Checking candidate '/usr/sbin/docker-runc'","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Candidate '/usr/sbin/docker-runc' does not meet requirements: error getting info for /usr/sbin/docker-runc: stat /usr/sbin/docker-runc: no such file or directory","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Checking candidate '/usr/bin/docker-runc'","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Checking candidate '/sbin/docker-runc'","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Candidate '/sbin/docker-runc' does not meet requirements: error getting info for /sbin/docker-runc: stat /sbin/docker-runc: no such file or directory","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Checking candidate '/bin/docker-runc'","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Found runtime binary '[/usr/bin/docker-runc /bin/docker-runc]'","time":"2022-08-24T17:02:07Z"}
{"level":"info","msg":"Using low-level runtime /usr/bin/docker-runc","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Using bundle directory: /run/containerd/io.containerd.runtime.v2.task/moby/4a13088f28ffc64d8e83f5dd1a0df77841462236943acce8cce6ea2ac6ccc219","time":"2022-08-24T17:02:07Z"}
{"level":"info","msg":"Using OCI specification file path: /run/containerd/io.containerd.runtime.v2.task/moby/4a13088f28ffc64d8e83f5dd1a0df77841462236943acce8cce6ea2ac6ccc219/config.json","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Is Tegra-based system? false: /sys/devices/soc0/family file not found","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Has NVML? true: found NVML library","time":"2022-08-24T17:02:07Z"}
{"level":"info","msg":"Auto-detected mode as 'legacy'","time":"2022-08-24T17:02:07Z"}
{"level":"info","msg":"Using prestart hook path: /usr/bin/nvidia-container-runtime-hook","time":"2022-08-24T17:02:07Z"}
{"level":"info","msg":"existing nvidia prestart hook found in OCI spec","time":"2022-08-24T17:02:07Z"}
{"level":"info","msg":"Applied required modification to OCI specification","time":"2022-08-24T17:02:07Z"}
{"level":"info","msg":"Forwarding command to runtime","time":"2022-08-24T17:02:07Z"}
{"level":"debug","msg":"Command line arguments: [nvidia-container-runtime --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v2.task/moby/4a13088f28ffc64d8e83f5dd1a0df77841462236943acce8cce6ea2ac6ccc219/log.json --log-format json delete --force 4a13088f28ffc64d8e83f5dd1a0df77841462236943acce8cce6ea2ac6ccc219]","time":"2022-08-24T17:02:08Z"}
{"level":"debug","msg":"Looking for runtime binary 'docker-runc'","time":"2022-08-24T17:02:08Z"}
{"level":"debug","msg":"Checking candidate '/opt/containerd/bin/docker-runc'","time":"2022-08-24T17:02:08Z"}
{"level":"debug","msg":"Candidate '/opt/containerd/bin/docker-runc' does not meet requirements: error getting info for /opt/containerd/bin/docker-runc: stat /opt/containerd/bin/docker-runc: no such file or directory","time":"2022-08-24T17:02:08Z"}
{"level":"debug","msg":"Checking candidate '/usr/local/sbin/docker-runc'","time":"2022-08-24T17:02:08Z"}
{"level":"debug","msg":"Candidate '/usr/local/sbin/docker-runc' does not meet requirements: error getting info for /usr/local/sbin/docker-runc: stat /usr/local/sbin/docker-runc: no such file or directory","time":"2022-08-24T17:02:08Z"}
{"level":"debug","msg":"Checking candidate '/usr/local/bin/docker-runc'","time":"2022-08-24T17:02:08Z"}
{"level":"debug","msg":"Candidate '/usr/local/bin/docker-runc' does not meet requirements: error getting info for /usr/local/bin/docker-runc: stat /usr/local/bin/docker-runc: no such file or directory","time":"2022-08-24T17:02:08Z"}
{"level":"debug","msg":"Checking candidate '/usr/sbin/docker-runc'","time":"2022-08-24T17:02:08Z"}
{"level":"debug","msg":"Candidate '/usr/sbin/docker-runc' does not meet requirements: error getting info for /usr/sbin/docker-runc: stat /usr/sbin/docker-runc: no such file or directory","time":"2022-08-24T17:02:08Z"}
{"level":"debug","msg":"Checking candidate '/usr/bin/docker-runc'","time":"2022-08-24T17:02:08Z"}
{"level":"debug","msg":"Checking candidate '/sbin/docker-runc'","time":"2022-08-24T17:02:08Z"}
{"level":"debug","msg":"Candidate '/sbin/docker-runc' does not meet requirements: error getting info for /sbin/docker-runc: stat /sbin/docker-runc: no such file or directory","time":"2022-08-24T17:02:08Z"}
{"level":"debug","msg":"Checking candidate '/bin/docker-runc'","time":"2022-08-24T17:02:08Z"}
{"level":"debug","msg":"Found runtime binary '[/usr/bin/docker-runc /bin/docker-runc]'","time":"2022-08-24T17:02:08Z"}
{"level":"info","msg":"Using low-level runtime /usr/bin/docker-runc","time":"2022-08-24T17:02:08Z"}
{"level":"debug","msg":"Skipping modifier for non-create subcommand","time":"2022-08-24T17:02:08Z"}

Still producing the error: docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "nvidia-smi": executable file not found in $PATH: unknown.

@elezar
Copy link
Member

elezar commented Aug 25, 2022

Thanks for the additional logs. The 1.11.0-rc.2 logs clearly show:

W0824 16:55:19.394903 29690 nvc_info.c:425] missing binary nvidia-smi

whereas the earlier info output shows:

I0824 11:46:06.154184 12009 nvc_info.c:299] selecting /usr/bin/nvidia-smi

Where is nvidia-smi installed on the host?

Since you are running the docker command as sudo it could be that the PATH environment variable is not set correctly in this case. What is the output of:

sudo env

@carlitosgs
Copy link
Author

carlitosgs commented Aug 25, 2022 •
edited
Loading

nvidia-smi is installed in /usr/bin/nvidia-smi

The output of sudo env is:
PATH=/sbin:/bin:/usr/sbin:/usr/bin

Edit: Running without sudo yields same error.

@TonyTromp
Copy link

TonyTromp commented May 10, 2023 •
edited
Loading

seems nvidia-smi is none in the PATHs directive: not in /usr/bin/nvidia-smi either.

PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

root@0d6fccc75e10:/# which nvidia-smi
# nothing

echo $PATH
/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

however

root@0d6fccc75e10:/# ls /usr/local/nvidia/bin/
ls: cannot access '/usr/local/nvidia/bin/': No such file or directory

This is on: nvidia/cuda:11.0.3-base-ubuntu20.04

@elezar
Copy link
Member

elezar commented May 10, 2023

@TonyTromp I assume this is in a docker container? How was docker installed?

I see that the --gpus flag was used. Could you also configure docker to use the nvidia-container-runtime as per our documentation and also specify the --runtime nvidia flag?

@TonyTromp
Copy link

The error put me off-guard, as this is typical when the Docker OCI can not find the binairy in the PATH directive. However the root cause was all on me, i didnt release a VFIO driver and thus there was no GPU to be passed through.

@elezar
Copy link
Member

elezar commented Nov 19, 2023

Closing this as done following the last comment. If there are still problems, please consider opening a new issue against https://github.com/NVIDIA/nvidia-container-toolkit.

@elezar elezar closed this as completed Nov 19, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@elezar @TonyTromp @carlitosgs