nvidia-smi fails to run in container

[kamikat]

Following instructions from README and installed nvidia-docker2==2.0.2 with package manager. But it failed when tried to run nvidia-smi with docker run --runtime=nvidia --rm nvidia/cuda:8.0 nvidia-smi.

Console output:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:296: starting container process caused "exec: \"nvidia-smi\": executable file not found in $PATH": unknown.

Running sh command produces the same output:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:296: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown.

And here's the system information:

# uname -a
Linux localhost 4.10.0-42-generic #46~16.04.1-Ubuntu SMP Mon Dec 4 15:57:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

# docker version
Client:
 Version:	17.12.0-ce
 API version:	1.35
 Go version:	go1.9.2
 Git commit:	c97c6d6
 Built:	Wed Dec 27 20:11:19 2017
 OS/Arch:	linux/amd64

Server:
 Engine:
  Version:	17.12.0-ce
  API version:	1.35 (minimum version 1.12)
  Go version:	go1.9.2
  Git commit:	c97c6d6
  Built:	Wed Dec 27 20:09:53 2017
  OS/Arch:	linux/amd64
  Experimental:	false

# apt show nvidia-docker2
Package: nvidia-docker2
Version: 2.0.2+docker17.12.0-1
Priority: optional
Section: utils
Maintainer: NVIDIA CORPORATION <cudatools@nvidia.com>
Installed-Size: 18.4 kB
Depends: nvidia-container-runtime (= 1.1.1+docker17.12.0-1), docker-ce (= 17.12.0~ce-0~ubuntu) | docker-ee (= 17.12.0~ee-0~ubuntu)
Breaks: nvidia-docker
Replaces: nvidia-docker
Homepage: https://github.com/NVIDIA/nvidia-docker/wiki
Download-Size: 2,782 B
APT-Manual-Installed: yes
APT-Sources: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  Packages
Description: nvidia-docker CLI wrapper
 Replaces nvidia-docker with a new implementation based on
 nvidia-container-runtime

# nvidia-smi
Wed Jan 10 17:36:09 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 381.22                 Driver Version: 381.22                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 0000:01:00.0      On |                  N/A |
| 27%   41C    P8     7W / 180W |    240MiB /  8110MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1463    G   /usr/lib/xorg/Xorg                              19MiB |
|    0      1992    G   /usr/lib/xorg/Xorg                              78MiB |
|    0      2196    G   /usr/bin/gnome-shell                           111MiB |
+-----------------------------------------------------------------------------+

# nvidia-container-cli -V
version: 1.0.0
build date: 2017-10-30T23:47+00:00
build revision: ec15c7233bd2de821ad5127cb0de6b52d9d2083c
build compiler: gcc-5 5.4.0 20160609
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections

[flx42]

nvidia/cuda:8.0 tag is not published anymore, I replaced it by an empty image since Docker Hub seems to never unpublish tags. Use nvidia/cuda:8.0-devel instead.