Does not respect NV_GPU

[dtmoodie]

I'm trying to run several instances of an application in different containers on different GPUs.
Currently my script roughly is as follows:
#!/bin/bash
GPU=$1
export NV_GPU=$GPU
nvidia-docker run ...

However no matter what I set the GPU to, the application runs on GPU 0. No error messages occur, and I've tried using the GPU's unique ID as well.
This is with docker version 17.03.1-ce and nvidia-docker 1.0.1-1_amd64.

[flx42]

Can you show me the output of nvidia-smi you're seeing in your test cases?

[dtmoodie]

I've fixed it by changing my script to:

#!/bin/bash
GPU=$1
nvidia-docker run -e CUDA_VISIBLE_DEVICES=$GPU ...

I will get your nvidia-smi with the old script once the current running processes are done.

[flx42]

That doesn't seem right at all. You shouldn't need to use CUDA_VISIBLE_DEVICES.

[sohussain]

I'm having exactly the same issue. With Docker version Docker version 17.05.0-ce, build 89658be and nvidia-docker 1.0.1

Output of nvidia-smi is

Thu Sep 14 14:02:52 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN X (Pascal)    Off  | 0000:05:00.0      On |                  N/A |
| 31%   53C    P8    21W / 250W |    468MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN X (Pascal)    Off  | 0000:06:00.0     Off |                  N/A |
| 33%   56C    P8    21W / 250W |      3MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  TITAN X (Pascal)    Off  | 0000:09:00.0     Off |                  N/A |
| 31%   53C    P8    20W / 250W |      3MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  TITAN X (Pascal)    Off  | 0000:0A:00.0     Off |                  N/A |
| 41%   71C    P2   198W / 250W |  11559MiB / 12189MiB |     62%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1187    G   /usr/lib/xorg/Xorg                             287MiB |
|    0      2658    G   compiz                                         176MiB |
|    3     19119    C   ./build/tools/caffe                          11556MiB |
+-----------------------------------------------------------------------------+

[flx42]

Please share the exact commands you are using.

Note that --privileged will not work in combination with NV_GPU, since it gives you access to all devices.

[sohussain]

Sorry for the delay.

I tried the command below with setting NV_GPU both separately and in-line
sudo nvidia-docker run --net dockernet --ip 172.18.0.5 -it --name occupancy-container --volumes-from data-store occupancy_server
But caffe always loads the network on the GPU 0 as mentioned in nvidia-smi. The values I tried for NV_GPU 0, 1 , 2 and 3 seperately

[hholst80]

This works just fine for me. Dual GPU system:

On host:

# nvidia-smi -q | grep UUID
    GPU UUID                        : GPU-795bed68-e9e5-350f-4402-39df809d08a9
    GPU UUID                        : GPU-203c34c5-5d4c-8c5b-5f70-8dccbe505ffb

# NV_GPU=0 nvidia-docker run -it --rm nvidia/cuda
$ nvidia-smi -q | grep UUID
    GPU UUID                        : GPU-795bed68-e9e5-350f-4402-39df809d08a9

# NV_GPU=1 nvidia-docker run -it --rm nvidia/cuda
$ nvidia-smi -q | grep UUID
    GPU UUID                        : GPU-203c34c5-5d4c-8c5b-5f70-8dccbe505ffb

# docker --version
Docker version 17.06.0-ce, build 02c1d87
# dpkg -l | grep nvidia-docker
ii  nvidia-docker                                   1.0.1-1                                        amd64        NVIDIA Docker container tools

[sohussain]

I used the same commands as @hholst80 but with sudo as my machine does not have the docker group setup yet. Regardless of the value I set NV_GPU to, it always has access to all GPUs. Could sudo be the reason I am unable to get GPU isolation working?

@flx42 Is using sudo equivalent to using docker with the --privileged flag for this particular case? Although I don't see anything of this sort mentioned here https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities.

[3XX0]

Did you make sure you were using sudo -E or sudo NV_GPU=0 ...?

[sohussain]

@3XX0 Oh! thanks, forgive my ignorance about sudo usage.

If OP is fine with it then this issue should be closed.

[dtmoodie]

Yes I believe all corner cases have been flushed out sufficiently to cover my issues. Thanks for the discussion.