Multi-GPU forward pass fails if first GPU id is not 0

[achaiah]

I can specify multiple GPU IDs and train a network using them just fine. However, when it comes time to run a validation dataset through the trained network, pytorch throws an error when using a list of GPU IDs unless the GPU ID list starts with id 0.

Here's a sample function that will cause an error:

def visualize_model(model=None, num_images=5, use_gpu=False, gpu_ids=[0]):
    print("Using GPU: " + str(gpu_ids[0]))
    for i, data in enumerate(dset_loaders['val']):
        inputs, labels = data
        if use_gpu:
            inputs, labels = Variable(inputs.cuda(device=gpu_ids[0])), Variable(labels.cuda(device=gpu_ids[0]))
        else:
            inputs, labels = Variable(inputs), Variable(labels)

        outputs = model(inputs)      # ERROR HERE!
        _, preds = torch.max(outputs.data, 1)

        plt.figure()
        imshow(inputs.cpu().data[0])
        plt.title('pred: {}'.format(dset_classes[labels.data[0]]))
        plt.show()

        if i == num_images - 1:
            break

If I train the network using gpu_ids = [0,1,2] the function above executes with no problem. However, if I train the network using gpu_ids = [1,2,3] it will throw an error.

[kuangliu]

try CUDA_VISIBLE_DEVICES=1,2,3 python train.py

[achaiah]

When I try that I get an error:
RuntimeError: cuda runtime error (10) : invalid device ordinal at torch/csrc/cuda/Module.cpp:84

[soumith]

you dont have 4 GPUs, looks like it.

CUDA_VISIBLE_DEVICES=1,2,3 python train.py

This means it will try to use GPU2, 3, 4
Change this appropriately depending on the number of GPUs you have.
You can check number of GPUs with the command nvidia-smi

[achaiah]

Hmm I most certainly do have four GPUs, as the network trains on GPUs 1,2,3 just fine (per my original post). The error has to do with something in the torch/pytorch pipeline assuming the wrong GPU when trying to load validation data (unless I misunderstand how to load the data onto GPU1, which the method above is supposed to demonstrate).

RuntimeError: all tensors must be on devices[0]

To wit:

$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.48                 Driver Version: 367.48                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 0000:02:00.0     Off |                  N/A |
| 24%   32C    P8    12W / 180W |    320MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 1080    Off  | 0000:03:00.0     Off |                  N/A |
| 24%   36C    P8    12W / 180W |    119MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 1080    Off  | 0000:81:00.0     Off |                  N/A |
| 24%   36C    P8    13W / 180W |    119MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX 1080    Off  | 0000:82:00.0     Off |                  N/A |
| 24%   35C    P8    12W / 180W |    119MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

[albanD]

If you use @kuangliu suggestion of using CUDA_VISIBLE_DEVICES=1,2,3 you need to set gpu_ids = [0,1,2] because what the env variable is going to do is make sure that if you ask for device 0, you get 1, if you ask for 1, you get 2 etc...

In your case, if you say you training code works but not your validation, you may want to check what are the difference between them if you don't want to go with the CUDA_VISIBLE_DEVICES solution.
In particular, make sure that you set the use_gpu argument and everything else in the same way.

[soumith]

okay so conclusion is:
gpu_ids = [0,1,2].
Then do:

CUDA_VISIBLE_DEVICES="1,2,3" python train.py

[achaiah]

Ah... wow that's so confusing! Yes, I can confirm that this works. Wonder if there's any way to make it more obvious.

[albanD]

If you want to get funky, you can actually do CUDA_VISIBLE_DEVICES=3,1,2 and asking for 0 will get you device 3, asking for 1 will get you device 1 and asking for device 2 will get you device 2.
This may be a bit confusing at first but it is actually quite powerful !

[achaiah]

Yeah I get that now that it's been highlighted, however, I bet anyone trying to learn the nuances of this will be confused for a while as well. For my own enlightenment, what would be the reason where you might want to care whether your tensors live on GPU 3 vs GPU 1 if you're intending to use all of the GPUs anyway?

[albanD]

The CUDA_VISIBLE_DEVICES is provided by nvidia and impact the nvidia driver directly (pytorch is not responsible for that at all). So even though in our case we usually only use it to mask out GPUs, some other application need this finer control.

[achaiah]

So if you're simply indexing within the CUDA_VISIBLE_DEVICES list when specifying device in pytorch why not match the NVIDIA GPU id rather than the index within the CUDA_VISIBLE_DEVICES list?

[albanD]

We don't really have access to that, the nvidia driver is doing this conversion.
Undoing this conversion done in the nvidia driver in pytorch would be super confusing for anyone knowing what CUDA_VISIBLE_DEVICES is doing. And google search would fail you if you get any problem because the behaviour in pytorch would sudently become different from everything else.