Multi GPU Support

[rusty1s]

Hi,

do CUDA extensions currently support MultiGPUs? Putting all tensors onto device torch.device('cuda:1') in benchmark.py either yields

RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /.../pytorch/aten/src/THC/generic/THCTensorCopy.cpp:70

when trying to print out the resulting tensors (e.g. print(new_h)) or prints out only zeros (e.g. print(new_h.cpu().tolist())).

[goldsborough]

Support for different GPU devices should be totally fine. Are you sure that torch.cuda.device_count() is greater than one in your case? Can you also post all the information about PyTorch from the issue template? There may be an issue with your PyTorch version that causes this.

[rusty1s]

torch.cuda.device_count() prints out 2.

• OS: Ubuntu 16.04
• PyTorch version: 0.4.1
• How you installed PyTorch (conda, pip, source): conda
• Python version: 3.6.5
• CUDA/cuDNN version: 8.0/6.0
• GPU models and configuration: 1080Ti
• GCC version (if compiling from source): 5.4

[rusty1s]

Update: Sorry, you were right! I didn't know I needed to set the device via setCudaDevice for kernels to work properly. It all works fine now :)

[ClementPinard]

For people having trouble with multi gpu support, see https://discuss.pytorch.org/t/c-cuda-extension-with-multiple-gpus/91241