RuntimeError: CUDA error: initialization error

[Hananel-Hazan]

🐛 Bug

Trying to make the default tensor location to cuda, torch.utils.data.DataLoader produce RuntimeError: CUDA error: initialization error

To Reproduce

1. copy MNIST example link.
2. add the lines:
   device_id = 0
   if use_cuda:
   torch.cuda.set_device(torch.device("cuda:" + str(device_id) if torch.cuda.is_available() else "cpu"))
   torch.set_default_tensor_type('torch.cuda.FloatTensor')

Before (or after) line in the example

Result

cause to a error RuntimeError: CUDA error: initialization error

workaround

In order to solve it kwargs need to be empty
line meet to be kwargs = {}

Expected behavior

default location should not create an error

Environment

Collecting environment information...
PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.6 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
CMake version: version 3.10.0

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 8.0.61
GPU models and configuration:
GPU 0: GeForce GTX TITAN X
GPU 1: GeForce GTX TITAN X

Nvidia driver version: 410.104
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.6.0.21
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.5.1

Versions of relevant libraries:
[pip3] numpy==1.16.3
[pip3] torch==1.1.0
[pip3] torchvision==0.3.0
[conda] Could not collect

Process finished with exit code 0

[ailzhang]

cc: @ssnl is this a valid use case for dataloader?

[ssnl]

use spawn start method. see docs.
https://pytorch.org/docs/master/notes/multiprocessing.html#cuda-in-multiprocessing

[Hananel-Hazan]

@ssnl Thanks for the answer.
Can you elaborate more about your answer?

[ssnl]

@Hananel-Hazan Please see the link I posted above.

[Hananel-Hazan]

@ssnl Thank you very much for the link. The explanation was very illuminating, useful and helpful.

[zuoxingdong]

@ssnl For me, spawn method gives me error AttributeError: Can't pickle local object 'make_vec_env.<locals>.f', I have a method make_vec_env imported in the main file.

Why spawn method tries to pickle this method ? Because it not called within the individual processes.

[mbchang]

After putting in

import torch.multiprocessing as mp
mp.set_start_method('spawn')

I also get the same error AttributeError: Can't pickle local object. Would anyone have any ideas how to fix this?

[ssnl]

Hi, for pickling errors, you likely are using some objects that python pickled can not handle, e.g. objects that has pointers to functions defined in local scopes. In some cases the code can be modified so that the object become picklable, e.g by moving locally-defined functions to global scope. Alternatively you can investigate using other pickers like cloudpickle or dill.

…

On Fri, Nov 15, 2019 at 15:40 mbchang ***@***.***> wrote: After putting in import torch.multiprocessing as mp mp.set_start_method('spawn') I also get the same error AttributeError: Can't pickle local object. Would anyone have any ideas how to fix this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21092?email_source=notifications&email_token=ABLJMZK5FZGF7CVQ54ZBX4TQT4CLFA5CNFSM4HQWB6EKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEGUYYI#issuecomment-554519649>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLJMZJ2QKC3DTO7BKUR2EDQT4CLFANCNFSM4HQWB6EA> .

[renesax14]

use spawn start method. see docs.
https://pytorch.org/docs/master/notes/multiprocessing.html#cuda-in-multiprocessing

Can you explain to us what the error means in the first place?

What does

RuntimeError: CUDA driver error: initialization error” 

mean?

https://discuss.pytorch.org/t/what-does-runtimeerror-cuda-driver-error-initialization-error-mean/87505

[renesax14]

@ssnl Thank you very much for the link. The explanation was very illuminating, useful and helpful.

How was that link useful at all?

Threads are dangerous, ok I already knew that. What exactly did you get out of it?

[renesax14]

🐛 Bug

Trying to make the default tensor location to cuda, torch.utils.data.DataLoader produce RuntimeError: CUDA error: initialization error

To Reproduce

1. copy MNIST example link.
2. add the lines:
   device_id = 0
   if use_cuda:
   torch.cuda.set_device(torch.device("cuda:" + str(device_id) if torch.cuda.is_available() else "cpu"))
   torch.set_default_tensor_type('torch.cuda.FloatTensor')

Before (or after) line in the example

Result

cause to a error RuntimeError: CUDA error: initialization error

workaround

In order to solve it kwargs need to be empty
line meet to be kwargs = {}

Expected behavior

default location should not create an error

Environment

Collecting environment information...
PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.6 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
CMake version: version 3.10.0

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 8.0.61
GPU models and configuration:
GPU 0: GeForce GTX TITAN X
GPU 1: GeForce GTX TITAN X

Nvidia driver version: 410.104
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.6.0.21
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.5.1

Versions of relevant libraries:
[pip3] numpy==1.16.3
[pip3] torch==1.1.0
[pip3] torchvision==0.3.0
[conda] Could not collect

Process finished with exit code 0

What did you end up doing to solve your problem? Why did it happen? Did you use pin_memory=True?