cuda runtime error (3): we're not detecting bad forks

[mrshenli]

🐛 Bug

I am not sure if this is reproducible for every environment, but I hit the following error when trying to set cuda device in processes. What is weird is that the error disappears if I remove the line x = torch.rand(20, 2).cuda() right after the for loop.

Traceback (most recent call last):
  File "/home/shenli/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/shenli/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "dist_bug.py", line 5, in run
    torch.cuda.set_device(rank)
  File "/home/shenli/project/pytorch/torch/cuda/__init__.py", line 265, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (3) : initialization error at ../torch/csrc/cuda/Module.cpp:33

Edit: based on the discussion below, the solution should be fixing the "bad fork" error detection, which is duplicated with #17359.

To Reproduce

import torch
from torch.multiprocessing import Process

def run(rank):
    torch.cuda.set_device(rank)

if __name__ == "__main__":
    size = 2
    processes = []
    for rank in range(size):
        # it would work fine without the line below
        x = torch.rand(20, 2).cuda()
        p = Process(target=run, args=(rank,))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

Environment

PyTorch version: 1.1.0a0+63214b5
Is debug build: No
CUDA used to build PyTorch: 9.2.88

OS: CentOS Linux 7 (Core)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
CMake version: version 3.12.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 9.2.88
GPU models and configuration:
GPU 0: Tesla M40
GPU 1: Tesla M40

Nvidia driver version: 396.26
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.15.4
[pip] torch==1.1.0a0+63214b5
[conda] blas 1.0 mkl
[conda] mkl 2019.1 144
[conda] mkl-include 2019.1 144
[conda] mkl_fft 1.0.6 py37hd81dba3_0
[conda] mkl_random 1.0.2 py37hd81dba3_0
[conda] torch 1.1.0a0+63214b5 dev_0

[ssnl]

You should really use spawn context when using cuda with multiprocessing. See https://pytorch.org/docs/master/notes/multiprocessing.html#sharing-cuda-tensors

[colesbury]

CUDA does not support fork. You can't use CUDA in the parent process and then use it in a forked subprocess (without an exec; "spawn" does fork + exec)

We attempt to detect this and issue an error:

pytorch/torch/cuda/__init__.py

Lines 180 to 185 in b312e9d

	def _after_fork(arg):
	global _initialized, _in_bad_fork
	if _initialized and _original_pid != os.getpid():
	_initialized = False
	_in_bad_fork = True
	_CudaBase.__new__ = _lazy_new

pytorch/torch/cuda/__init__.py

Lines 152 to 161 in b312e9d

	if _in_bad_fork:
	from sys import version_info
	if version_info < (3, 4):
	msg = ("To use CUDA with multiprocessing, you must use Python "
	"3.4+ and the 'spawn' start method")
	else:
	msg = ("To use CUDA with multiprocessing, you must use the "
	"'spawn' start method")
	raise RuntimeError(
	"Cannot re-initialize CUDA in forked subprocess. " + msg)

This appears to no longer work. Probably, something is using CUDA before _lazy_init() is called. It's worth fixing this "bad fork" error detection.

[ssnl]

@colesbury @mrshenli If description of this issue is going to be changed to fixing the "bad fork" error detection, we can close #17359 as duplicate.

[tshrjn]

I'm facing a similar issue when using a custom collate_fn with num_workers>0, I get the following error on the line which just makes zero tensor with appropriate device like this src = torch.zeros((bs, max_src_len), dtype=torch.long, device=self.device) :

  File "/path/to/miniconda3/envs/myenv/lib/python3.7/site-packages/tqdm/_tqdm.py", line 979, in __iter__
    for obj in iterable:
  File "/path/to/miniconda3/envs/myenv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 637, in __next__
    return self._process_next_batch(batch)
  File "/path/to/miniconda3/envs/myenv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 658, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
RuntimeError: Traceback (most recent call last):
  File "/path/to/miniconda3/envs/myenv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 138, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/path/to/mycodebase/src/data.py", line 636, in _batch_helper
    (bs, max_src_len), dtype=torch.long, device=self.device)
RuntimeError: CUDA error: initialization error

It's works when num_workers>0 on cpu and also works on gpu with num_workers=0,
but not with num_workers>0 on gpu which makes the throughput quite low.

[gchanan]

removed 1.1 milestone; it didn't make it and doesn't seem blocking for 1.2.

[soumith]

@gchanan this is a blocker for 1.2. We regressed on a fundamental user error message.

[soumith]

(means, lots of confused users, lots of support burden)

[gchanan]

I'm happy to add it to 1.2, just noting:

• We regressed in an earlier version, not 1.1
• It's marked medium priority.