torch.cuda.current_device() is changed by CuPy after 10.0
#6569
Comments
fangwei123456
changed the title
torch.cuda.current_device() is changed by CuPy in some machinestorch.cuda.current_device() is changed by CuPy
|
I further test multiple version of CuPy and confirm that version |
fangwei123456
changed the title
torch.cuda.current_device() is changed by CuPytorch.cuda.current_device() is changed by CuPy after 10.0
|
Remove this line and it should work:
The "current device" is semantics provided by CUDA and not by each library. |
|
Thanks, it works well. But if I also remove this line:
def relu(x: torch.Tensor):
device_id = x.get_device()
# torch.cuda.set_device(device_id)
print('1:', torch.cuda.current_device())
y = torch.zeros_like(x)
assert device_id >= 0
# with cupy.cuda.Device(device_id):
kernel = cupy.RawKernel(kernel_code, 'relu')
threads = 1024
N = x.numel()
blocks = (N + threads - 1) // threads
x = x.contiguous()
y = y.contiguous()
N = cupy.asarray(N)
print(f'N.device={N.device}')
kernel((blocks,), (threads,), (x.data_ptr(), y.data_ptr(), N))
print('2:', torch.cuda.current_device())
return y
device = 'cuda:1'
x = torch.rand([8], device=device) - 0.5
y = relu(x)
print(f'x={x}')
print(f'y={y}')Then I will get wrong outputs: How can I set the device of a CuPy array (e.g., |
|
Or, the best way is using |
Yes, the point is to use the same library to switch the current device across the codebase. |
|
OK, thanks! |
Description
Recently, @Yanqi-Chen reports a bug with using a Pytorch module accelerated by CuPy when training with Distributed Data Parallel (DDP).
In DDP training, each process uses
torch.cuda.current_device()as its default device. But he finds that CuPy will changetorch.cuda.current_device(). For example, when training with 4 GPUs,torch.cuda.current_device()should be0, 1, 2, 3in each process. But after using CuPy, all processes'storch.cuda.current_device()will output0.To Reproduce
I run the following codes to reproduce this problem:
In machine A, I get outputs:
You can find that the default device is changed from 1 to 0 after
with cupy.cuda.Device(device_id).However, in machine B, I get outputs:
In machine B, the default device is not changed.
Installation
Source (
pip install cupy)Environment
In machine A:
In machine B:
Additional Information
No response
The text was updated successfully, but these errors were encountered: