Parameter "device" in SupervisedRunner has no effect

[yhn112]

🐛 Bug Report

When running with device=torch.device('cuda:0') or device='cuda:0' model still uses all the GPUs.
When running with device=torch.device('cpu') or device='cpu' an error RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cpu appears.

How To Reproduce

Run any sort of training using SupervisedRunner(device=torch.device('cpu')) or SupervisedRunner(device=torch.device('cuda:0')) on a machine with multiple GPUs.

Environment

PyTorch version: 1.5.0
OS: Ubuntu 18.04
Python version: 3.7
CUDA runtime version: 10.2```

[github-actions]

Hi! Thank you for your contribution! Great first issue!

[AlekseySh]

I faced with the same problem

[AlekseySh]

You can use CUDA_VISIBLE_DEVICES as work around

[yhn112]

I certainly do use CUDA_VISIBLE_DEVICES. It doesn't help me deal with such severe bugs in such a popular framework. =)

[yhn112]

I'm sorry to do that, but
ping @Scitator
how do you do? =)

[Scitator]

@yhn112 please, check the hotfix :)
#815

[Scitator]

now, it's your turn 😂
#809