evaluation script runs wrong cuda devise

[IgorDavidyuk]

vision/references/detection/engine.py

Line 86 in 7253e81

torch.cuda.synchronize()

torch.cuda.synchronize(device=None) takes the cuda device name as a parameter, if nothing is given it uses current_ cuda device which is not always the one intended for use.

Moreover, the evaluation script implies that the model is loaded on the same device that passed as an argument.

[fmassa]

Hi,

Good questions!

torch.cuda.synchronize(device=None) takes the cuda device name as a parameter, if nothing is given it uses current_ cuda device which is not always the one intended for use.

If running on a single device, the current recommended way of choosing a different device is to use CUDA_VISIBLE_DEVICES environment variable.

CUDA_VISIBLE_DEVICES=1 python train.py

Plus, the model can potentially be on multiple devices as well, so care should be taken when using it. We generally use a single GPU per process, so this is fine in most cases.

Moreover, the evaluation script implies that the model is loaded on the same device that passed as an argument.

Yes, but as there is no reliable way of getting the device of a model (which can be part on CPU, part on the GPU, and in on multiple GPUs), we assume that the device that is passed is the one that we want to do the evaluation on.

We could add an extra guard model.to(device) and pass torch.cuda.synchronize(device) in the evaluation function, but I'm not 100% convinced this is necessary.

Thoughts?

[IgorDavidyuk]

We could add an extra guard model.to(device) and pass torch.cuda.synchronize(device) in the evaluation function, but I'm not 100% convinced this is necessary.

I guess there is no easy solution

[marcusturewicz]

I ran into the following error when running on a CPU only device:

AssertionError: 
Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from
http://www.nvidia.com/Download/index.aspx

so I changed it to:

if torch.cuda.is_available():
    torch.cuda.synchronize()

[alfonsomhc]

I find this issue when running the example: https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

I have installed cpu-only pytorch:
pip3 install torch==1.8.1+cpu torchvision==0.9.1+cpu torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html

The error message that I get:

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-10-1ea96e94502e> in <module>
      8     lr_scheduler.step()
      9     # evaluate on the test dataset
---> 10     evaluate(model, data_loader_test, device=device)

~/anaconda3/lib/python3.8/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     25         def decorate_context(*args, **kwargs):
     26             with self.__class__():
---> 27                 return func(*args, **kwargs)
     28         return cast(F, decorate_context)
     29 

~/tmp/pytorch_example/engine.py in evaluate(model, data_loader, device)
     84         targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
     85 
---> 86         torch.cuda.synchronize()
     87         model_time = time.time()
     88         outputs = model(image)

~/anaconda3/lib/python3.8/site-packages/torch/cuda/__init__.py in synchronize(device)
    398             if :attr:`device` is ``None`` (default).
    399     """
--> 400     _lazy_init()
    401     with torch.cuda.device(device):
    402         return torch._C._cuda_synchronize()

~/anaconda3/lib/python3.8/site-packages/torch/cuda/__init__.py in _lazy_init()
    162                 "multiprocessing, you must use the 'spawn' start method")
    163         if not hasattr(torch._C, '_cuda_getDeviceCount'):
--> 164             raise AssertionError("Torch not compiled with CUDA enabled")
    165         if _cudart is None:
    166             raise AssertionError(

AssertionError: Torch not compiled with CUDA enabled

As suggested by @marcusturewicz, I have replaced torch.cuda.synchronize() with:

if torch.cuda.is_available():
    torch.cuda.synchronize()

and then the example runs correctly.

[fmassa]

The evaluation not running on CPU has been fixed in #3674
For evaluating in different CUDA devices, the recommended approach is to run with CUDA_VISIBLE_DEVICES as follows

CUDA_VISIBLE_DEVICES=1 python train.py

as described in #1204 (comment)

As such, I'm closing this issue, but let us know if you have further questions