Evaluation Method does not work for detection

[aliceinland]

🐛 Bug

After the training process, when running the evaluation function available here (https://github.com/pytorch/vision/blob/dc42f933f3343c76727dbfba6e4242f1bcb8e1a0/references/detection/engine.py), the process gets stuck without any error. I left the evaluation method running for two days but there are no error or any suggestion of what this problem could be when interrupting the process.

Expected behaviour

The result of this function is supposed to be the IoU metrics (the image is taken from the tutorial available on PyTorch)

Environment

• PyTorch version: 1.7.0a0+7036e91
• OS: Ubuntu 18.04
• How you installed PyTorch / torchvision: pip
• Build command you used (if compiling from source): //
• Python version: 3.6
• CUDA/cuDNN version: //
• GPU models and configuration: Tesla T4

[fmassa]

Hi,

Can you share more information on how you launched the evaluation, and where it stopped when you interrupted the process?

[datumbox]

Is there any chance you changed any of the NMS related hyper-params of the model? I had a situation in the past where my training froze due to producing massive amount of predictions. I believe coco eval has a sorting component and very large values can kill it.

[fmassa]

Closing due to inactivity

[Jivitesh2001]

Hi,
I am also facing the same issue; the evaluate function gets stuck using DDP training, and it uses the whole GPU memory and utilization. I am using MaskRCNN model.
https://github.com/pytorch/vision/blob/dc42f933f3343c76727dbfba6e4242f1bcb8e1a0/references/detection/engine.py#L102
In the mentioned line 102 of the above file, the code gets stuck and never comes out of it, and the training process freezes.
@fmassa Can you please help me regarding this issue?