Can't YOLOV5 be detected with multithreading?

[bulingda]

Environment:
Python 3.7
Pytorch 1.5.1
Yolov5 v1.0

Bug:
RuntimeError: The expanded size of the tensor (50) must match the existing size (25) at non-singleton dimension 3. Target sizes: [1, 3, 28, 50, 2]. Tensor sizes: [3, 14, 25, 2]
The detailed error information is shown in the figure below:

Description:
I use ThreadPoolExecutor in Python concurrent-future to implement multithreading detection, and the above error occurs whenever my number of tasks (detection) is greater than and 1.
The error occurred here
I was doing two detection tasks at the same time when the error occurred, and I wondered if the shapes didn't match because of information confusion between threads.

Thank you very much for helping me solve this problem.

[github-actions]

👋 Hello @bulingda, thank you for your interest in 🚀 YOLOv5! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://www.ultralytics.com or email Glenn Jocher at glenn.jocher@ultralytics.com.

Requirements

Python 3.8 or later with all requirements.txt dependencies installed, including torch>=1.7. To install run:

$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab Notebook with free GPU:
• Kaggle Notebook with free GPU: https://www.kaggle.com/ultralytics/yolov5
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Docker Image https://hub.docker.com/r/ultralytics/yolov5. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), testing (test.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[bulingda]

I don't know whether my question belongs to this tag or not. If not, I can modify it.Thanks

[glenn-jocher]

@bulingda we don't generally assist in debugging code modifications.

You can see an example of a multithreaded dataloader here, though this is only used for multiple webcam streams at the moment:

yolov5/utils/datasets.py

Line 258 in b1cf25d

class LoadStreams: # multiple IP or RTSP cameras

If you want finer control over your inference you might want to build your own code around the YOLOv5 PyTorch Hub autoshape models:
https://github.com/ultralytics/yolov5#pytorch-hub

[bulingda]

@bulingda we don't generally assist in debugging code modifications.

You can see an example of a multithreaded dataloader here, though this is only used for multiple webcam streams at the moment:

yolov5/utils/datasets.py

Line 258 in b1cf25d

class LoadStreams: # multiple IP or RTSP cameras

If you want finer control over your inference you might want to build your own code around the YOLOv5 PyTorch Hub autoshape models:
https://github.com/ultralytics/yolov5#pytorch-hub

Thank you very much for your tips

[olehb]

@bulingda check out this PR https://github.com/ultralytics/yolov5/pull/2120/files
I removed mutable state from Detect class and it made the model thread-safe in my service. Hope it helps.

[glenn-jocher]

@olehb @bulingda ah I see the problem here. I wonder if there's a different way to solve this that doesn't require grid recomputation on every image? It's not just that it adds additional pytorch inference overhead, but it also makes export more complicated if the grid computation is included in the export (for this reason the export code runs inference once (to capture grids as constants) before actually tracing the model (using the grid constants) to produce a simplified export.

[bulingda]

Thank you very much for your attention. I have also solved this problem. Here is my solution. @olehb @glenn-jocher

[glenn-jocher]

@bulingda @olehb perhaps another option is to create a single large grid on init (rather than 3 smaller ones), and then simply slice the large grid to what's required for each output, i.e.:
grid = self.grid[:, :, :nx, :ny]

I profiled this to be about 3x faster than creating a grid with self._make_grid(), and this would avoid redefining the grid, so theoretically thread safe?

[Simon717]

I encountered the same problem, the core is that different threads have made different modifications to self.grid

[glenn-jocher]

@Simon717 see PR #2120 for an inference thread-safe version which handles Detect() grid layer differently.