Reduce inference time

[devendraswamy]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Need a solution/suggestion for reducing the 2sec inference time to less then 1 sec.

Additional

Hi, I developed a 4 models for 4 object detectionsin single inference script and I was successfully inference the 4 models with 2 sec at once. Actually I need a solution for reducing the 2sec inference time to less then 1 sec. I'm trying with torch.set_num_threads( ) and multiprocessing separately but I did not getting results in 1 sec. kindly help me to reduce the inference time(in cpu). Thank you in advance.

[github-actions]

👋 Hello @devendraswamy, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email support@ultralytics.com.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@devendraswamy 👋 Hello! Thanks for asking about inference speed issues. YOLOv5 🚀 can be run on CPU (i.e. --device cpu, slow) or GPU if available (i.e. --device 0, faster). You can determine your inference device by viewing the YOLOv5 console output:

detect.py inference

python detect.py --weights yolov5s.pt --img 640 --conf 0.25 --source data/images/

YOLOv5 PyTorch Hub inference

import torch

# Model
model = torch.hub.load('ultralytics/yolov5', 'yolov5s')

# Images
dir = 'https://ultralytics.com/images/'
imgs = [dir + f for f in ('zidane.jpg', 'bus.jpg')]  # batch of images

# Inference
results = model(imgs)
results.print()  # or .show(), .save()
# Speed: 631.5ms pre-process, 19.2ms inference, 1.6ms NMS per image at shape (2, 3, 640, 640)

Increase Speeds

If you would like to increase your inference speed some options are:

• Use batched inference with YOLOv5 PyTorch Hub
• Reduce --img-size, i.e. 1280 -> 640 -> 320
• Reduce model size, i.e. YOLOv5x -> YOLOv5l -> YOLOv5m -> YOLOv5s -> YOLOv5n
• Use half precision FP16 inference with python detect.py --half and python val.py --half
• Use a faster GPUs, i.e.: P100 -> V100 -> A100
• Export to ONNX or OpenVINO for up to 3x CPU speedup (CPU Benchmarks)
• Export to TensorRT for up to 5x GPU speedup
• Use a free GPU backends with up to 16GB of CUDA memory:

Good luck 🍀 and let us know if you have any other questions!

[devendraswamy]

@devendraswamy 👋 Hello! Thanks for asking about inference speed issues. YOLOv5 🚀 can be run on CPU (i.e. --device cpu, slow) or GPU if available (i.e. --device 0, faster). You can determine your inference device by viewing the YOLOv5 console output:

detect.py inference

python detect.py --weights yolov5s.pt --img 640 --conf 0.25 --source data/images/

YOLOv5 PyTorch Hub inference

import torch

# Model
model = torch.hub.load('ultralytics/yolov5', 'yolov5s')

# Images
dir = 'https://ultralytics.com/images/'
imgs = [dir + f for f in ('zidane.jpg', 'bus.jpg')]  # batch of images

# Inference
results = model(imgs)
results.print()  # or .show(), .save()
# Speed: 631.5ms pre-process, 19.2ms inference, 1.6ms NMS per image at shape (2, 3, 640, 640)

Increase Speeds

If you would like to increase your inference speed some options are:

• Use batched inference with YOLOv5 PyTorch Hub
• Reduce --img-size, i.e. 1280 -> 640 -> 320
• Reduce model size, i.e. YOLOv5x -> YOLOv5l -> YOLOv5m -> YOLOv5s -> YOLOv5n
• Use half precision FP16 inference with python detect.py --half and python val.py --half
• Use a faster GPUs, i.e.: P100 -> V100 -> A100
• Export to ONNX or OpenVINO for up to 3x CPU speedup (CPU Benchmarks)
• Export to TensorRT for up to 5x GPU speedup
• Use a free GPU backends with up to 16GB of CUDA memory:

Good luck 🍀 and let us know if you have any other questions!

Thank you for your valuable reply and I am using yolov5s model with 640 image size and pytorch 1.10.1 and also I tried the inference with ONNX model with 640 image size but my inference time is 2sec only. As for now my prediction time is 2 sec for 4 models. if possible please guide me to reduce the inference time to 1sec.

[devendraswamy]

@devendraswamy 👋 Hello! Thanks for asking about inference speed issues. YOLOv5 🚀 can be run on CPU (i.e. --device cpu, slow) or GPU if available (i.e. --device 0, faster). You can determine your inference device by viewing the YOLOv5 console output:

detect.py inference

python detect.py --weights yolov5s.pt --img 640 --conf 0.25 --source data/images/

### YOLOv5 PyTorch [Hub](https://docs.ultralytics.com/yolov5/tutorials/pytorch_hub_model_loading) inference ```python import torch

Model

model = torch.hub.load('ultralytics/yolov5', 'yolov5s')

Images

dir = 'https://ultralytics.com/images/'
imgs = [dir + f for f in ('zidane.jpg', 'bus.jpg')] # batch of images

Inference

results = model(imgs)
results.print() # or .show(), .save()

Speed: 631.5ms pre-process, 19.2ms inference, 1.6ms NMS per image at shape (2, 3, 640, 640)

    
      
    

      
    

    
  
### Increase Speeds
If you would like to increase your inference speed some options are:

* Use batched inference with [YOLOv5 PyTorch Hub](https://docs.ultralytics.com/yolov5/tutorials/pytorch_hub_model_loading)
* Reduce `--img-size`, i.e. 1280 -> 640 -> 320
* Reduce model size, i.e. YOLOv5x -> YOLOv5l -> YOLOv5m -> YOLOv5s -> YOLOv5n
* Use half precision FP16 inference with `python detect.py --half` and `python val.py --half`
* Use a faster GPUs, i.e.: P100 -> V100 -> A100
* [Export](https://docs.ultralytics.com/yolov5/tutorials/model_export) to ONNX or OpenVINO for up to 3x CPU speedup ([CPU Benchmarks](https://github.com/ultralytics/yolov5/pull/6613))
* [Export](https://docs.ultralytics.com/yolov5/tutorials/model_export) to TensorRT for up to 5x GPU speedup
* Use a free GPU backends with up to 16GB of CUDA memory: [![Open In Colab](https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667)](https://colab.research.google.com/github/ultralytics/yolov5/blob/master/tutorial.ipynb) [![Open In Kaggle](https://camo.githubusercontent.com/a08ca511178e691ace596a95d334f73cf4ce06e83a5c4a5169b8bb68cac27bef/68747470733a2f2f6b6167676c652e636f6d2f7374617469632f696d616765732f6f70656e2d696e2d6b6167676c652e737667)](https://www.kaggle.com/models/ultralytics/yolov5)

Good luck 🍀 and let us know if you have any other questions!

Thank you for your valuable reply and I am using yolov5s model with 640 image size and pytorch 1.10.1 and also I tried the inference with ONNX model with 640 image size but my inference time is 2sec only. As for now my prediction time is 2 sec for 4 models. if possible please guide me to reduce the inference time to 1sec.

Any one please help to reduce the inference time. Thank you.

[DavidBaldsiefen]

@devendraswamy did you attempt using a half-precision model?

If you have a NVIDIA GPU available, the easiest way to speed up inference may be to use TensorRT:

python export.py --weights my-weights.pt --include engine --device 0 then python detect.py --weights my-weights.engine --device 0

[devendraswamy]

Thank you for your valuable reply and here the problem is the machine will continuously run with CPU support only. That's why i didn't tried with half precision method. If any suggestions are their, Kindly reply. Thank you in advance for your support.

…

On Mon, Mar 7, 2022, 18:32 DavidB ***@***.***> wrote: @devendraswamy <https://github.com/devendraswamy> did you attempt using a half-precision model? If you have a NVIDIA GPU available, the easiest way to speed up inference may be to use TensorRT: python export.py --weights my-weights.pt --include engine --device 0 then python detect.py --weights my-weights.engine --device 0 — Reply to this email directly, view it on GitHub <#6736 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLD6LSOLFO5XGUXRZOVPXLU6X46DANCNFSM5PANXNMA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[DavidBaldsiefen]

Is your goal increasing throughput or reducing latency? If it is about throughput, you could try increasing batch size. Latency is more difficult to increase, especially on CPU-bound devices. Do you use a custom trained model?

[devendraswamy]

My goal is reduce the inference time and I'm using custom model trained weights for inference. Actually it's taking 2 sec for inference the all the four objects with four different weight files at a once. I need to decreases the four models inference time to 1 sec , any help grateful to you.

…

On Mon, Mar 7, 2022, 18:44 DavidB ***@***.***> wrote: Is your goal increasing throughput or reducing latency? If it is about throughput, you could try increasing batch size. Latency is more difficult to increase, especially on CPU-bound devices. Do you use a custom trained model? — Reply to this email directly, view it on GitHub <#6736 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLD6LTBSEFE47MRD45PN73U6X6LFANCNFSM5PANXNMA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[DavidBaldsiefen]

Is there a reason you are using four different models as opposed to one? Do I understand correctly that you try to run inference on a single image with four different models simultaneously?

[devendraswamy]

Thank you for your support . The hardware will gives four parts of image , i have to detect the objects in that four parts image and gives the results to front end with in 1 sec. currently its taking 2 sec , any suggestions kindly let me know.

[DavidBaldsiefen]

@devendraswamy have you tried passing all four images at once by using batch-size 4?

[devendraswamy]

@devendraswamy have you tried passing all four images at once by using batch-size 4?

That was not possible because four images are predicted with four different weight files , any other suggestions ,please let me know.

[glenn-jocher]

@devendraswamy individual models may provide the absolute best mAP, but for most use cases best practice would be to combine your four datasets into one 4-class dataset and train a single model on it. See https://community.ultralytics.com/t/how-to-combine-weights-to-detect-from-multiple-datasets/38/9

[github-actions]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

• Wiki – https://github.com/ultralytics/yolov5/wiki
• Tutorials – https://docs.ultralytics.com/yolov5
• Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

• Ultralytics HUB – https://ultralytics.com/hub
• Vision API – https://ultralytics.com/yolov5
• About Us – https://ultralytics.com/about
• Join Our Team – https://ultralytics.com/work
• Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

[devendraswamy]

But As per Accuracy concern , I need to to train the models individually , Please let me know any another solution for Inference time reduction , I train the all the model with 640 image resolution and I am using onnx models.

Thank you in advance

[PankajBarai]

My goal is reduce the inference time and I'm using custom model trained weights for inference. Actually it's taking 2 sec for inference the all the four objects with four different weight files at a once. I need to decreases the four models inference time to 1 sec , any help grateful to you.
…
On Mon, Mar 7, 2022, 18:44 DavidB @.> wrote: Is your goal increasing throughput or reducing latency? If it is about throughput, you could try increasing batch size. Latency is more difficult to increase, especially on CPU-bound devices. Do you use a custom trained model? — Reply to this email directly, view it on GitHub <#6736 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALLD6LTBSEFE47MRD45PN73U6X6LFANCNFSM5PANXNMA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.>

Can you share how u reduce inference time

[glenn-jocher]

@PankajBarai 👋 Hello! Thanks for asking about inference speed issues. PyTorch Hub speeds will vary by hardware, software, model, inference settings, etc. Our default example in Colab with a V100 looks like this:

YOLOv5 🚀 can be run on CPU (i.e. --device cpu, slow) or GPU if available (i.e. --device 0, faster). You can determine your inference device by viewing the YOLOv5 console output:

detect.py inference

python detect.py --weights yolov5s.pt --img 640 --conf 0.25 --source data/images/

YOLOv5 PyTorch Hub inference

import torch

# Model
model = torch.hub.load('ultralytics/yolov5', 'yolov5s')

# Images
dir = 'https://ultralytics.com/images/'
imgs = [dir + f for f in ('zidane.jpg', 'bus.jpg')]  # batch of images

# Inference
results = model(imgs)
results.print()  # or .show(), .save()
# Speed: 631.5ms pre-process, 19.2ms inference, 1.6ms NMS per image at shape (2, 3, 640, 640)

Increase Speeds

If you would like to increase your inference speed some options are:

• Use batched inference with YOLOv5 PyTorch Hub
• Reduce --img-size, i.e. 1280 -> 640 -> 320
• Reduce model size, i.e. YOLOv5x -> YOLOv5l -> YOLOv5m -> YOLOv5s -> YOLOv5n
• Use half precision FP16 inference with python detect.py --half and python val.py --half
• Use a faster GPUs, i.e.: P100 -> V100 -> A100
• Export to ONNX or OpenVINO for up to 3x CPU speedup (CPU Benchmarks)
• Export to TensorRT for up to 5x GPU speedup (GPU Benchmarks)
• Use a free GPU backends with up to 16GB of CUDA memory:

Good luck 🍀 and let us know if you have any other questions!