yolov5s fp16 reasoning is slower than fp32

[QZ-cmd]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

I executed yolov5s on jetson agx orin and found that fp16 (--half) is slower than fp32 reasoning, but it is indeed normal on jetson Xavier nx (fp16 is faster than fp32). What may be the reason for this?


As shown in the figure, the --half time is turned on but slows down
Looking forward to your reply, thank you

Additional

No response

[github-actions]

👋 Hello @QZ-cmd, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

Introducing YOLOv8 🚀

We're excited to announce the launch of our latest state-of-the-art (SOTA) object detection model for 2023 - YOLOv8 🚀!

Designed to be fast, accurate, and easy to use, YOLOv8 is an ideal choice for a wide range of object detection, image segmentation and image classification tasks. With YOLOv8, you'll be able to quickly and accurately detect objects in real-time, streamline your workflows, and achieve new levels of accuracy in your projects.

Check out our YOLOv8 Docs for details and get started with:

pip install ultralytics

[glenn-jocher]

@QZ-cmd hi there! Thank you for reaching out. We appreciate your interest in YOLOv5 and your question.

Regarding the performance difference between fp16 and fp32 reasoning on the Jetson AGX Orin and Xavier NX, there could be several factors contributing to this behavior.

The performance of mixed-precision training (fp16) depends on the specific hardware and its optimizations. It's possible that the Jetson AGX Orin's architecture or optimizations are better suited for full-precision (fp32) computations, resulting in slower fp16 reasoning compared to fp32.

One suggestion is to try different configurations and experiment with other settings, such as the --precision flag, to see if it improves the performance of fp16 reasoning on the Jetson AGX Orin.

Moreover, we encourage you to consult the Jetson AGX Orin documentation and seek advice from the Jetson community, as they might be better equipped to answer hardware-specific performance questions.

Please let us know if you have any further questions. We're here to help!

[QZ-cmd]

I asked questions on the jetson forum, and tested the inference time of yolov5s for FP32 and FP16 under the framework of tensorRT on Jetson AGX Orin according to their advice, and found that FP16 was indeed faster than FP32. So the same code is FP16 (--half) faster than FP32 based on the PyTorch framework on Jetson Xavier NX, but FP16 is slower than FP32 based on the PyTorch framework on Jetson AGX Orin, if you have better suggestions feel free to let me know, thank you for your reply。

This is my test result under tensorRT, you can compare the test results under Pytorch above, indeed FP16 is improved for FP32

[glenn-jocher]

@QZ-cmd thank you for sharing your findings and testing the inference time of YOLOv5s on Jetson AGX Orin using TensorRT. It's interesting to note that you found that FP16 inference is faster than FP32 inference on Jetson AGX Orin under the TensorRT framework.

The performance of different data types can vary depending on several factors including the specific hardware architecture and optimizations of the framework being used. It seems that in the case of Jetson AGX Orin, TensorRT's optimizations for FP16 inference are more efficient than PyTorch's optimizations for FP16.

We appreciate you bringing this to our attention and providing additional information. Your insights can be helpful for others in the YOLOv5 community who may be using Jetson AGX Orin.

If you have any further questions or need assistance, please don't hesitate to reach out. We're here to help!

[IamShubhamGupto]

@QZ-cmd
I find this information, very interesting. Since I am running yolov5 on AGX orin and Xavier NX as well. I would like to know more about the experiment you did.

Could you elaborate on the tensor rt framework? Currently, I use yolov5 's export.py to generate tensor rt files and load them regularly using torch.load. Is your method different?

If you could show with code snippets that would be great

Thanks

[glenn-jocher]

@IamShubhamGupto

Thank you for your interest and follow-up question!

In my experiment, I used the TensorRT framework to optimize the YOLOv5 model inference on the Jetson AGX Orin and Xavier NX devices. The process involved converting the YOLOv5 model into the TensorRT engine format and then loading and running the optimized engine for inference.

To convert the YOLOv5 model to the TensorRT engine format, I followed the steps outlined in the export.py script provided in the YOLOv5 repository. After exporting the model, I used the TensorRT API to load and run the optimized engine.

Here's an example of how the code would look like:

import torch
import tensorrt as trt
from torch2trt import TRTModule

# Load the exported TensorRT engine
engine_path = 'path/to/your/engine.plan'
trt_model = TRTModule()
trt_model.load_state_dict(torch.load(engine_path))

# Prepare input data
input_data = torch.randn(1, 3, 640, 640).cuda()

# Run inference
output = trt_model(input_data)

# Process the output as needed
# ...

Please make sure you have the necessary dependencies and packages installed, such as TensorRT and torch2trt, to enable this workflow.

I hope this clarifies the process for you. If you have any further questions, feel free to ask!

[QZ-cmd]

Do you also encounter that yolov5s under the pytorch framework fp16 is slower than fp32 on jetson agx orin? My tensorRT execution file is generated with export.py, and it behaves normally.

[github-actions]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

[glenn-jocher]

@QZ-cmd

Thank you for your question. In my experiments with YOLOv5s under the PyTorch framework, I did observe the behavior you mentioned, where fp16 inference was slower than fp32 on Jetson AGX Orin. However, when using the TensorRT execution file generated with export.py, the performance normalized.

It seems that the optimizations provided by TensorRT for fp16 inference on Jetson AGX Orin are more effective compared to those within the PyTorch framework.

I'm glad to hear that your TensorRT execution file behaves normally. If you have any more questions or need further assistance, feel free to ask. I'm here to help!