Exporting an INT8 tensorrt engine

[LieceC]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

I have been trying for a while to modify the yolov5 code in order to export an int8 tensorrt engine on a jetson xavier AGX using tensorrt 8.2.3.0 on python 3.7. However, I am facing an error during the serialization. I'm not sure it has been answered before.
I'm using the latest yolov5n model, that's the command I used for the export:
python export.py --weights yolov5n_dot_two.pt --include engine --int8 --device 0 --imgsz 5056 > outputs.txt

When a certain tactic appear for the Conv_3 operation, it seems to provoke an error, here is the last few lines of tensorrt in verbose mode :
[04/07/2022-14:36:52] [TRT] [V] Conv_3 Set Tactic Name: sm70_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x256x8_stage1_warpsize1x4x1_g1_ffma Tactic: 929479145088554885 [04/07/2022-14:36:52] [TRT] [V] Deleting timing cache: 341 entries, 964 hits [04/07/2022-14:36:52] [TRT] [E] 1: Unexpected exception

One thing to note is that the export for fp32 and fp16 version works without any flaw and only the int8 version seems to have this error. I can attach my calibration.py file if necessary.

Additional

No response

[github-actions]

👋 Hello @LieceC, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email support@ultralytics.com.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@LieceC not all combination of arguments are supported for all export formats. --int8 is intended for CoreML and TFLite quantization, i.e.

yolov5/export.py

Line 554 in b7faeda

parser.add_argument('--int8', action='store_true', help='CoreML/TF INT8 quantization')

If you'd like to try to extend int8 support to TRT that would be awesome though!

Please see our ✅ Contributing Guide to get started.

[LieceC]

Hello @glenn-jocher, I know its not supported as of now so I tried modify it by adding a calibration file and slightly modifying the export_engine function. I believe I have seen somehow talk about Int8 not having relevant results in one of the issues (Maybe the one that did the engine function ?).

[glenn-jocher]

@LieceC I don't know about the feasibility or requirements for INT8 TRT exports.

@imyhxy implemented the baseline export in #5699 and then export was updated to forced FP16 mode in #6798 by @DavidBaldsiefen. The current export is fixed at FP16 regardless of arguments, with the model automatically casting FP32 inputs to FP16 if necessary.

[LieceC]

I see, I'll look at it and see if I can find where that error come from

[RoyCopter]

Search before asking

* [x]  I have searched the YOLOv5 [issues](https://github.com/ultralytics/yolov5/issues) and [discussions](https://github.com/ultralytics/yolov5/discussions) and found no similar questions.

Question

I have been trying for a while to modify the yolov5 code in order to export an int8 tensorrt engine on a jetson xavier AGX using tensorrt 8.2.3.0 on python 3.7. However, I am facing an error during the serialization. I'm not sure it has been answered before. I'm using the latest yolov5n model, that's the command I used for the export: python export.py --weights yolov5n_dot_two.pt --include engine --int8 --device 0 --imgsz 5056 > outputs.txt

When a certain tactic appear for the Conv_3 operation, it seems to provoke an error, here is the last few lines of tensorrt in verbose mode : [04/07/2022-14:36:52] [TRT] [V] Conv_3 Set Tactic Name: sm70_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x256x8_stage1_warpsize1x4x1_g1_ffma Tactic: 929479145088554885 [04/07/2022-14:36:52] [TRT] [V] Deleting timing cache: 341 entries, 964 hits [04/07/2022-14:36:52] [TRT] [E] 1: Unexpected exception

One thing to note is that the export for fp32 and fp16 version works without any flaw and only the int8 version seems to have this error. I can attach my calibration.py file if necessary.

Additional

No response

@LieceC how did you manage to install tensorrt/pytorch and everthing with python3.7 on a Jetson platform?

[LieceC]

Search before asking

* [x]  I have searched the YOLOv5 [issues](https://github.com/ultralytics/yolov5/issues) and [discussions](https://github.com/ultralytics/yolov5/discussions) and found no similar questions.

Question

I have been trying for a while to modify the yolov5 code in order to export an int8 tensorrt engine on a jetson xavier AGX using tensorrt 8.2.3.0 on python 3.7. However, I am facing an error during the serialization. I'm not sure it has been answered before. I'm using the latest yolov5n model, that's the command I used for the export: python export.py --weights yolov5n_dot_two.pt --include engine --int8 --device 0 --imgsz 5056 > outputs.txt
When a certain tactic appear for the Conv_3 operation, it seems to provoke an error, here is the last few lines of tensorrt in verbose mode : [04/07/2022-14:36:52] [TRT] [V] Conv_3 Set Tactic Name: sm70_xmma_fprop_implicit_gemm_f32f32_f32f32_f32_nhwckrsc_nhwc_tilesize64x256x8_stage1_warpsize1x4x1_g1_ffma Tactic: 929479145088554885 [04/07/2022-14:36:52] [TRT] [V] Deleting timing cache: 341 entries, 964 hits [04/07/2022-14:36:52] [TRT] [E] 1: Unexpected exception
One thing to note is that the export for fp32 and fp16 version works without any flaw and only the int8 version seems to have this error. I can attach my calibration.py file if necessary.

Additional

No response

@LieceC how did you manage to install tensorrt/pytorch and everthing with python3.7 on a Jetson platform?

I compiled them from source

[LieceC]

I suspect the problem comes from the TensorRT python bindings, generating an int8 file from the trtexec command seems to work without any issue.