Make compatible with TensorRT 10 release #9510

Burhan-Q · 2024-04-02T12:38:50Z

Search before asking

I have searched the YOLOv8 issues and found no similar bug report.

YOLOv8 Component

Other

Bug

Looks like new/different methods with TensorRT 10 that aren't compatible with the existing implementation in Autobackend class. Have not yet tested export, but wanted to open issue. Will add details with attempt for exporting.

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>
  File "Q:\ML_dev\tensorrt_test\.trt\Lib\site-packages\ultralytics\engine\model.py", line 423, in predict
    self.predictor.setup_model(model=self.model, verbose=is_cli)
  File "Q:\ML_dev\tensorrt_test\.trt\Lib\site-packages\ultralytics\engine\predictor.py", line 341, in setup_model
    self.model = AutoBackend(
                 ^^^^^^^^^^^^
  File "Q:\ML_dev\tensorrt_test\.trt\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "Q:\ML_dev\tensorrt_test\.trt\Lib\site-packages\ultralytics\nn\autobackend.py", line 216, in __init__
    context = model.create_execution_context()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'create_execution_context'

Environment

Ultralytics YOLOv8.1.16 🚀 Python-3.11.6 torch-2.2.0+cu121 CUDA:0 (NVIDIA GeForce RTX 3080, 12288MiB)
Setup complete ✅ (12 CPUs, 31.9 GB RAM, 391.9/1863.0 GB disk)

OS                  Windows-10-10.0.19045-SP0
Environment         Windows
Python              3.11.6
Install             pip
RAM                 31.86 GB
CPU                 Intel Core(TM) i5-10600K 4.10GHz
CUDA                12.1

matplotlib          ✅ 3.8.2>=3.3.0
opencv-python       ✅ 4.9.0.80>=4.6.0
pillow              ✅ 9.3.0>=7.1.2
pyyaml              ✅ 6.0.1>=5.3.1
requests            ✅ 2.28.1>=2.23.0
scipy               ✅ 1.12.0>=1.4.1
torch               ✅ 2.2.0+cu121>=1.8.0
torchvision         ✅ 0.17.0+cu121>=0.9.0
tqdm                ✅ 4.66.1>=4.64.0
psutil              ✅ 5.9.8
py-cpuinfo          ✅ 9.0.0
thop                ✅ 0.1.1-2209072238>=0.1.1
pandas              ✅ 2.2.0>=1.1.4
seaborn             ✅ 0.13.1>=0.11.0

TensorRT info

nvidia-cublas-cu12       12.3.4.1
nvidia-cuda-nvrtc-cu12   12.3.107
nvidia-cuda-runtime-cu12 12.3.101
nvidia-cudnn-cu12        8.9.7.29
tensorrt                 10.0.0b6
tensorrt-cu12            10.0.0b6
tensorrt-cu12_bindings   10.0.0b6
tensorrt-cu12_libs       10.0.0b6

Minimal Reproducible Example

from ultralytics import YOLO

model = YOLO("yolov8n.engine", task="detect")
results = model.predict()

Additional

Same issue present on Linux

Ultralytics YOLOv8.1.42 🚀 Python-3.11.7 torch-2.1.2+cu121 CUDA:0 (NVIDIA GeForce RTX 2060, 5924MiB)
Setup complete ✅ (12 CPUs, 15.6 GB RAM, 95.6/101.0 GB disk)

OS                  Linux-6.6.10-76060610-generic-x86_64-with-glibc2.35
Environment         Linux
Python              3.11.7
Install             pip
RAM                 15.56 GB
CPU                 AMD Ryzen 5 1600 Six-Core Processor
CUDA                12.1

matplotlib          ✅ 3.8.3>=3.3.0
opencv-python       ✅ 4.9.0.80>=4.6.0
pillow              ✅ 10.3.0>=7.1.2
pyyaml              ✅ 6.0.1>=5.3.1
requests            ✅ 2.31.0>=2.23.0
scipy               ✅ 1.12.0>=1.4.1
torch               ✅ 2.1.2+cu121>=1.8.0
torchvision         ✅ 0.16.2+cu121>=0.9.0
tqdm                ✅ 4.66.2>=4.64.0
psutil              ✅ 5.9.8
py-cpuinfo          ✅ 9.0.0
thop                ✅ 0.1.1-2209072238>=0.1.1
pandas              ✅ 2.2.1>=1.1.4
seaborn             ✅ 0.13.2>=0.11.0

TensorRT info

nvidia-cublas-cu12       12.1.3.1
nvidia-cuda-cupti-cu12   12.1.105
nvidia-cuda-nvrtc-cu12   12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12        8.9.2.26
nvidia-cufft-cu12        11.0.2.54
nvidia-curand-cu12       10.3.2.106
nvidia-cusolver-cu12     11.4.5.107
nvidia-cusparse-cu12     12.1.0.106
nvidia-nccl-cu12         2.19.3
nvidia-nvjitlink-cu12    12.4.99
nvidia-nvtx-cu12         12.1.105
tensorrt                 10.0.0b6
tensorrt-cu12            10.0.0b6
tensorrt-cu12_bindings   10.0.0b6
tensorrt-cu12_libs       10.0.0b6

Are you willing to submit a PR?

Yes I'd like to help by submitting a PR!

The text was updated successfully, but these errors were encountered:

Burhan-Q · 2024-04-02T12:47:06Z

Linux export attempt failed

from ultralytics import YOLO
model = YOLO("yolov8n.pt")
out = model.export(format="engine")

WARNING ⚠️ TensorRT requires GPU export, automatically assigning device=0
Ultralytics YOLOv8.1.42 🚀 Python-3.11.7 torch-2.1.2+cu121 CUDA:0 (NVIDIA GeForce RTX 2060, 5924MiB)
YOLOv8n summary (fused): 168 layers, 3151904 parameters, 0 gradients, 8.7 GFLOPs

ONNX: starting export with onnx 1.16.0 opset 17...
ONNX: export success ✅ 6.5s, saved as 'yolov8n.onnx' (12.2 MB)

TensorRT: starting export with TensorRT 10.0.0b6...
[04/02/2024-08:45:45] [TRT] [I] [MemUsageChange] Init CUDA: CPU +2, GPU +0, now: CPU 562, GPU 569 (MiB)
[04/02/2024-08:45:47] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +944, GPU +178, now: CPU 1642, GPU 747 (MiB)

TensorRT: export failure ❌ 9.1s: 'tensorrt_bindings.tensorrt.IBuilderConfig' object has no attribute 'max_workspace_size'

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "trt10/.trt10/lib/python3.11/site-packages/ultralytics/engine/model.py", line 600, in export
    return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "trt10/.trt10/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "trt10/.trt10/lib/python3.11/site-packages/ultralytics/engine/exporter.py", line 285, in __call__
    f[1], _ = self.export_engine()
              ^^^^^^^^^^^^^^^^^^^^
  File "trt10/.trt10/lib/python3.11/site-packages/ultralytics/engine/exporter.py", line 138, in outer_func
    raise e
  File "trt10/.trt10/lib/python3.11/site-packages/ultralytics/engine/exporter.py", line 133, in outer_func
    f, model = inner_func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "trt10/.trt10/lib/python3.11/site-packages/ultralytics/engine/exporter.py", line 678, in export_engine
    config.max_workspace_size = int(self.args.workspace * (1 << 30))
    ^^^^^^^^^^^^^^^^^^^^^^^^^

AttributeError: 'tensorrt_bindings.tensorrt.IBuilderConfig' object has no attribute 'max_workspace_size'

Burhan-Q · 2024-04-02T12:49:17Z

Windows export attempt failed

from ultralytics import YOLO
model = YOLO("yolov8n.pt")
out = model.export(format="engine")

WARNING ⚠️ TensorRT requires GPU export, automatically assigning device=0
Ultralytics YOLOv8.1.16 🚀 Python-3.11.6 torch-2.2.0+cu121 CUDA:0 (NVIDIA GeForce RTX 3080, 12288MiB)
YOLOv8n summary (fused): 168 layers, 3151904 parameters, 0 gradients, 8.7 GFLOPs

PyTorch: starting from 'yolov8n.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 84, 8400) (6.2 MB)

ONNX: starting export with onnx 1.15.0 opset 17...
ONNX: export success ✅ 0.8s, saved as 'yolov8n.onnx' (12.2 MB)

TensorRT: starting export with TensorRT 10.0.0b6...
[04/02/2024-08:47:47] [TRT] [I] [MemUsageChange] Init CUDA: CPU +3, GPU +0, now: CPU 19158, GPU 1574 (MiB)
[04/02/2024-08:47:50] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +2613, GPU +310, now: CPU 22026, GPU 1884 (MiB)

TensorRT: export failure ❌ 3.7s: 'tensorrt_bindings.tensorrt.IBuilderConfig' object has no attribute 'max_workspace_size'

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "Q:\ML_dev\tensorrt_test\.trt\Lib\site-packages\ultralytics\engine\model.py", line 578, in export
    return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Q:\ML_dev\tensorrt_test\.trt\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "Q:\ML_dev\tensorrt_test\.trt\Lib\site-packages\ultralytics\engine\exporter.py", line 275, in __call__
    f[1], _ = self.export_engine()
              ^^^^^^^^^^^^^^^^^^^^
  File "Q:\ML_dev\tensorrt_test\.trt\Lib\site-packages\ultralytics\engine\exporter.py", line 136, in outer_func
    raise e
  File "Q:\ML_dev\tensorrt_test\.trt\Lib\site-packages\ultralytics\engine\exporter.py", line 131, in outer_func
    f, model = inner_func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Q:\ML_dev\tensorrt_test\.trt\Lib\site-packages\ultralytics\engine\exporter.py", line 663, in export_engine
    config.max_workspace_size = self.args.workspace * 1 << 30
    ^^^^^^^^^^^^^^^^^^^^^^^^^

AttributeError: 'tensorrt_bindings.tensorrt.IBuilderConfig' object has no attribute 'max_workspace_size'

Burhan-Q · 2024-04-02T15:45:18Z

Confirmed error with Ultralytics YOLOv8.1.42 on Windows and Linux with TensorRT 10.0.0b6. Tested functional on Windows with TensorRT 9.2.0.post12.dev5 and Linux with TensorRT 8.6.1.post1

Burhan-Q · 2024-04-02T17:17:01Z

I think I have a couple of small patches to address issues with the export solved. The inference looks like it will need more effort to make work.

Burhan-Q · 2024-04-02T21:25:06Z

Note

Inference and export compatibility is working in #9516 but I would like to figure out a better way to handle version handling

Burhan-Q added bug Something isn't working dependencies Dependency-related topics labels Apr 2, 2024

Burhan-Q assigned glenn-jocher Apr 2, 2024

Burhan-Q linked a pull request Apr 2, 2024 that will close this issue

ultralytics 8.1.46 add TensorRT 10 support #9516

Merged

10 tasks

Burhan-Q self-assigned this Apr 2, 2024

glenn-jocher closed this as completed in #9516 Apr 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make compatible with TensorRT 10 release #9510

Make compatible with TensorRT 10 release #9510

Burhan-Q commented Apr 2, 2024 •

edited

Burhan-Q commented Apr 2, 2024 •

edited

Burhan-Q commented Apr 2, 2024

Burhan-Q commented Apr 2, 2024 •

edited

Burhan-Q commented Apr 2, 2024

Burhan-Q commented Apr 2, 2024

Make compatible with TensorRT 10 release #9510

Make compatible with TensorRT 10 release #9510

Comments

Burhan-Q commented Apr 2, 2024 • edited

Search before asking

YOLOv8 Component

Bug

Environment

TensorRT info

Minimal Reproducible Example

Additional

TensorRT info

Are you willing to submit a PR?

Burhan-Q commented Apr 2, 2024 • edited

Linux export attempt failed

Burhan-Q commented Apr 2, 2024

Windows export attempt failed

Burhan-Q commented Apr 2, 2024 • edited

Burhan-Q commented Apr 2, 2024

Burhan-Q commented Apr 2, 2024

Burhan-Q commented Apr 2, 2024 •

edited

Burhan-Q commented Apr 2, 2024 •

edited

Burhan-Q commented Apr 2, 2024 •

edited