export RTDETR to Openvino failed

[Kayzwer]

Search before asking

• I have searched the YOLOv8 issues and found no similar bug report.

YOLOv8 Component

Export

Bug

terminal output

Ultralytics YOLOv8.1.27 🚀 Python-3.11.7 torch-2.2.1+cu121 CPU (12th Gen Intel Core(TM) i7-12650H)
rt-detr-l summary: 494 layers, 32148140 parameters, 0 gradients, 103.8 GFLOPs

PyTorch: starting from 'rtdetr-l.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 300, 84) (63.4 MB)

OpenVINO: starting export with openvino 2024.0.0-14509-34caeefd078-releases/2024/0...
OpenVINO: export failure ❌ 4.9s: Check 'is_conversion_successful' failed at src/frontends/pytorch/src/frontend.cpp:143:
FrontEnd API failed with OpConversionFailure:
Model wasn't fully converted. Failed operations detailed log:
-- aten::mul with a message:
Exception happened during conversion of operation __module.model.28/aten::mul with schema aten::mul.Tensor(Tensor self, Tensor other) -> Tensor
Check 'args_et.is_dynamic() || args_et != element::boolean' failed at src/core/src/op/util/binary_elementwise_arithmetic.cpp:25:
While validating node 'opset1::Multiply Multiply_8133 (opset1::Greater __module.model.28/aten::gt/Greater[0]:boolean[...], opset1::Less __module.model.28/aten::lt/Less[0]:boolean[...]) -> (dynamic[...])' with friendly_name 'Multiply_8133':
Arguments cannot have boolean element type (argument element type: boolean).

Summary:
-- Conversion is failed for: aten::mul

Traceback (most recent call last):
  File "/home/imagineai/ultralytics/temp.py", line 5, in <module>
    model.export(format="openvino")
  File "/home/imagineai/ultralytics/ultralytics/engine/model.py", line 588, in export
    return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imagineai/env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/imagineai/ultralytics/ultralytics/engine/exporter.py", line 289, in __call__
    f[3], _ = self.export_openvino()
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imagineai/ultralytics/ultralytics/engine/exporter.py", line 138, in outer_func
    raise e
  File "/home/imagineai/ultralytics/ultralytics/engine/exporter.py", line 133, in outer_func
    f, model = inner_func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imagineai/ultralytics/ultralytics/engine/exporter.py", line 419, in export_openvino
    ov_model = ov.convert_model(
               ^^^^^^^^^^^^^^^^^
  File "/home/imagineai/env/lib/python3.11/site-packages/openvino/tools/ovc/convert.py", line 100, in convert_model
    ov_model, _ = _convert(cli_parser, params, True)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imagineai/env/lib/python3.11/site-packages/openvino/tools/ovc/convert_impl.py", line 535, in _convert
    raise e
  File "/home/imagineai/env/lib/python3.11/site-packages/openvino/tools/ovc/convert_impl.py", line 477, in _convert
    ov_model = driver(argv, {"conversion_parameters": non_default_params})
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imagineai/env/lib/python3.11/site-packages/openvino/tools/ovc/convert_impl.py", line 228, in driver
    ov_model = moc_emit_ir(prepare_ir(argv), argv)
                           ^^^^^^^^^^^^^^^^
  File "/home/imagineai/env/lib/python3.11/site-packages/openvino/tools/ovc/convert_impl.py", line 177, in prepare_ir
    ov_model = moc_pipeline(argv, moc_front_end)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imagineai/env/lib/python3.11/site-packages/openvino/tools/ovc/moc_frontend/pipeline.py", line 244, in moc_pipeline
    ov_model = moc_front_end.convert(input_model)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imagineai/env/lib/python3.11/site-packages/openvino/frontend/frontend.py", line 18, in convert
    converted_model = super().convert(model)
                      ^^^^^^^^^^^^^^^^^^^^^^
openvino._pyopenvino.OpConversionFailure: Check 'is_conversion_successful' failed at src/frontends/pytorch/src/frontend.cpp:143:
FrontEnd API failed with OpConversionFailure:
Model wasn't fully converted. Failed operations detailed log:
-- aten::mul with a message:
Exception happened during conversion of operation __module.model.28/aten::mul with schema aten::mul.Tensor(Tensor self, Tensor other) -> Tensor
Check 'args_et.is_dynamic() || args_et != element::boolean' failed at src/core/src/op/util/binary_elementwise_arithmetic.cpp:25:
While validating node 'opset1::Multiply Multiply_8133 (opset1::Greater __module.model.28/aten::gt/Greater[0]:boolean[...], opset1::Less __module.model.28/aten::lt/Less[0]:boolean[...]) -> (dynamic[...])' with friendly_name 'Multiply_8133':
Arguments cannot have boolean element type (argument element type: boolean).

Summary:
-- Conversion is failed for: aten::mul

Environment

No response

Minimal Reproducible Example

from ultralytics import RTDETR

model = RTDETR("./rtdetr-l.pt")
model.export(format="openvino")

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[github-actions]

👋 Hello @Kayzwer, thank you for your interest in Ultralytics YOLOv8 🚀! We recommend a visit to the Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Join the vibrant Ultralytics Discord 🎧 community for real-time conversations and collaborations. This platform offers a perfect space to inquire, showcase your work, and connect with fellow Ultralytics users.

Install

Pip install the ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8.

pip install ultralytics

Environments

YOLOv8 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLOv8 Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@Kayzwer hey there! 👋 It looks like you've encountered an issue while trying to export the RTDETR model to OpenVINO. The error message suggests a problem with the aten::mul operation, specifically when trying to convert operations that involve boolean tensors.

This is a known limitation with certain operations in PyTorch models when converting to OpenVINO, as OpenVINO expects inputs to be non-boolean for the Multiply operation.

A potential workaround could involve modifying the model to replace boolean operations with equivalent operations that result in non-boolean tensors before export. However, this might require some changes to the model's source code.

As an immediate step, could you please ensure that you're using the latest versions of both Ultralytics YOLOv8 and OpenVINO? Sometimes, compatibility issues are resolved in newer releases.

If the issue persists, it would be great if you could open an issue on the Ultralytics YOLOv8 GitHub repository with the details you've provided here. Our team can then look into this more closely and work on a fix or provide a more specific workaround.

Thanks for bringing this to our attention, and we appreciate your patience as we work through this! 🚀

[Kayzwer]

@Kayzwer hey there! 👋 It looks like you've encountered an issue while trying to export the RTDETR model to OpenVINO. The error message suggests a problem with the aten::mul operation, specifically when trying to convert operations that involve boolean tensors.

This is a known limitation with certain operations in PyTorch models when converting to OpenVINO, as OpenVINO expects inputs to be non-boolean for the Multiply operation.

A potential workaround could involve modifying the model to replace boolean operations with equivalent operations that result in non-boolean tensors before export. However, this might require some changes to the model's source code.

As an immediate step, could you please ensure that you're using the latest versions of both Ultralytics YOLOv8 and OpenVINO? Sometimes, compatibility issues are resolved in newer releases.

If the issue persists, it would be great if you could open an issue on the Ultralytics YOLOv8 GitHub repository with the details you've provided here. Our team can then look into this more closely and work on a fix or provide a more specific workaround.

Thanks for bringing this to our attention, and we appreciate your patience as we work through this! 🚀

I will do a quick check later to look for multiplication operations that involve boolean

[Kayzwer]

@glenn-jocher this pull request fixes the RTDETR openvino export problem.

[glenn-jocher]

Thanks for the update and the pull request! 🚀 We'll review it shortly to ensure everything aligns with our standards and then proceed from there. Great work on addressing the RTDETR export issue!