TensorRTExecutionProvider error during session initialization

[cabinader]

Describe the issue

I'm exporting an nnUNetV2 model from torch to onnx fromat using torch.dynamo_export. I can launch the model with onnxruntime on the CPUExecutionProvider as well as the CUDAExecutionProvider. However, when using the TensorRTExecutionProvider I get the following error when initialiazing my session:

onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /onnxruntime_src/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc:2185 SubGraphCollection_t onnxruntime::TensorrtExecutionProvider::GetSupportedList(SubGraphCollection_t, int, int, const onnxruntime::GraphViewer&, bool*) const graph_build.Resolve().IsOK() was false.

I've found some issues describing this error and people were mentioning the possibility that it could come from the fact that TensorRT does not deal with dynamic shapes. Therefore I've exported my model with fixed shapes but it dit not solve the issue.

Moreover when using the polygraphy library it seems like there is no problem

polygraphy run checkpoint_final_torch2.4.1+cu121_onnx1.16.1.onnx --trt

I get the following

Many thanks for your help !

To reproduce

Ubuntu 22.04
Cuda 12.2
Cudnn 9.4.0
TensorRT 10.2.0
onnxruntime-gpu 1.19.2
onxx 1.16.1 (for model export)
torch 2.4.1 (for model export)

Urgency

No response

Platform

Linux

OS Version

Ubuntu 22.04

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.19.0

ONNX Runtime API

Python

Architecture

X64

Execution Provider

TensorRT

Execution Provider Library Version

TensorRT 10.2

[chilo-ms]

Hi, Could you share the model so that we can repro from our side?

[cabinader]

Sorry for the late reply. I can't share the original model for confidentiality reasons. However, I could reproduce the issue on the exact same architecture with a publicly available model. Here's a link to download it: https://we.tl/t-1xrZOLot4p

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[TheLorma200]

I'm having a similar problem, have you found any solution?

[enze5088]

I'm having a similar problem too

[E:onnxruntime:, inference_session.cc:2280 operator()] Exception during initialization: /onnxruntime_src/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc:2263 SubGraphCollection_t onnxruntime::TensorrtExecutionProvider::GetSupportedList(SubGraphCollection_t, int, int, const onnxruntime::GraphViewer&, bool*) const [ONNXRuntimeError] : 1 : FAIL : TensorRT input: /new/encoder/layer.0/attention/Split_output_1 has no shape specified. Please run shape inference on the onnx model first. Details can be found in https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html#shape-inference-for-tensorrt-subgraphs

[chilo-ms]

I'm having a similar problem too

[E:onnxruntime:, inference_session.cc:2280 operator()] Exception during initialization: /onnxruntime_src/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc:2263 SubGraphCollection_t onnxruntime::TensorrtExecutionProvider::GetSupportedList(SubGraphCollection_t, int, int, const onnxruntime::GraphViewer&, bool*) const [ONNXRuntimeError] : 1 : FAIL : TensorRT input: /new/encoder/layer.0/attention/Split_output_1 has no shape specified. Please run shape inference on the onnx model first. Details can be found in https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html#shape-inference-for-tensorrt-subgraphs

Like mentioned in https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html#shape-inference-for-tensorrt-subgraphs, please run symbolic_shape_infer.py on the model first, so that the nodes inside the model have shape info.

symbolic_shape_infer.py --input model.onnx --output model_shaped.onnx --auto_merge