Concat layer gives warnings in TensorRT (only)

[BengtGustafsson]

Describe the issue

When creating ort::Session for most of our models using TensorRT we get this message on the Warning level. When we use CPU,DML or CUDA providers we don't get any error and the models generate the same output as in the source PyTorch. As all other providers work with our models I think this is an issue in Tensor RT or its EP. I tried to grep for the strings but didn't find anything so I'm guessing this happens in TensorRT itself.

onnxruntime: [2024-09-20 10:04:33 WARNING] Profile kMAX values are not self-consistent. /blocks.29/Concat: axis 2 dimensions must be equal for concatenation on axis 1. Condition '==' violated: 44 != 45. Instruction: CHECK_EQUAL 44 45. [tensorrt_execution_provider.h:90 onnxruntime::TensorrtLogger::log]

I think the model then produces correct results but in general we get the wrong result on Tensor RT so it is a bit hard to say. We're eagerly awaiting the result of: #21457 where our problem may or may not be the same as those other users which get discrepancies. We get such problems also with models that don't produce the above warning, so it is unclear what is going on.

To reproduce

Unfortunately I can't deliver a model at this time. Maybe I can get hold of an onnx file with fake coefficients if you really need it to debug this.

Urgency

No response

Platform

Windows

OS Version

Windows 11

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.19.2

ONNX Runtime API

C++

Architecture

X64

Execution Provider

TensorRT

Execution Provider Library Version

TensorRT 10.4.0.26 on CUDA 11.6

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[BengtGustafsson]

This system of making issues stale because you didn't react to them is ridiculous!