[BUG on Onnxruntime+TensorRT] Assertion failed: input_dims.nbDims == 4 || input_dims.nbDims == 5 instanceNormalizationPlugin/instanceNormalizationPlugin.cu:181

[yetingqiaqia]

Describe the bug
Hi, I built and installed Onnxruntime+TensorRT from scratch by following this wiki: https://onnxruntime.ai/docs/build/eps.html#tensorrt

But, when I used it to run a fp16 model with "TensorrtExecutionProvider", it failed with below error msg:

2021-12-07 00:43:42.752827823 [E:onnxruntime:Default, tensorrt_execution_provider.h:51 log] [2021-12-07 00:43:42     BUG] Assertion failed: input_dims.nbDims == 4 || input_dims.nbDims == 5
instanceNormalizationPlugin/instanceNormalizationPlugin.cu:181
Aborting...

Aborted (core dumped)

However, if I run the same model with "CUDAExecutionProvider". It works fine.
So, is it a bug in Onnxruntime+TensorRT? How to solve it?

Urgency
This bug prevents me from using ORT+TRT.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): 18.04
• ONNX Runtime installed from (source or binary): latest source code
• ONNX Runtime version: 1.10.0
• Python version: 3.6
• Visual Studio version (if applicable): N/A
• GCC/Compiler version (if compiling from source): 7.5.0
• CUDA/cuDNN version: CUDA 11.5, CuDNN 8.3.1, TensorRT 8.2.1
• GPU model and memory: V100, 16GB

To Reproduce
To reproduce, use below code and model:
-model: https://drive.google.com/file/d/1lvoLHigqiWaB3eFPoStQ22L5Avh316h8/view?usp=sharing
-code:

import numpy as np
import time

np.random.seed(123)
input = np.random.rand(8, 3, 480, 480).astype(np.float32)

def onnx_evaluate():
    import onnxruntime as ort
    sess_options = ort.SessionOptions()
    sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
    sess = ort.InferenceSession("graph_opset12_optimized_fp16_keep_io_True_unblockInstanceNormalization.onnx", sess_options,
            providers=[
                ("TensorrtExecutionProvider", {
                    'trt_fp16_enable': True,
                    }),
                "CUDAExecutionProvider"
                ]
    )

    input_name = sess.get_inputs()[0].name
    label_name = sess.get_outputs()[0].name

    options = sess.get_provider_options()
    print("Provider: ", sess.get_providers())
    print("Provider options:", options)
    print("Load model succefully!")

    warm_up_start_stamp = time.time()
    pred = sess.run([label_name], {input_name: input})[0]
    print(f"It takes {time.time()-warm_up_start_stamp} to finish warm-up.")
    print(pred)
    print("Test finished successfully!")

onnx_evaluate()

The error msg is:

But, if we change providers to CUDAExecutionProvider only, like below:

    sess = ort.InferenceSession("graph_opset12_optimized_fp16_keep_io_True_unblockInstanceNormalization.onnx", sess_options,
            providers=[
                #("TensorrtExecutionProvider", {
                #    'trt_fp16_enable': True,
                #    }),
                "CUDAExecutionProvider"
                ]
    )

It will finish successfully:

Expected behavior
I expect TensorrtExecutionProvider should at least work.

[stevenlix]

I run your script in ORT1.9+TRT8.0+CUDA11.3 and it finished without error.
However I see NaN in output. Can you share your FP32 model?

[yetingqiaqia]

@stevenlix Here is the FP32 model: https://drive.google.com/file/d/1LIFFxD8O39VITABQuH0H0Zx4kT3Wk-xd/view?usp=sharing
FYI, ORT1.9 has a bug on FP16 of InstanceNorm op, it was fixed in this PR: #9879 and was merged in ORT1.10.
That's why I was using ORT1.10 to test.

[stevenlix]

Thanks. It's ok to use ORT1.10 or ORT master. But please use TRT8.0 with it. TRT8.2 is not officially supported in ORT yet.

[stevenlix]

I run your FP32 model in ORT1.9+TRT8.0+CUDA11.3. The script finished successfully and generated correct output. The NaN issue on the FP16 model is probably because certain attribute constants in the FP16 model is underflowed (too small) in FP16 precision.
You can actually directly run FP32 model in TRT EP. Just enable trt_fp16_enable to run it in FP16 precision. No FP16 model is needed for TRT.
I will try your model in ORT1.10+TRT8.0 to see if there is any issue.

[yetingqiaqia]

Thanks. For this The NaN issue on the FP16 model is probably because certain attribute constants in the FP16 model is underflowed (too small) in FP16 precision. I actually tried ORT1.10+CUDA11.5, there is no issue at all. I am not sure why there is a NaN issue when using TRT8. I feel it is a bug on TRT8.

[stevenlix]

TRT may handle FP32->FP16 conversion differently to CUDA implementation. We see similar behavior before.

[yetingqiaqia]

For downgrading from TRT8.2 to TRT8.0, should I build the ORT code again? Is there easier way to install ORT like pip? It takes me days on installing different CUDA version, TRT version and compiling the ORT code.

[stevenlix]

https://github.com/microsoft/onnxruntime/releases/tag/v1.10.0
"Starting from ORT1.10, Python GPU package now includes both TensorRT and CUDA EPs. Note: EPs need to be explicitly registered to ensure the correct provider is used. e.g. InferenceSession('model.onnx', providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider']). Please also ensure you have appropriate TensorRT dependencies and CUDA dependencies installed."
Here is the python wheel, https://pypi.org/project/onnxruntime-gpu/#files

[yetingqiaqia]

https://github.com/microsoft/onnxruntime/releases/tag/v1.10.0 "Starting from ORT1.10, Python GPU package now includes both TensorRT and CUDA EPs. Note: EPs need to be explicitly registered to ensure the correct provider is used. e.g. InferenceSession('model.onnx', providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider']). Please also ensure you have appropriate TensorRT dependencies and CUDA dependencies installed." Here is the python wheel, https://pypi.org/project/onnxruntime-gpu/#files

This is great!

[stale]

This issue has been automatically marked as stale due to inactivity and will be closed in 7 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[snnn]

This issue has been automatically closed as 'not planned' because it has been marked as 'stale' for more than 30 days without activity. If you believe this is still an issue, please feel free to reopen it.