dynamic shape model ValueError "could not broadcast input array from shape (224,398,3) into shape (1,1,3)"

[ghylander]

Description

I have a TF2 model with undefined input shape, and I have exported it with tf2onnx with the corresponding cli argument.
The onnx -> TRT process exits with no error code. However when I try to perform an inference I get the tittle error.

in the title error, 224, 398 is the shape of the input test image

Environment

TensorRT Version: 7.1.3
NVIDIA GPU: jetson nano
NVIDIA Driver Version: jetpack 4.5.1
CUDA Version: 10.2
CUDNN Version: 8.0
Operating System: ubuntu 18.04
Python Version (if applicable): 3.8.5
Tensorflow Version (if applicable): 2.5

Relevant Files

Here the model graph:

Here the TF2 model summary:

This is the command I use to export TF2 -> onnx:
python -m tf2onnx.convert --saved-model saved_model --inputs input_2:0[-1,-1,-1,3] --output test_no_shape.onnx

This is the command I use to export onnx -> TRT:
/usr/src/tensorrt/bin/trtexec --onnx=test_no_shape.onnx --fp16 --workspace=1024 --warmUp=2 --dumpProfile --verbose --explicitBatch --saveEngine=test_no_shape.engine

[ttyio]

@ghylander , could you add --minShapes --optShapes --maxShapes --shapes in the trtexec command, else the image is 1x1 by default, and seems your model not support image input size 1x1.
Thanks!

[ghylander]

Thanks @ttyio, correct me if i'm wrong about the usage of the flags:
--minShapes --optShapes --maxShapes are used when exporting onnx->TRT, this implicitly creates an optimization profile
--shapes is to be used when running inference using the TRT engine

if I understood the documentation correctly, I can create a single TRT runtime, and run multiple input shapes on the same runtime, as long as all the shapes are supported by the optimization profile

For example, if I'm feeding my network images of sizes 128x128 and 198x198, I can setup a single runtime, and when I run inference I just add the flag --shapes 128:128 and --shapes 198:198 accordingly

Are there any guidelines on how to specify min, max and optimal shapes? For the time being, I am only expecting 2 specific image shapes to enter the network

[ttyio]

Hello @ghylander , yes you are right, and the option looks like this: --shapes input:1x3x128x128,

more documents is in
https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#opt_profiles

for the guidelines, there is one in
https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#faq

Q: How do I create an engine that is optimized for several different batch sizes?
A: While TensorRT allows an engine optimized for a given batch size to run at any smaller size, the performance for those smaller sizes can not be as well-optimized. To optimize for multiple different batch sizes, create optimization profiles at the dimensions that are assigned to OptProfilerSelector::kOPT.

[ghylander]

what would the syntax of --shapes be when used in a python script?
supposing I have a TRT engine file with 2 optimization profiles, how would I run inference?

If I understood correctly, I would:
First, create the first execution context (context = trtEngine.create_execution_context())
Then, set optimization profile for that context (context = context.set_optimizatio_profile_async(0, stream))

Repeat for the other optimization profile:
First, create the first execution context (context2 = trtEngine.create_execution_context())
Then, set optimization profile for that context (context2 = context2.set_optimizatio_profile_async(1, stream2))

How would I then define the shapes and indexes for inference for each context?
So far, working without dynamic shapes, I've used:

input_shape = trtEngine.get_binding_shape(0)
output_shape = trtEngine.get_binding_shape(1)
context.set_binding_shape(0, input_shape)

[ttyio]

Hello @ghylander , here is one example of selecting profile and bind dimension to this profile, thanks

https://github.com/NVIDIA/TensorRT/blob/main/demo/BERT/inference.py#L132

[ghylander]

ok, thanks, I think I got it

[ghylander]

@ttyio Hi, thanks for the help.
I'm very sorry to bother you again, but how would I define multiple (>1) optimization profiles when exporting onnx - > TRT engine from the CLI?
By using --minShapes --optShapes --maxShapes I create a single optimization profile

[ttyio]

Hi @ghylander ,
multiple profiles are currently not supported in trtexec,
https://github.com/NVIDIA/TensorRT/blob/main/samples/common/sampleInference.cpp#L217

you have to write your own script. code for reference:
https://github.com/NVIDIA/TensorRT/blob/main/demo/BERT/builder.py#L352

[ghylander]

Thanks a lot for the help, this kind of information is quite hard to find

[ttyio]

@ghylander sorry for the inconvenience, now we are keeping improve the tutorial in https://github.com/NVIDIA/TensorRT/tree/main/quickstart, hopefully we will make these information centralized.

Thanks