Failure due to batch size of TensorRT 8.6.1 when running inference on NVIDIA RTX A2000 8GB Laptop GPU

[nullkatar]

Description

I am trying to convert sligtly modified version of YOSO from pytorch to TRT. I cannot make it work with batch size 8.

Can you please point me to exact step in my description which I did incorrectly? Thanks a lot in advance.

Environment

TensorRT Version: TensorRT 8.6.1

NVIDIA GPU: NVIDIA RTX A2000 8GB Laptop GPU

NVIDIA Driver Version: 535.154.05

CUDA Version: 12.2

CUDNN Version:

Operating System:

Python Version (if applicable): 3.10.12

Tensorflow Version (if applicable): N/A

PyTorch Version (if applicable): 2.2.0

Baremetal or Container (if so, version): N/A

Steps To Reproduce

1. Get pytorch weights of modified model

2. Transform them to onnx using deform-conv2d-onnx-exporter and export_model.py as it is said in detectron2. I want to clarify that batch is not dynamic, it is strictly 8.
   2.1. As a result of running this scrpt on image of size (3x512x910) I get the following ONNX file
   2.2. If I run the same script on batch of images of size (8x3x512x910) I get the other ONNX

3. After it I an trying to transform above mentioned onnx files to .engine files using the following command /usr/src/tensorrt/bin/trtexec --onnx=./bs_XXX.onnx --workspace=2048 --explicitBatch --saveEngine=./bs_XXX.engine --fp16

4. After I am trying to inference them using this script It works for batch_size=1, but not for batch_size=8.

I tried converting .onnx file of model with batch_size=1 and after using /usr/src/tensorrt/bin/trtexec --onnx=./bs_XXX.onnx --workspace=2048 command with --batch flag or --maxBatch, but trt fails.

Have you tried the latest release?: Yes

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): Yes

[zerollzeng]

I guess your onnx model only support batch size it. what is the model input shape? what error do you see?

[ttyio]

closing since no activity for more than 3 weeks, pls reopen if you still have question, thanks!

[cookie-yu]

Hello,

I am a new graduate who has recently joined the company. I'm currently working on exporting the YOSO model to the ONNX format for deployment. Would it be convenient for you to provide the complete export code? I would be extremely grateful if you could assist me with this.

My email address is: 1138125979@qq.com

Thank you very much for your time and consideration.

Best regards.