failure of TensorRT 8.6 when running ch_SVTRv2_rec on GPU A100

[zhink]

Description

Environment

TensorRT Version:8.6

NVIDIA GPU: A100

NVIDIA Driver Version: 525.60

CUDA Version:12.0

CUDNN Version:8.8

Operating System:

Python Version (if applicable):

Tensorflow Version (if applicable):

PyTorch Version (if applicable):

Baremetal or Container (if so, version):

Relevant Files

Model link:

Error Log is

E0320 01:58:33.225591 30356 helper.h:131] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin (No Myelin Error exists)
E0320 02:01:23.348383 30356 helper.h:131] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin (No Myelin Error exists)

Connecting to https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/general_ocr_rec_001.png ...
Downloading general_ocr_rec_001.png ...

[== ] 5.56%
[==================================================] 100.00%
E0320 02:04:13.403535 30356 helper.h:131] 7: [shapeMachine.cpp::executeContinuation::864] Error Code 7: Internal Error ((Unnamed Layer* 22) [Slice]: ISliceLayer has out of bounds access on axis 0 Condition '<' violated: 581 >= 512. Instruction: CHECK_LESS 581 512.)
E0320 02:04:13.425848 30356 helper.h:131] 7: [shapeMachine.cpp::executeContinuation::864] Error Code 7: Internal Error ((Unnamed Layer* 22) [Slice]: ISliceLayer has out of bounds access on axis 0 Condition '<' violated: 581 >= 512. Instruction: CHECK_LESS 581 512.)
E0320 02:04:13.451680 30356 helper.h:131] 7: [shapeMachine.cpp::executeContinuation::864] Error Code 7: Internal Error ((Unnamed Layer* 22) [Slice]: ISliceLayer has out of bounds access on axis 0 Condition '<' violated: 581 >= 512. Instruction: CHECK_LESS 581 512.)
Traceback (most recent call last):
File "/paddle/newtrt/fp16/PaddleX/paddlex/utils/result_saver.py", line 29, in wrap
result = func(self, *args, **kwargs)
File "/paddle/newtrt/fp16/PaddleX/paddlex/engine.py", line 50, in run
for res in self._model.predict():
File "/paddle/newtrt/fp16/PaddleX/paddlex/model.py", line 117, in predict
yield from predictor(**predict_kwargs)
File "/paddle/newtrt/fp16/PaddleX/paddlex/inference/models/base/predictor/basic_predictor.py", line 134, in call
yield from self.apply(input, **kwargs)
File "/paddle/newtrt/fp16/PaddleX/paddlex/inference/models/base/predictor/base_predictor.py", line 150, in apply
prediction = self.process(batch_data, **kwargs)
File "/paddle/newtrt/fp16/PaddleX/paddlex/inference/models/text_recognition/predictor.py", line 73, in process
batch_preds = self.infer(x=x)
File "/paddle/newtrt/fp16/PaddleX/paddlex/inference/models/common/static_infer.py", line 335, in call
pred = self.infer(x)
File "/paddle/newtrt/fp16/PaddleX/paddlex/inference/models/common/static_infer.py", line 291, in call
self.predictor.run()
ValueError: (InvalidArgument) The input tensor's dimension should be equal to or greater than the axis's size. But received input tensor's dimension is 0, axis's size is 3
[Hint: Expected x_rank >= axis_size, but received x_rank:0 < axis_size:3.] (at /paddle/newtrt/Paddle/paddle/phi/infermeta/unary.cc:5305)
[operator < transpose2 > error]
End

Q

The model does not explicitly add the ISliceLayer layer. Why are the ISliceLayer-related logs displayed?

Steps To Reproduce

Commands or scripts:

Have you tried the latest release?:

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):

[lix19937]

Can you upload the onnx file ?

[yuanyao-nv]

The ONNX model might not contain Slice nodes but the ONNX parser can still add Slice layer when creating network definition, see https://github.com/onnx/onnx-tensorrt/blob/6ba67d3428e05f690145373ca87fb8d32f98df45/builtin_op_importers.cpp

[poweiw]

Closing due to inactive. Please feel free to reopen!