[Bug] The pointPillars model got wrong output when I use TensorRT acceleration

[ykqyzzs]

Checklist

• I have searched related issues but cannot get the expected help.
• 2. I have read the FAQ documentation but cannot get the expected help.
• 3. The bug has not been fixed in the latest version.

Describe the bug

Hi,
I used mmdeploy/tools/test.py to test the converted pointpillars onnx model, it successfully finished but the AP result is abnormal:

----------- AP11 Results ------------

Pedestrian AP11@0.50, 0.50, 0.50:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
Pedestrian AP11@0.50, 0.25, 0.25:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
Cyclist AP11@0.50, 0.50, 0.50:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
Cyclist AP11@0.50, 0.25, 0.25:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
Car AP11@0.70, 0.70, 0.70:
bbox AP11:0.0000, 9.0909, 9.0909
bev AP11:0.0000, 9.0909, 9.0909
3d AP11:0.0000, 9.0909, 9.0909
aos AP11:0.00, 9.09, 9.09
Car AP11@0.70, 0.50, 0.50:
bbox AP11:0.0000, 9.0909, 9.0909
bev AP11:0.0000, 9.0909, 9.0909
3d AP11:0.0000, 9.0909, 9.0909
aos AP11:0.00, 9.09, 9.09

Overall AP11@easy, moderate, hard:
bbox AP11:0.0000, 3.0303, 3.0303
bev AP11:0.0000, 3.0303, 3.0303
3d AP11:0.0000, 3.0303, 3.0303
aos AP11:0.00, 3.03, 3.03

----------- AP40 Results ------------

Pedestrian AP40@0.50, 0.50, 0.50:
bbox AP40:0.0000, 0.0000, 0.0000
bev AP40:0.0000, 0.0000, 0.0000
3d AP40:0.0000, 0.0000, 0.0000
aos AP40:0.00, 0.00, 0.00
Pedestrian AP40@0.50, 0.25, 0.25:
bbox AP40:0.0000, 0.0000, 0.0000
bev AP40:0.0000, 0.0000, 0.0000
3d AP40:0.0000, 0.0000, 0.0000
aos AP40:0.00, 0.00, 0.00
Cyclist AP40@0.50, 0.50, 0.50:
bbox AP40:0.0000, 0.0000, 0.0000
bev AP40:0.0000, 0.0000, 0.0000
3d AP40:0.0000, 0.0000, 0.0000
aos AP40:0.00, 0.00, 0.00
Cyclist AP40@0.50, 0.25, 0.25:
bbox AP40:0.0000, 0.0000, 0.0000
bev AP40:0.0000, 0.0000, 0.0000
3d AP40:0.0000, 0.0000, 0.0000
aos AP40:0.00, 0.00, 0.00
Car AP40@0.70, 0.70, 0.70:
bbox AP40:0.0000, 2.5000, 2.5000
bev AP40:0.0000, 2.5000, 2.5000
3d AP40:0.0000, 2.5000, 2.5000
aos AP40:0.00, 2.50, 2.50
Car AP40@0.70, 0.50, 0.50:
bbox AP40:0.0000, 2.5000, 2.5000
bev AP40:0.0000, 2.5000, 2.5000
3d AP40:0.0000, 2.5000, 2.5000
aos AP40:0.00, 2.50, 2.50

Overall AP40@easy, moderate, hard:
bbox AP40:0.0000, 0.8333, 0.8333
bev AP40:0.0000, 0.8333, 0.8333
3d AP40:0.0000, 0.8333, 0.8333
aos AP40:0.00, 0.83, 0.83

Then I checked the data_loader, the inputs seems correct，and I have checked the onnx model, it seems ok and visualization by netron is almost same as the pointpillars onnx file from this link(which I found from this issue，NVIDIA/TensorRT#2338):
https://drive.google.com/file/d/1FuZJWLIsJyUsUk_lM1euXzyPgagu-tXj/view?usp=sharing
I have tried the both onnx file, converted into .engine file and test, but got the same results.

So I print the outputs then found that, outputs = task_processor.single_gpu_test(model, data_loader, args.show, args.show_dir) returned the empty outputs such as:

{'boxes_3d': LiDARInstance3DBoxes(
tensor([], size=(0, 7))), 'scores_3d': tensor([]), 'labels_3d': tensor([], dtype=torch.int64)}, {'boxes_3d': LiDARInstance3DBoxes(
tensor([], size=(0, 7))), 'scores_3d': tensor([]), 'labels_3d': tensor([], dtype=torch.int64)}, {'boxes_3d': LiDARInstance3DBoxes(
tensor([], size=(0, 7))), 'scores_3d': tensor([]), 'labels_3d': tensor([], dtype=torch.int64)}, {'boxes_3d': LiDARInstance3DBoxes(
tensor([], size=(0, 7))), 'scores_3d': tensor([]), 'labels_3d': tensor([], dtype=torch.int64)}, {'boxes_3d': LiDARInstance3DBoxes(
tensor([], size=(0, 7))), 'scores_3d': tensor([]), 'labels_3d': tensor([], dtype=torch.int64)}, {'boxes_3d': LiDARInstance3DBoxes(
tensor([], size=(0, 7))), 'scores_3d': tensor([]), 'labels_3d': tensor([], dtype=torch.int64)}, ... ...

So, I would like to ask, what may be the cause of the the wrong test results? and how to solve it?
Thanks very much!

Reproduction

I use this command to convert model:
python mmdeploy/tools/deploy.py mmdeploy/configs/mmdet3d/voxel-detection/voxel-detection_tensorrt_dynamic-kitti-32x4.py mmdetection3d/configs/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class.py checkpoints/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class_20220301_150306-37dc2420.pth mmdetection3d/demo/data/kitti/kitti_000008.bin --work-dir work-dir2 --device cuda:0 --show

I use this command to test the converted model:
python ../mmdeploy/tools/test.py ../mmdeploy/configs/mmdet3d/voxel-detection/voxel-detection_tensorrt_dynamic-kitti-32x4.py ./configs/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class.py --model ../mmdeploy/work-dir2/end2end.engine --metrics bbox --device cuda:0

Environment

2022-12-12 18:08:33,002 - mmdeploy - INFO - 

2022-12-12 18:08:33,002 - mmdeploy - INFO - **********Environmental information**********
2022-12-12 18:08:33,416 - mmdeploy - INFO - sys.platform: linux
2022-12-12 18:08:33,416 - mmdeploy - INFO - Python: 3.7.13 (default, Mar 29 2022, 02:18:16) [GCC 7.5.0]
2022-12-12 18:08:33,416 - mmdeploy - INFO - CUDA available: True
2022-12-12 18:08:33,416 - mmdeploy - INFO - GPU 0,1,2,3,4,5,6,7,8,9: NVIDIA GeForce RTX 3090
2022-12-12 18:08:33,416 - mmdeploy - INFO - CUDA_HOME: /usr/local/cuda
2022-12-12 18:08:33,416 - mmdeploy - INFO - NVCC: Cuda compilation tools, release 11.3, V11.3.109
2022-12-12 18:08:33,416 - mmdeploy - INFO - GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
2022-12-12 18:08:33,416 - mmdeploy - INFO - PyTorch: 1.11.0
2022-12-12 18:08:33,416 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.5.2 (Git Hash a9302535553c73243c632ad3c4c80beec3d19a1e)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.3
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
  - CuDNN 8.2
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.11.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 

2022-12-12 18:08:33,416 - mmdeploy - INFO - TorchVision: 0.12.0
2022-12-12 18:08:33,416 - mmdeploy - INFO - OpenCV: 4.5.5
2022-12-12 18:08:33,416 - mmdeploy - INFO - MMCV: 1.5.2
2022-12-12 18:08:33,416 - mmdeploy - INFO - MMCV Compiler: GCC 7.5
2022-12-12 18:08:33,416 - mmdeploy - INFO - MMCV CUDA Compiler: 11.3
2022-12-12 18:08:33,416 - mmdeploy - INFO - MMDeploy: 0.10.0+99040d5
2022-12-12 18:08:33,416 - mmdeploy - INFO - 

2022-12-12 18:08:33,417 - mmdeploy - INFO - **********Backend information**********
2022-12-12 18:08:34,080 - mmdeploy - INFO - onnxruntime: None	ops_is_avaliable : False
2022-12-12 18:08:34,108 - mmdeploy - INFO - tensorrt: 8.5.1.7	ops_is_avaliable : True
2022-12-12 18:08:34,124 - mmdeploy - INFO - ncnn: None	ops_is_avaliable : False
2022-12-12 18:08:34,125 - mmdeploy - INFO - pplnn_is_avaliable: False
2022-12-12 18:08:34,126 - mmdeploy - INFO - openvino_is_avaliable: False
2022-12-12 18:08:34,142 - mmdeploy - INFO - snpe_is_available: False
2022-12-12 18:08:34,143 - mmdeploy - INFO - ascend_is_available: False
2022-12-12 18:08:34,144 - mmdeploy - INFO - coreml_is_available: False
2022-12-12 18:08:34,144 - mmdeploy - INFO - 

2022-12-12 18:08:34,144 - mmdeploy - INFO - **********Codebase information**********
2022-12-12 18:08:34,146 - mmdeploy - INFO - mmdet:	2.24.1
2022-12-12 18:08:34,146 - mmdeploy - INFO - mmseg:	0.24.1
2022-12-12 18:08:34,146 - mmdeploy - INFO - mmcls:	0.23.0
2022-12-12 18:08:34,146 - mmdeploy - INFO - mmocr:	None
2022-12-12 18:08:34,146 - mmdeploy - INFO - mmedit:	None
2022-12-12 18:08:34,146 - mmdeploy - INFO - mmdet3d:	1.0.0rc4
2022-12-12 18:08:34,146 - mmdeploy - INFO - mmpose:	None
2022-12-12 18:08:34,146 - mmdeploy - INFO - mmrotate:	None

Error traceback

Nothing

[ykqyzzs]

Add a little infomation, there are some worning when I run tools/test.py:

2022-12-12 16:14:11,925 - mmdeploy - INFO - Successfully loaded tensorrt plugins from /home/yangkang/anaconda3/envs/ri_fusion/lib/python3.7/site-packages/mmdeploy/lib/libmmdeploy_tensorrt_ops.so
2022-12-12 16:14:11,925 - mmdeploy - INFO - Successfully loaded tensorrt plugins from /home/yangkang/anaconda3/envs/ri_fusion/lib/python3.7/site-packages/mmdeploy/lib/libmmdeploy_tensorrt_ops.so
[12/12/2022-16:14:13] [TRT] [W] TensorRT was linked against cuDNN 8.6.0 but loaded cuDNN 8.2.1
[12/12/2022-16:14:13] [TRT] [W] TensorRT was linked against cuDNN 8.6.0 but loaded cuDNN 8.2.1
[12/12/2022-16:14:14] [TRT] [W] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See CUDA_MODULE_LOADING in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
[ ] 0/3769, elapsed: 0s, ETA:/home/yangkang/ri_fusion/rifusion_test/mmdet3d/models/dense_heads/anchor3d_head.py:85: UserWarning: dir_offset and dir_limit_offset will be depressed and be incorporated into box coder in the future
'dir_offset and dir_limit_offset will be depressed and be '
2022-12-12 16:14:19,713 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied
/home/yangkang/anaconda3/envs/ri_fusion/lib/python3.7/site-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1646755953518/work/aten/src/ATen/native/TensorShape.cpp:2228.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]

[ ] 1/3769, 0.5 task/s, elapsed: 2s, ETA: 7918s2022-12-12 16:14:20,237 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied

[ ] 2/3769, 0.8 task/s, elapsed: 3s, ETA: 4826s2022-12-12 16:14:20,700 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied
... ...
... ...
... ...
... ...
[>>>>>>>>>>>>>>>>>>>>>>>>> ] 3767/3769, 6276.0 task/s, elapsed: 1s, ETA: 0s
[>>>>>>>>>>>>>>>>>>>>>>>>> ] 3768/3769, 6276.4 task/s, elapsed: 1s, ETA: 0s
[>>>>>>>>>>>>>>>>>>>>>>>>>>] 3769/3769, 6276.8 task/s, elapsed: 1s, ETA: 0s
Result is saved to /tmp/tmpaslh9sjb/results.pkl.
OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.

----------- AP11 Results ------------

Pedestrian AP11@0.50, 0.50, 0.50:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
Pedestrian AP11@0.50, 0.25, 0.25:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
Cyclist AP11@0.50, 0.50, 0.50:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
Cyclist AP11@0.50, 0.25, 0.25:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
Car AP11@0.70, 0.70, 0.70:
bbox AP11:0.0000, 9.0909, 9.0909
bev AP11:0.0000, 9.0909, 9.0909
3d AP11:0.0000, 9.0909, 9.0909
aos AP11:0.00, 9.09, 9.09
Car AP11@0.70, 0.50, 0.50:
bbox AP11:0.0000, 9.0909, 9.0909
bev AP11:0.0000, 9.0909, 9.0909
3d AP11:0.0000, 9.0909, 9.0909
aos AP11:0.00, 9.09, 9.09

Overall AP11@easy, moderate, hard:
bbox AP11:0.0000, 3.0303, 3.0303
bev AP11:0.0000, 3.0303, 3.0303
3d AP11:0.0000, 3.0303, 3.0303
aos AP11:0.00, 3.03, 3.03

----------- AP40 Results ------------

Pedestrian AP40@0.50, 0.50, 0.50:
bbox AP40:0.0000, 0.0000, 0.0000
bev AP40:0.0000, 0.0000, 0.0000
3d AP40:0.0000, 0.0000, 0.0000
aos AP40:0.00, 0.00, 0.00
Pedestrian AP40@0.50, 0.25, 0.25:
bbox AP40:0.0000, 0.0000, 0.0000
bev AP40:0.0000, 0.0000, 0.0000
3d AP40:0.0000, 0.0000, 0.0000
aos AP40:0.00, 0.00, 0.00
Cyclist AP40@0.50, 0.50, 0.50:
bbox AP40:0.0000, 0.0000, 0.0000
bev AP40:0.0000, 0.0000, 0.0000
3d AP40:0.0000, 0.0000, 0.0000
aos AP40:0.00, 0.00, 0.00
Cyclist AP40@0.50, 0.25, 0.25:
bbox AP40:0.0000, 0.0000, 0.0000
bev AP40:0.0000, 0.0000, 0.0000
3d AP40:0.0000, 0.0000, 0.0000
aos AP40:0.00, 0.00, 0.00
Car AP40@0.70, 0.70, 0.70:
bbox AP40:0.0000, 2.5000, 2.5000
bev AP40:0.0000, 2.5000, 2.5000
3d AP40:0.0000, 2.5000, 2.5000
aos AP40:0.00, 2.50, 2.50
Car AP40@0.70, 0.50, 0.50:
bbox AP40:0.0000, 2.5000, 2.5000
bev AP40:0.0000, 2.5000, 2.5000
3d AP40:0.0000, 2.5000, 2.5000
aos AP40:0.00, 2.50, 2.50

Overall AP40@easy, moderate, hard:
bbox AP40:0.0000, 0.8333, 0.8333
bev AP40:0.0000, 0.8333, 0.8333
3d AP40:0.0000, 0.8333, 0.8333
aos AP40:0.00, 0.83, 0.83

[tpoisonooo]

Let me try to reproduce it with TRT8.5 later.

[tpoisonooo]

And I have tested pointpillars+centerpoint on cu102+TRT8.4 and passed.

[ykqyzzs]

Thanks for your reply,
I have tried cuda11.3+TRT8.4.3.1 @yestoday, got the same problem T_T. I used the kitti dataset to test, and then I looked at the outputs carefully, found not all the outputs were null, Only one frame of outputs got reasonable results:

I guess it's my envs or commands have something wrong... Maybe some important warning caused my situation? Would you please help me to check my command or running logs to solve the problem? @tpoisonooo
Thank you very much!

[ykqyzzs]

TRT8.4.3.1+CUDA11.3, running log is:

2022-12-17 10:19:01,704 - mmdeploy - INFO - Successfully loaded tensorrt plugins from /home/yangkang/anaconda3/envs/ri_fusion/lib/python3.7/site-packages/mmdeploy/lib/libmmdeploy_tensorrt_ops.so
2022-12-17 10:19:01,704 - mmdeploy - INFO - Successfully loaded tensorrt plugins from /home/yangkang/anaconda3/envs/ri_fusion/lib/python3.7/site-packages/mmdeploy/lib/libmmdeploy_tensorrt_ops.so
[ ] 0/3769, elapsed: 0s, ETA:/home/yangkang/ri_fusion/rifusion_test/mmdet3d/models/dense_heads/anchor3d_head.py:85: UserWarning: dir_offset and dir_limit_offset will be depressed and be incorporated into box coder in the future
'dir_offset and dir_limit_offset will be depressed and be '
2022-12-17 10:19:09,987 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied
/home/yangkang/anaconda3/envs/ri_fusion/lib/python3.7/site-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1646755953518/work/aten/src/ATen/native/TensorShape.cpp:2228.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]

[ ] 1/3769, 0.3 task/s, elapsed: 4s, ETA: 14365s2022-12-17 10:19:10,382 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied

[ ] 2/3769, 0.5 task/s, elapsed: 4s, ETA: 7894s2022-12-17 10:19:10,766 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied

[ ] 3/3769, 0.7 task/s, elapsed: 5s, ETA: 5740s2022-12-17 10:19:11,143 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied

[ ] 4/3769, 0.8 task/s, elapsed: 5s, ETA: 4658s2022-12-17 10:19:11,521 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied
... ...
... ...
... ...
Converting prediction to KITTI format
[ ] 0/3769, elapsed: 0s, ETA:
[ ] 1/3769, 3231.4 task/s, elapsed: 0s, ETA: 1s
[ ] 2/3769, 4082.0 task/s, elapsed: 0s, ETA: 1s
[ ] 3/3769, 4782.6 task/s, elapsed: 0s, ETA: 1s
[ ] 4/3769, 5304.2 task/s, elapsed: 0s, ETA: 1s
[ ] 5/3769, 5714.3 task/s, elapsed: 0s, ETA: 1s
... ...
... ...
... ...
[>>>>>>>>>>>>>>>>>>>>>>>>> ] 3767/3769, 7688.5 task/s, elapsed: 0s, ETA: 0s
[>>>>>>>>>>>>>>>>>>>>>>>>> ] 3768/3769, 7688.9 task/s, elapsed: 0s, ETA: 0s
[>>>>>>>>>>>>>>>>>>>>>>>>>>] 3769/3769, 7689.2 task/s, elapsed: 0s, ETA: 0s
Result is saved to /tmp/tmp_j1lcqev/results.pkl.
OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.

----------- AP11 Results ------------

Pedestrian AP11@0.50, 0.50, 0.50:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
... ...
... ...
... ...

[ykqyzzs]

I tested again on another server with CUDA10.2 + TensorRT8.4.3.1. The problem was gone, I got the correct results. thanks alot.
but I still want to know, what's the wrong reason on my envs in this issue...

[tpoisonooo]

I have no TRT source code, just some assumptions.

1. Open your .onnx file, you can get a Scatter or ScatterND operator
2. TRT has some bug when producing this operator with dynamic input shape

[kunnalparihar]

I tested again on another server with CUDA10.2 + TensorRT8.4.3.1. The problem was gone, I got the correct results. thanks alot. but I still want to know, what's the wrong reason on my envs in this issue...

Hi @ykqyzzs , I need your help on this.
What was Torch version you used?
Have you used docker ?

[ykqyzzs]

I tested again on another server with CUDA10.2 + TensorRT8.4.3.1. The problem was gone, I got the correct results. thanks alot. but I still want to know, what's the wrong reason on my envs in this issue...

Hi @ykqyzzs , I need your help on this. What was Torch version you used? Have you used docker ?

the pytorch version I used is 1.10.1;
without using docker, I deployed it directly in a new virtual environment, following the steps of the official document.
wish this information above can help you.

[WYYAHYT]

I tested again on another server with CUDA10.2 + TensorRT8.4.3.1. The problem was gone, I got the correct results. thanks alot. but I still want to know, what's the wrong reason on my envs in this issue...

I met the same error with CUDA11.4+TensorRT8.4.0.11 , and have you found the reason?

[ykqyzzs]

I tested again on another server with CUDA10.2 + TensorRT8.4.3.1. The problem was gone, I got the correct results. thanks alot. but I still want to know, what's the wrong reason on my envs in this issue...

I met the same error with CUDA11.4+TensorRT8.4.0.11 , and have you found the reason?

sorry I have no idea. I have cross-verified some cuda and trt versions, cuda11.3+trt8.4 also failed, cuda102+trt 8.4.3.1 passed. maybe you can have a try.