Fcos3D visualization error in the latest version

[Etah0409]

I try to visualize Fcos3D (mini dataset). After I git clone the latest version, and fix some bugs.

/mmdet3d/datasets/nuscenes+mono_dataset.py
def show():
...
img, img_metas = self._extract_data(i, pipeline,
['img', 'img_metas'])
# need to transpose channel to first dim
img = img.numpy().transpose(1, 2, 0)
...
the origin version will return NoneType to img_metas, and img is (900, 1600, 3) will be trans to (1600, 3, 900), which would cause a wrong mmcv.imwrite.

After I changed these 2 bugs to get the right img and img_metas, the visualization is like following:

It seems that the size of bbox is wrong.
blue one is the pred-imgs, and red one is the gt. This result tests on Nuscenes mini dataset with fcos3d_finetune_model.
my cmd is :
python ./tools/test.py ./configs/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d.py ./work_dirs/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune_20210427_091419-35aaaad0.pth --eval mAP --eval-options show=True out_dir=./work_dirs/fcos3d_vis_mini

Environment
sys.platform: linux
Python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:18) [GCC 10.3.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA RTX A6000
CUDA_HOME: /usr/local/cuda-11.1/
NVCC: Cuda compilation tools, release 11.1, V11.1.74
GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
PyTorch: 1.8.1
PyTorch compiling details: PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 11.1
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
• CuDNN 8.0.5
• Magma 2.5.2
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.2.2
OpenCV: 4.5.5
MMCV: 1.5.1
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 11.1
MMDetection: 2.24.1
MMSegmentation: 0.24.1
MMDetection3D: 1.0.0rc2+76e351a
spconv2.0: False

By the way, I tried to fixed the bugs as #1479(only fixed #1479 -> the 3 changed files), however I got another stranger visualization.

Looking forward to your kindly reply.

[YinengXiong]

Same issue, visualization results are correct on KITTI dataset, and results on nuScenes are questionable, and the problem is not related to the NMS bug

[Etah0409]

Same issue, visualization results are correct on KITTI dataset, and results on nuScenes are questionable, and the problem is not related to the NMS bug

After I dbug for a looong time, I guess I've known the reason.
Nuscenes dataset is different to some others, that its size is [w, l, h], but others are [w, h, l] (or vice versa). In the origin version, when we create dataset, Nuscenes will be different, so Fcos3D show it after swap its dims. But NOW when we create dataset, mmdetection3d has unified now, that's why Fcos3d shows the image with wrong size.
By the way, the show function in nuscenes-mono-datset seems wrong, so when we use the corrected dataset version to train, we'd better use the show function in single-GPU-test:
python ./tools/test.py config pth-file --eval bbox --show --show-dir ./work_dir

Then it won't call the trans in mono-dataset, and output the origin result(however, if u just test the pretrained model, u still need to swap the size by yourself).

[Tai-Wang]

Thanks for your report. @ZCMax Please reproduce this problem and fix it if necessary.