SOGDet: Semantic-Occupancy Guided Multi-view 3D Object Detection

The official implementation of SOGDet.

SOGDet: Semantic-Occupancy Guided Multi-view 3D Object Detection

Qiu Zhou, Jinming Cao, Hanchao Leng, Yifang Yin, Kun Yu, Roger Zimmermann, AAAI 2024.

Introduction

We propose a novel approach called SOGDet (Semantic-Occupancy Guided Multi-view 3D Object Detection), that leverages a 3D semantic-occupancy branch to improve the accuracy of 3D object detection. In particular, the physical context modeled by semantic occupancy helps the detector to perceive the scenes in a more holistic view. Our SOGDet is flexible to use and can be seamlessly integrated with most existing BEV-based methods.

Usage

Main Results

Method	mAP	NDS	Model
SOGDet-BO-r50	38.2	50.2	google
SOGDet-SE-r50*	38.8	50.6	google
SOGDet-BO-r101	43.9	55.4	google
SOGDet-SE-r101*	45.8	56.6	google

• Memory is tested in the training process with batch 1 and without using torch.checkpoint.

Environment Installation

Please see install.md.

Data Preparation

Please see install.md.

Train

For training process, we use config file in $SOGDet/configs/sogdet to define model, dataset and hyber parameters. Run the following command to start a training process. For example:

tools/dist_train.sh configs/sogdet-se-r50.py 8 

Test

For testing bbox scores, run the following command:

tools/dist_test.sh configs/sogdet-se-r50.py sogdet-se-r50.pth 8 --eval bbox  

For testing bbox & mIOU scores, run the following command:

tools/dist_test_plus.sh configs/sogdet-se-r50.py sogdet-se-r50.pth 8 --eval bbox

Visualization

For visualizing results, we should add parameter '--show-dir xxx' to save results during running test commands. For example:

tools/dist_test.sh configs/sogdet-se-r50.py sogdet-se-r50.pth 8 --eval bbox --show-dir work_dirs/sogdet-se-r50/  # for OD results
python tools/test_plus.py configs/sogdet-se-r50.py sogdet-se-r50.pth --eval bbox --show-dir work_dirs/sogdet-se-r50/  # for OCC results (no multi-process)

Run the following command to visualize OD & OCC visualization results.

python tools/vis_pred_gt_od_hybrid.py --bbox-path work_dirs/sogdet-se-r50/ --voxel-path work_dirs/sogdet-se-r50/ --save-path work_dirs/sogdet-se-r50/visual_pred/
python tools/vis_pred_gt_occ.py --voxel-path work_dirs/sogdet-se-r50/ --save-path work_dirs/sogdet-se-r50/

Citation

If you find this repo useful, please consider citing:

@inproceedings{zhou2024sogdet,
  title={SOGDet: Semantic-occupancy guided multi-view 3D object detection},
  author={Zhou, Qiu and Cao, Jinming and Leng, Hanchao and Yin, Yifang and Kun, Yu and Zimmermann, Roger},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={38},
  number={7},
  pages={7668--7676},
  year={2024}
}

Acknowledgement

This project is not possible without multiple great open-sourced code bases. We list some notable examples below.

• open-mmlab
• CenterPoint
• Lift-Splat-Shoot
• Swin Transformer
• BEVDet
• BEVDepth