UniMODE & MM-UniMODE

UniMODE: Unified Monocular 3D Object Detection

Towards Unified 3D Object Detection via Algorithm and Data Unification

Zhuoling Li, Xiaogang Xu, Ser-Nam Lim, Hengshuang Zhao

[Project Page] [UniMODE Paper] [MM-UniMODE Paper]

This is the official implementation of the paper "UniMODE: Unified Monocular 3D Object Detection" (published in CVPR2024) and "Towards Unified 3D Object Detection via Algorithm and Data Unification".

Contributions:

• We propose UniMODE, a monocular 3D object detector unifying diverse indoor and outdoor scenarios.

• We propose MM-UniMODE, a multi-modal 3D object detector unifying diverse indoor and outdoor scenarios.

• We release the first large-scale multi-modal 3D object detection dataset, MM-Omni3D.

Table of Contents:

1. Installation
2. MM-Omni3D Data
3. Training
4. Evaluation
5. License
6. Citing

Installation

We provide the script file install_env.sh to install all the dependencies. You can use the following command or run each command line in the script file step by step (recommended).

bash install_env.sh

MM-Omni3D Data

As the MM-Omni3D dataset is generated from the Omni3D dataset and is very large (hundreds of GB of data), we provide the code of generating data instead of data files. Refer to Data Preparation for more details. The generated MM-Omni3D dataset folder should look like:

datasets/

├── ARKitScenes/

├── KITTI_object/

├── MM-Omni3D/

├── SUNRGBD/

├── hypersim/

├── nuScenes/

└── objection/

Training

For training UniMODE, you can follow the script command template as follows:

python tools/train_net.py \
  --config-file configs/UniMODE.yaml \
  --num-gpus 16 \
  --num-machines 1 \
  --machine-rank 0 \
  --dist-url tcp://127.0.0.1:12345 \
  OUTPUT_DIR output/UniMODE

For training MM-UniMODE, you can follow the script command template as follows:

python tools/train_net.py \
  --config-file configs/MM_UniMODE.yaml \
  --num-gpus 16 \
  --num-machines 1 \
  --machine-rank 0 \
  --dist-url tcp://127.0.0.1:12345 \
  OUTPUT_DIR output/MM_UniMODE

Evaluation

For evaluating UniMODE, you can follow the script command template as follows:

python tools/train_net.py \
  --eval-only \
  --config-file configs/UniMODE.yaml \
  OUTPUT_DIR output/UniMODE \
  MODEL.WEIGHTS output/UniMODE/model_recent.pth

For evaluating MM-UniMODE, you can follow the script command template as follows:

python tools/train_net.py \
  --eval-only \
  --config-file configs/MM_UniMODE.yaml \
  OUTPUT_DIR output/UniMODE \
  MODEL.WEIGHTS output/MM_UniMODE/model_recent.pth

License

This project is released under CC-BY-NC 4.0.

Citing

Please use the following BibTeX entry if you use UniMODE, MM-UniMODE, or MM-Omni3D in your research or refer to our results.

@inproceedings{li2024unimode,
  title={UniMODE: Unified Monocular 3D Object Detection},
  author={Li, Zhuoling and Xu, Xiaogang and Lim, SerNam and Zhao, Hengshuang},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={16561--16570},
  year={2024}
}

@article{li2024towards,
  title={Towards Unified 3D Object Detection via Algorithm and Data Unification},
  author={Li, Zhuoling and Xu, Xiaogang and Lim, SerNam and Zhao, Hengshuang},
  journal={arXiv:2402.18573},
  year={2024}
}