VEDet: Viewpoint Equivariance for Multi-View 3D Object Detection (CVPR 2023)

This is the official implementation of CVPR 2023 paper Viewpoint Equivariance for Multi-View 3D Object Detection authored by Dian Chen, Jie Li, Vitor Guizilini, Rares Ambrus, and Adrien Gaidon, at Toyota Research Institute. We introduce viewpoint equivariance on view-conditioned object queries achieving state-of-the-art 3D object performance.

• [May 4, 2023] Our code and models are released!
• [Mar. 27, 2023] Our code and models will be released soon. Please stay tuned!

Contents

• Install
• Dataset preparation
• Training
• Inference
• License
• Reference

Install

We provide instructions for using docker environment and pip/conda environment (docker is recommended for portability and reproducibility). Please refer to INSTALL.md for detailed instructions.

Dataset preparation

Please download the full NuScenes dataset from the official website, and preprocess the meta data following the instructions from MMDetection3D to obtain the .pkl files with mmdet3d format. For convenience we provide the preprocessed .pkl files for nuscenes dataset here. Put the .pkl files under the NuScenes folder.

Training

To train a model with the provided configs, please run the following:

# run distributed training with 8 GPUs
# tools/dist_train.sh <config path> 8 --work-dir <save dir> --cfg-options <overrides>

# for example:
tools/dist_train.sh projects/configs/vedet_vovnet_p4_1600x640_2vview_2frame.py 8 --work-dir work_dirs/vedet_vovnet_p4_1600x640_2vview_2frame/

Before running the training with V2-99 backbone, please download the DD3D pre-trained weights from here.

We provide results on the NuScenes val set from the paper, as summarized below.

config	mAP	NDS	resolution	backbone	context	download
vedet_vovnet_p4_1600x640_2vview_2frame	0.451	0.527	1600x640	V2-99	current + 1 past frame	model / log

Inference

To run inference with a checkpoint, please run the following:

# run distributed evaluation with 8 GPUs
# tools/dist_test.sh <config path> <ckpt path> 8 --eval bbox

# for example:
tools/dist_test.sh projects/configs/vedet_vovnet_p4_1600x640_2vview_2frame.py work_dirs/vedet_vovnet_p4_1600x640_2vview_2frame/latest.pth 8 --eval bbox

License

We release this repo under the CC BY-NC 4.0 license.

Reference

If you have any questions, feel free to open an issue under this repo, or contact us at dian.chen@tri.global. If you find this work helpful to your research, please consider citing us:

@article{chen2023viewpoint,
  title={Viewpoint Equivariance for Multi-View 3D Object Detection},
  author={Chen, Dian and Li, Jie and Guizilini, Vitor and Ambrus, Rares and Gaidon, Adrien},
  journal={arXiv preprint arXiv:2303.14548},
  year={2023}
}

We also thank the authors of detr3d and petr/petrv2.