📖 CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detection (NeurIPS2023)

🔥Please star CoDA ⭐ and share it. Thanks🔥

[Paper]   [Project Page]

Yang Cao, Yihan Zeng, Hang Xu, Dan Xu
The Hong Kong University of Science and Technology
Huawei Noah's Ark Lab

🚩 Updates

☑ Latest papers&codes about open-vocabulary perception are collected here.

☑ All the codes, data and pretrained models have been released!

☑ The training and testing codes have been released.

☑ The pretrained models have been released.

☑ The OV-setting SUN-RGBD datasets have been released.

☑ The OV-setting ScanNet datasets have been released.

☑ Paper LaTeX codes are available at https://scienhub.com/Yang/CoDA.

Framework

Samples

Installation

Our code is based on PyTorch 1.8.1, torchvision==0.9.1, CUDA 10.1 and Python 3.7. It may work with other versions.

Please also install the following Python dependencies:

matplotlib
opencv-python
plyfile
'trimesh>=2.35.39,<2.35.40'
'networkx>=2.2,<2.3'
scipy

Please install pointnet2 layers by running

cd third_party/pointnet2 && python setup.py install

Please install a Cythonized implementation of gIOU for faster training.

conda install cython
cd utils && python cython_compile.py build_ext --inplace

Dataset preparation

To achieve the OV setting, we re-organize the original ScanNet and SUN RGB-D and adopt annotations of more categories. Please directly download the ov-setting datasets we provide here: OV SUN RGB-D and OV ScanNet.

Then run for the downloaded *.tar file:

mv sunrgbd_trainval.tar Data/sunrgb_d/
mv sunrgbd_v1_revised_0415.tar Data/sunrgb_d/
cd Data/sunrgb_d/
tar -xvf sunrgbd_v1_revised_0415.tar
tar -xvf sunrgbd_trainval.tar
cd -
mv scannet200_data.tar.* Data/scannet/
cd Data/scannet/
cat scannet200_data.tar.* | tar -xvf
cd -

Evaluation

Download the pretrained models here. Then run:

bash test_release_models.sh

Training

bash scripts/coda_sunrgbd_stage1.sh
bash scripts/coda_sunrgbd_stage2.sh

📜 BibTeX

If CoDA is helpful, please cite:

@inproceedings{cao2023coda,
  title={CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detection},
  author={Cao, Yang and Zeng, Yihan and Xu, Hang  and  Xu, Dan},
  booktitle={NeurIPS},
  year={2023}
}   

📧 Contact

If you have any question or collaboration need (research purpose or commercial purpose), please email yangcao.cs@gmail.com.

📜 Acknowledgement

CoDA is inspired by CLIP and 3DETR. We appreciate their great codes.