SiCP: Simultaneous Individual and Cooperative Perception for 3D Object Detection in Connected and Automated Vehicles

SiCP Architecture

Complementary Feature Fusion

Features

• Support SOTA detector

  • PointPillars
  • SECOND
  • Pixor
  • VoxelNet

• Support SOTA cooperative perception models

  • F-Cooper [SEC2019]
  • Attentive Fusion [ICRA2022]
  • V2X-ViT [ECCV2022]
  • Where2comm [NeurIPS2022]
  • CoBEVT [CoRL2022]

• Support dataset

  • OPV2V [ICRA2022]
  • V2V4Real [CVPR2023] (Will relase the code soon)

Dataset Preparation

• Download the OPV2V and V2V4Real datasets.
• After downloading the dataset, place the data into the following structure.

├── opv2v_data_dumping
│   ├── train
│   │   │── 2021_08_22_22_30_58
│   ├── validate
│   ├── test

Installation

1. download SiCP github to your local folder

git clone https://github.com/DarrenQu/SiCP.git
cd SiCP

2. create a conda environment (python >= 3.7)

conda create -n sicp python=3.7
conda activate sicp

3. Pytorch Installation (>= 1.12.0 Required)

conda install pytorch==1.12.0 torchvision==0.13.0 cudatoolkit=11.3 -c pytorch -c conda-forge

4. spconv 2.x Installation (if you are using CUDA 11.3)

pip install spconv-cu113

5. Install other dependencies

pip install -r requirements.txt
python setup.py develop

6. Install bbx nms calculation cuda version

python opencood/utils/setup.py build_ext --inplace

Train the model

To train the model, run the following command.

python opencood/tools/train.py --hypes_yaml ${CONFIG_FILE} [--model_dir  ${CHECKPOINT_FOLDER}]

• hypes_yaml: the path of configuration file, e.g. opencood/hypes_yaml/point_pillar_sicp.yaml.
• model_dir(optional): the path of checkpoint.
• More explaination refer to this repo.

Test the model

First, ensure that the validation_dir parameter in the config.yaml file, located in your checkpoint folder, is set to the path of the testing dataset, for example, opv2v_data_dumping/test.

python opencood/tools/inference.py --model_dir ${CHECKPOINT_FOLDER} --fusion_method ${FUSION_STRATEGY} [--show_vis] [--show_sequence]

• model_dir: the path of saved model.
• fusion_method: about the fusion strategy, 'early', 'late', and 'intermediate'.
• show_vis: whether to visualize the detection overlay with point cloud.
• show_sequence: visualize in a video stream.

Acknowledgement

This project is impossible without these excellent codebases OpenCOOD, CoAlign and V2V4Real.

Citation

@article{qu2023sicp,
  title={SiCP: Simultaneous Individual and Cooperative Perception for 3D Object Detection in Connected and Automated Vehicles},
  author={Qu, Deyuan and Chen, Qi and Bai, Tianyu and Qin, Andy and Lu, Hongsheng and Fan, Heng and Fu, Song and Yang, Qing},
  journal={arXiv preprint arXiv:2312.04822},
  year={2023}
}