CRN: Camera Radar Net for Accurate, Robust, Efficient 3D Perception

CRN_demo.mp4

CRN: Camera Radar Net for Accurate, Robust, Efficient 3D Perception
Youngseok Kim, Juyeb Shin, Sanmin Kim, In-Jae Lee, Jun Won Choi, Dongsuk Kum
ICCV 2023

Abstract

In this paper, we propose Camera Radar Net (CRN), a novel camera-radar fusion framework that generates a semantically rich and spatially accurate bird's-eye-view (BEV) feature map for various tasks. To overcome the lack of spatial information in an image, we transform perspective view image features to BEV with the help of sparse but accurate radar points. We further aggregate image and radar feature maps in BEV using multi-modal deformable attention designed to tackle the spatial misalignment between inputs. CRN with real-time setting operates at 20 FPS while achieving comparable performance to LiDAR detectors on nuScenes, and even outperforms at a far distance on 100m setting. Moreover, CRN with offline setting yields 62.4% NDS, 57.5% mAP on nuScenes test set and ranks first among all camera and camera-radar 3D object detectors.

Getting Started

Installation

# clone repo
git clone https://github.com/youngskkim/CRN.git

cd CRN

# setup conda environment
conda env create --file CRN.yaml
conda activate CRN

# install dependencies
pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 -f https://download.pytorch.org/whl/torch_stable.html
pip install pytorch-lightning==1.6.0
mim install mmcv==1.6.0
mim install mmsegmentation==0.28.0
mim install mmdet==2.25.2

cd mmdetection3d
pip install -v -e .
cd ..

python setup.py develop  # GPU required

Data preparation

Step 0. Download nuScenes dataset.

Step 1. Symlink the dataset folder to ./data/nuScenes/.

ln -s [nuscenes root] ./data/nuScenes/

Step 2. Create annotation file. This will generate nuscenes_infos_{train,val}.pkl.

python scripts/gen_info.py

Step 3. Generate ground truth depth.
Note: this process requires LiDAR keyframes.

python scripts/gen_depth_gt.py

Step 4. Generate radar point cloud in perspective view. You can download pre-generated radar point cloud here.
Note: this process requires radar blobs (in addition to keyframe) to utilize sweeps.

python scripts/gen_radar_bev.py  # accumulate sweeps and transform to LiDAR coords
python scripts/gen_radar_pv.py  # transform to camera coords

The folder structure will be as follows:

CRN
├── data
│   ├── nuScenes
│   │   ├── nuscenes_infos_train.pkl
│   │   ├── nuscenes_infos_val.pkl
│   │   ├── maps
│   │   ├── samples
│   │   ├── sweeps
|   |   ├── depth_gt
|   |   ├── radar_bev_filter  # temporary folder, safe to delete
|   |   ├── radar_pv_filter
|   |   ├── v1.0-trainval

Training and Evaluation

Training

python [EXP_PATH] --amp_backend native -b 4 --gpus 4

Evaluation
Note: use -b 1 --gpus 1 to measure inference time.

python [EXP_PATH] --ckpt_path [CKPT_PATH] -e -b 4 --gpus 4

Model Zoo

All models use 4 keyframes and are trained without CBGS.
All latency numbers are measured with batch size 1, GPU warm-up, and FP16 precision.

Method	Backbone	NDS	mAP	FPS	Params	Config	Checkpoint
BEVDepth	R50	47.1	36.7	29.7	77.6 M	config	model
CRN	R18	54.2	44.9	29.4	37.2 M	config	model
CRN	R50	56.2	47.3	22.7	61.4 M	config	model

Features

• BEV segmentation checkpoints
• BEV segmentation code
• 3D detection checkpoints
• 3D detection code
• Code release

Acknowledgement

This project is based on excellent open source projects:

• BEVDepth
• BEVFormer
• MMDetection3D

Citation

If this work is helpful for your research, please consider citing the following BibTeX entry.

@inproceedings{kim2023crn,
    title={Crn: Camera radar net for accurate, robust, efficient 3d perception},
    author={Kim, Youngseok and Shin, Juyeb and Kim, Sanmin and Lee, In-Jae and Choi, Jun Won and Kum, Dongsuk},
    booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
    pages={17615--17626},
    year={2023}
}