RRPN: Radar Region Proposal Network for Sensor Fusion in Autonomous Vehicles

Introduction

RRPN is a Region Proposal Network (RPN) exploiting Radar detections to propose Regions of Interest (RoI) for object detection in autonomous vehicles. RRPN provides real-time RoIs for any two-stage object detection network while achieving precision and recall values higher than or on par with vision based RPNs. We evaluate RRPN using the Fast R-CNN network on the NuScenes dataset and compare the results with the Selective Search algorithm.

Contents

1. Requirements
2. Installation
3. Demo
4. Usage

---

Requirements

The followig requirements need to be installed before installing RRPN. Using a virtual environment is optional but highly recommended.

• Python3.7

  RRPN requires Python >= 3.7.0 to work. You can download Python here.

• Virtual Environment

  1. Install virtualenv and virtualenvwrapper

  2. Create a virtual environment for Python 3.7

     mkvirtualenv RRPN --python /usr/local/bin/python3.7

     Note: path to your Python3.7 may be different.

• Caffe2

  • Follow the steps here to install Caffe2 with GPU support.

---

Installation

1. Clone the repo and install the prerequisites in a Python3.7 virtual environment. Here the repository is cloned to the ~/RRPN folder. Change the BASE_DIR variable below if you want to clone it to another directory.

   BASE_DIR='~/RRPN'
   cd $BASE_DIR
   clone https://github.com/mrnabati/RRPN .
   
   # Install prerequisites in the virtual environment
   workon RRPN
   pip install -r requirements.txt

2. Install the COCO API (pycocotools)

   COCOAPI_DIR='~/cocoapi'
   git clone https://github.com/cocodataset/cocoapi.git $COCOAPI
   cd $COCOAPI_DIR/PythonAPI
   
   # Install into global site-packages
   make install
   
   # Alternatively, if you do not have permissions or prefer
   # not to install the COCO API into global site-packages:
   python setup.py install
   
   # Its safe to remove the downloaded repository now
   rm -r $COCOAPI_DIR

3. Set up the detectron's Python modules:

   cd $BASE_DIR/detectron && make
   
   # Check that Detectron tests pass
   python $BASE_DIR/detectron/tests/test_spatial_narrow_as_op.py

4. Download the NuScenes dataset archive files from its Download Page, unpack the archive files to $BASE_DIR/data/datasets/nuscenes/ without overwriting folders that occur in multiple archives. Eventually you should have the following folder structure:

   $BASE_DIR/data/datasets/nuscenes/
     |- maps
     |- samples
     |- sweeps
     |_ v0.1

5. Add the following symlink for detectron dataset:

   ln -s $BASE_DIR/data/datasets/nucoco $BASE_DIR/detectron/detectron/datasets/data/nucoco

Demo

Will be added soon.

Usage

1. Convert the NuScenes dataset to COCO format using the convert_nuscenes_to_nucoco.sh script in the experiments folder:

   cd $BASE_DIR/experiments
   bash ./convert_nuscenes_to_nucoco.sh

   By default, only the key-frame images in the NuScenes dataset are used to generate the new dataset. Change the INCLUDE_SWEEPS parameter in the script to True if you want to include the non key-frame images as well.

2. Generate RRPN proposals for the newly generated dataset:

   bash ./generate_rrpn_proposals.sh

3. Use the finetune_nucoco.sh script to start training using the RRPN proposals.

   bash ./finetune_nucoco.sh