CERBERUS: CEnterR Based Ent-to-end peRception Using a Single model [arXiv]

This is the official code repository for "CERBERUS: Simple and Effective All-In-One Automotive Perception Model with Multi Task Learning"

Model

CERBERUS is a Deep-Learning based model for automotive perception with a Multi-task learning approach. It simultaneously perform object detection (with classification of the occlusion state), lane estimation and scene classification. This model is trained on BDD100k and perform well on real world driving footage, moreover we provide code for inference on Nvidia Jetson boards using the TensorRT framework.

Demo

(Video, click image to reproduce)

Requirements

This codebase is built using PyTorch and PyTorch-Lightning, together with several other libraries. To replicate our experimental setup we highly recommend to create and Ananconda environment, if you don't have Anaconda installed yet you can install it.

• Clone this repository

git clone git@github.com:cscribano/CERBERUS.git
cd CERBERUS

• Create a new Aanaconda environment

conda create -n cerberus python=3.8

• Install requirements

conda activate cerberus
conda install pytorch==1.8.1 torchvision==0.9.1 cudatoolkit=11.1 -c pytorch -c conda-forge -y
pip install -r requirements.txt

To perform inference using TensorRT the framework must be installed following the NVidia documentation. We officially supportonly TensorRT version 8.4.0.

Inference

We provide a set of pretrained models above (Model Zoo) both as pytorch checkpoints and exported to onnx (for inference using TensorRT).

Inference using PyTorch

• Download the configuration file and the corresponding .pth checkpoint.
• The sample inference script perform inference on a video file, as an example you can find the public footage used for our demo here (credit)
• Perform inference:

cd CERBERUS
python inference/run.py -c <path_to_config>.json -w <path_to_checkpoint>.pth -v <path_to_video>.mp4

• In addition you can add the option -o true to perform conversion to .onnx

Interence using TensorRT

The sample inference code is similar to the PyTorch version, but it requires as input the .onnx model and the video file. When a model is loaded for the first time it is converted to tensorrt, this process usually takes a very long time.

cd CERBERUS
python inference/run_tensorrt.py -m <path_to_model>.onnx -v <path_to_video>.mp4

Running on NVidia Jetson

We provide a script to automatically install all the requirements on an Nvidia board, it is only tested on Jetson AGX Xavier and Jetson Nano on a fresh install of JetPack 4.6.1

git clone git@github.com:cscribano/CERBERUS.git
cd CERBERUS/docs
chmod +x deps.sh
./deps.sh

The installation process require a very long time, at the end reboot is required. When done inference can be performed using TensorRT as described above.

Model Zoo

Model	configuration	weights	onnx
resnet34_simple	resnet34_simple.json	resnet34_simple.pth	resnet34_simple_sim.onnx
resnet34_bifpn	resnet34_bifpn.json	resnet34_bifpn.pth	resnet34_bifpn_sim.onnx
resnet50_bifpn	resnet50_bifpn.json	resnet50_bifpn.pth	resnet50_bifpn_sim.onnx
resnet101_bifpn	resnet101_bifpn.json	resnet101_bifpn.pth	resnet101_bifpn_sim.onnx
efficientnetb2_bifpn	efficientnetb2_bifpn.json	efficientnetb2_bifpn.pth	efficientnetb2_bifpn_sim.onnx
mobilenetv2_bifpn	mobilenetv2_bifpn.json	mobilenetv2_bifpn.pth	mobilenetv2_bifpn_sim.onnx

Training

Dataset

Only BDD100K is supported, the dataset is available for free from the autors (https://bdd-data.berkeley.edu/). After downloading the dataset a pre-processing step is required to encode the lane makins annotations in the format required by Cerberus, we provide the script data/bdd100k_lane_keypoints.py to take care of this step.

cd CERBERUS/data
python bdd100k_lane_keypoints.py -i <BDD100k_root>/bdd100k_images/images/100k -l <BDD100k_root>/bdd100k_lanes/labels/lane

The files train_25_new.pt and val_25_new.pt will be saved in the /data directory. If you have any problem in this step we provide the preprocessed labels here

Start training

• Adjust the configuration .json file in ./conf/experiments accordingly to the directories where you extracted the dataset and the processet lane labels.
• Run the training loop:

python main.py --exp_name=<name_of_conf_file>

Please notice that the .json extension is NOT part of the experiment name: i.e, conf file is "experiments/resnet34_bifpn.json" --> --exp_name=resnet34_bifpn

References

Presented at PNARUDE Workshop @ IROS

If you use CERBERUS in your research, please cite the following paper.

@article{scribano2022cerberus,
  title={CERBERUS: Simple and Effective All-In-One Automotive Perception Model with Multi Task Learning},
  author={Scribano, Carmelo and Franchini, Giorgia and Olmedo, Ignacio Sanudo and Bertogna, Marko},
  journal={arXiv preprint arXiv:2210.00756},
  year={2022}
}

For use in commercial solutions, write at carmelo.scribano@unimore.it or refer to https://hipert.unimore.it/ .