DyRoNet: A Low-Rank Adapter Enhanced Dynamic Routing Network for Streaming Perception

This repository provides the source code for the paper titled DyRoNet: A Low-Rank Adapter Enhanced Dynamic Routing Network for Streaming Perception along with explanations of the relevant training and testing methods.

The advancement of autonomous driving systems hinges on the ability to achieve low-latency and high-accuracy perception. To address this critical need, this paper introduces Dynamic Routering Network (DyRoNet), a low-rank enhanced dynamic routing framework designed for streaming perception in autonomous driving systems. DyRoNet integrates a suite of pre-trained branch networks, each meticulously fine-tuned to function under distinct environmental conditions. At its core, the framework offers a speed router module, developed to assess and route input data to the most suitable branch for processing. This approach not only addresses the inherent limitations of conventional models in adapting to diverse driving conditions but also ensures the balance between performance and efficiency. Extensive experimental evaluations demonstrating the adaptability of DyRoNet to diverse branch selection strategies, resulting in significant performance enhancements across different scenarios. This work not only establishes a new benchmark for streaming perception but also provides valuable engineering insights for future work. For further information and updates on the project, please visit https://tastevision.github.io/DyRoNet/.

Installation

The basic anaconda environment and follow the install guidelines from DAMO-StreamNet. Based on that, an additional package loralib is needed:

pip install loralib

Quick Start

Dataset Preparation

Follow Argoverse-HD setup instructions

Model Preparation

Due to the utilization of pre-trained parameters from StreamYOLO, LongShortNet, and DAMO-StreamNet in DyRoNet, it is necessary to download the following weight files before training or evaluating DyRoNet:

Model	Download URL	Target Path
StreamYOLO (S)	link	./models/checkpoints/
StreamYOLO (M)	link	./models/checkpoints/
StreamYOLO (L)	link	./models/checkpoints/
LongShortNet (S)	link	./models/checkpoints/
LongShortNet (M)	link	./models/checkpoints/
LongShortNet (L)	link	./models/checkpoints/
DAMO-StreamNet (S)	link	./models/checkpoints/
DAMO-StreamNet (M)	link	./models/checkpoints/
DAMO-StreamNet (L)	link	./models/checkpoints/

The Teacher models of DAMO-StreamNet available here.

Training

bash run_train_lora.sh # for lora fine-tuning
bash run_train_full.sh # for full fine-tuning

Evaluation

You can obtain the weight files using the training method mentioned above. Meanwhile, we also provide the optimal weights which you can download from here and place them under the ./models/ directory.

bash run_eval_lora.sh # for lora fine-tuning
bash run_eval_full.sh # for full fine-tuning

Citation

If you think this work is helpful, please cite:

@misc{huang2024dyronet,
      title={DyRoNet: Dynamic Routing and Low-Rank Adapters for Autonomous Driving Streaming Perception}, 
      author={Xiang Huang and Zhi-Qi Cheng and Jun-Yan He and Chenyang Li and Wangmeng Xiang and Baigui Sun and Xiao Wu},
      year={2024},
      eprint={2403.05050},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}