Language Prompt for Autonomous Driving
Dongming Wu*, Wencheng Han*, Tiancai Wang, Yingfei Liu, Xiangyu Zhang, Jianbing Shen
This is the official implementation of Language Prompt for Autonomous Driving.
- We propose a new large-scale language prompt set for driving scenes, named NuPrompt. As far as we know, it is the first dataset specializing in multiple 3D objects of interest from video domain.
- We construct a new prompt-based driving perceiving task, which requires using a language prompt as a semantic cue to predict object trajectories.
- We develop a simple end-to-end baseline model, called PromptTrack, which effectively fuses cross-modal features in a newly built prompt reasoning branch to predict referent objects, showing impressive performance.
- [2023.09.11] Our paper is released at arXiv.
We expand nuScenes dataset with annotating language prompts, named NuPrompt. It is a large-scale dataset for language prompt in driving scenes, which contains 35,367 language prompts for 3D objects, each for an average of 5.3 instances. Thanks to nuScenes, our descriptions are closed to real-driving nature and complexity, covering a 3D, multi-view, and multi-frame space.
Our model is built upon PF-Track. We add a new prompt reasoning branch to predict the prompt-referred objects. The code will be released at future.
If you find our work useful in your research, please consider citing it.
@article{wu2023language,
title={Language Prompt for Autonomous Driving},
author={Wu, Dongming and Han, Wencheng and Wang, Tiancai and Liu, Yingfei and Zhang, Xiangyu and Shen, Jianbing},
journal={arXiv preprint},
year={2023}
}
We thank the authors that open the following projects.