Our team is actively working towards releasing the code for this project.
We appreciate your patience and understanding as we navigate the necessary processes.
World models have demonstrated superiority in autonomous driving, particularly in the generation of multi-view driving videos. However, significant challenges still exist in generating customized driving videos. In this paper, we propose DriveDreamer-2, which builds upon the framework of DriveDreamer and incorporates a Large Language Model (LLM) to generate user-defined driving videos. Specifically, an LLM interface is initially incorporated to convert a user's query into agent trajectories. Subsequently, a HDMap, adhering to traffic regulations, is generated based on the trajectories. Ultimately, we propose the Unified Multi-View Model to enhance temporal and spatial coherence in the generated driving videos. DriveDreamer-2 is the first world model to generate customized driving videos, it can generate uncommon driving videos (e.g., vehicles abruptly cut in) in a user-friendly manner. Besides, experimental results demonstrate that the generated videos enhance the training of driving perception methods (e.g., 3D detection and tracking). Furthermore, video generation quality of DriveDreamer-2 surpasses other state-of-the-art methods, showcasing FID and FVD scores of 11.2 and 55.7, representing relative improvements of 30% and 50%.
- [2024/03/11] Repository Initialization.
Daytime / rainy day / at night, a car abruptly cutting in from the right rear of ego-car.
cut_in_right.mp4
Rainy day, car abruptly cutting in from the left rear of ego-car. (long video)
cut_in.mp4
Daytime, the ego-car changes lanes to the right side. (long video)
change_lane.mp4
Rainy day, a person crosses the road in the front of the ego-car. (long video)
cross_road.mp4
Daytime / rainy day / at night, ego-car drives through urban street, surrounded by a flow of vehicles on both sides.
vehicle_both_side.mp4
Daytime / rainy day / at night, a bus is positioned to the left front of the ego-car, with a pedestrian near the bus.
bus.mp4
Rainy day, the windshield wipers of the truck are continuously clearing the windshield.
windshield_wiper.mp4
Rainy day, the ego-car makes a left turn at the traffic signal, with vehicles behind proceeding straight through the intersection. (long video)
left_turn.mp4
Daytime, the ego-car drives straight through the traffic light, with a truck situated to the left front and pedestrians crossing on the right side. (long video)
go_straight.mp4
If this work is helpful for your research, please consider citing the following BibTeX entry.
@article{zhao2024drive,
title={DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation},
author={Zhao, Guosheng and Wang, Xiaofeng and Zhu, Zheng and Chen, Xinze and Huang, Guan and Bao, Xiaoyi and Wang, Xingang},
journal={arXiv preprint arXiv:2403.06845},
year={2024}
}