Awesome Vision–Language–Action Models for Autonomous Driving 🚗

[Survey Slides] [自动驾驶之心解读] [具身进化中文解读] [焉知汽车中文解读] [知乎中文解读]

Welcome to Awesome VLA4AD—a curated, continuously updated collection of research papers and resources on Vision–Language–Action models for Autonomous Driving (VLA4AD). This repository tracks the latest advances in VLA4AD, from explanatory perception modules to end-to-end reasoning and control architectures.

Our latest survey is here. We invite your feedback and discussion!

⭐️ Follow & Star to stay up to date!
🤝 Contributions welcome—if you know of new papers, datasets, or tools, please open an issue or submit a PR.
📬 Questions or suggestions? Reach us at sicong.jiang@mail.mcgill.ca or qka23@mails.tsinghua.edu.cn.

---

📜 Citation

If this project is useful in your work, we'd appreciate a star 🌟 and a citation of our survey.

@article{jiang2025survey,
  title={A Survey on Vision-Language-Action Models for Autonomous Driving},
  author={Jiang, Sicong and Huang, Zilin and Qian, Kangan and Luo, Ziang and Zhu, Tianze and Zhong, Yang and Tang, Yihong and Kong, Menglin and Wang, Yunlong and Jiao, Siwen and others},
  journal={arXiv preprint arXiv:2506.24044},
  year={2025}
}

---

📚 Table of Contents

• 🔥 Motivation & Paradigm Shift
• 🚀 Overview of VLA4AD
• 🏆 Awesome VLA4AD Papers
  • 1️⃣ Pre-VLA: VLM as Explainers
  • 2️⃣ Modular VLA4AD
  • 3️⃣ End-to-End VLA4AD
  • 4️⃣ Reasoning-Augmented VLA4AD
• 📊 Datasets & Benchmarks
• ⚙️ Installation & Usage

---

🔥 Motivation & Paradigm Shift

The development of autonomous driving has progressed from modular pipelines to fully integrated systems. This survey summarizes the latest advances into three core paradigms:

• End-to-End AD: Direct sensor-to-control mapping—efficient but opaque and weak on rare scenarios.

  • Flow: Sensors → Network → Actions

• VLMs for AD: Adds language reasoning—boosts explainability but doesn’t drive the vehicle.

  • Flow: Sensors → VLM → Answers

• VLA for AD: Unifies vision, language, and control in one policy—understands instructions, reasons, acts, and explains.

  • Flow: Sensors → Multimodal Encoder → LLM/VLM → Decoder → Actions

Figure 1. (a) conventional end-to-end AD, (b) vision-language models as explainers, (c) full Vision–Language–Action systems.

---

🚀 Overview of VLA4AD

A typical VLA4AD model follows an “Input–Process–Output” flow, unifying environment perception, instruction understanding, and vehicle control.

Figure 2. Overview of VLA4AD, integrating vision, language, and action modules.

A snapshot of the field’s evolution through four successive stages—from VLM-as-explainer to augmented, reasoning-centric agents:

Figure 3. Progression of VLA4AD models: (1) VLMs as passive explainers; (2) Modular VLA with intermediate representations; (3) End-to-end VLA mapping sensors directly to actions; (4) Augmented VLA with long-horizon reasoning and tool use.

The following table shows the representative models of VLA4AD and their inside modules:

Table 1. Representative VLA4AD Models (2023–2025). Sensor Inputs: Single = single forward-facing camera input; Multi = multi-view camera input; State = vehicle state information & other sensor input. Outputs: LLC= low-level control, Traj.= future trajectory, Multi.= multiple tasks such as perception, prediction or planning.

---

🏆 Awesome VLA4AD Papers

1️⃣ Pre-VLA: VLM as Explainers

Model	Year	Key Features	Link
DriveGPT-4	2023	Scene Narration, QA	https://arxiv.org/abs/2310.01412 / Code
TS-VLM	2025	Text-guided Attention	https://arxiv.org/abs/2505.12670 / Code
DynRsl-VLM	2025	Adaptive Resolution	https://arxiv.org/abs/2503.11265

2️⃣ Modular VLA4AD

Model	Year	Key Features	Link
RAG-Driver	2024	Retrieval-Augmented	https://arxiv.org/abs/2402.10828 / Code
OpenDriveVLA	2025	Language-guided Planning	https://arxiv.org/abs/2503.23463 / Code
DriveMoE	2025	Expert Routing	https://arxiv.org/abs/2505.16278 / Code
LangCoop	2025	V2V Coordination	https://arxiv.org/abs/2504.13406 / Code
SafeAuto	2025	Rule-based Safety	https://arxiv.org/abs/2503.00211 / Code
ReCogDrive	2025	Diffusion + RL	https://arxiv.org/abs/2506.08052 / Code

3️⃣ End-to-End VLA4AD

Model	Year	Key Features	Link
ADriver-I	2023	Diffusion-based World Model	https://arxiv.org/abs/2311.13549
EMMA	2024	Detection + Planning	https://arxiv.org/abs/2410.23262 / Code
CoVLA-Agent	2024	Caption + Trajectory	https://arxiv.org/abs/2408.10845 / Code
SimLingo	2025	Action Dreaming	https://arxiv.org/abs/2503.09594 / Code
DiffVLA	2025	Sparse-Dense Diffusion	https://arxiv.org/abs/2505.19381
S4-Driver	2025	Sparse 3D Representation	https://arxiv.org/abs/2505.24139

4️⃣ Reasoning-Augmented VLA4AD

Model	Year	Key Features	Link
ORION	2025	Memory + Rationales	https://arxiv.org/abs/2503.19755 / Code
Impromptu-VLA	2025	CoT-Aligned Planning	https://arxiv.org/abs/2505.23757 / Code
FSDrive	2025	Visual Reasoning	https://arxiv.org/abs/2505.17685 / Code
AutoVLA	2025	Drive Tokens + CoT	https://arxiv.org/abs/2506.13757 / Code
Drive-R1	2025	CoT-Aligned Planning	https://arxiv.org/abs/2506.18234
Alpamayo-R1	2025	Chain of Causation Reasoning	https://arxiv.org/abs/2511.00088

---

📊 Datasets & Benchmarks

Name	Year	Modality	Task	URL
BDD100K / BDD-X	2018	Video + Rationales	Captioning, QA	bdd-data.berkeley.edu
nuScenes	2020	Camera, LiDAR, Radar	Detection, QA	www.nuscenes.org
Bench2Drive	2024	CARLA Simulator	Closed-loop Driving	Github
Reason2Drive	2024	Video–QA	CoT-Chain Consistency	Github
Impromptu-VLA Dataset	2025	Video + QA + Traj	Corner-Case Testing	Github
NuInteract	2025	Multi-view QA	3D QA	Github
DriveAction	2025	In-the-wild QA	High-level Actions	HuggingFace

---

⚙️ Installation & Usage

git clone https://github.com/JohnsonJiang1996/Awesome-VLA4AD.git
cd Awesome-VLA4AD
# Browse papers, datasets & code samples in each folder

---