Skip to content
BranchesTags

Repository files navigation

Awesome RL-VLA for Robotic Manipulation πŸ€–

[Paper]

A curated list of papers and resources on Reinforcement Learning of Vision-Language-Action (RL-VLA) models for Robotic Manipulation. This repository provides a comprehensive overview of training paradigms, methodologies, and state-of-the-art approaches in RL-VLA research.

πŸ“’ Latest News

πŸ”₯ [November 2025] Our comprehensive survey paper "A Survey on Reinforcement Learning of Vision-Language-Action Models for Robotic Manipulation" is now available on TechRxiv! Stay tuned for future updates.

πŸ“– Table of Contents

πŸ” Overview

RL training is crucial for enabling VLAs to generalize out-of-distribution (OOD) from large-scale pre-trained data. Existing RL-VLA training paradigms can be categorized into three types based on how agents obtain and utilize feedback from the environment:

  • Online RL-VLA: Direct interaction with the environment during training
  • Offline RL-VLA: Learning from static datasets without further environmental interaction
  • Test-time RL-VLA: Models adapt their behavior during deployment without altering parameters

πŸš€ Training Paradigms

Offline RL-VLA

Offline RL trains VLA models on pre-collected static datasets, enabling learning independently from environment interactions. This paradigm is suitable for high-risk or resource-constrained deployment scenarios.

Key Research Directions:

  • Data Utilization: Effective utilization of static datasets for policy improvement
  • Objective Modification: Customizing RL objectives for novel architectures and data augmentation

Online RL-VLA

Online RL-VLA enables interactive policy learning through continuous environment interaction, empowering pre-trained VLAs with adaptive closed-loop control capability for real-world OOD environments.

Key Research Directions:

  • Policy Optimization: Direct policy improvement based on environmental rewards
  • Sample Efficiency: Learning effective policies with limited interaction budget
  • Active Exploration: Efficient exploration strategies for higher performance gains
  • Training Stability: Ensuring consistent policy updates and convergence
  • Infrastructure: Scalable frameworks for online RL-VLA training

Test-time RL-VLA

Test-time RL-VLA adapts behavior during deployment through lightweight updates, addressing the expensive cost of full model fine-tuning in real-world scenarios.

Key Adaptation Mechanisms:

  • Value Guidance: Using pre-trained value functions to influence action selection
  • Memory Buffer Guidance: Retrieving relevant historical experiences during inference
  • Planning-guided Adaptation: Explicit reasoning over future action sequences

πŸ“š Paper Collection

Legend

  • Action: AR (Autoregressive), Diffusion, Flow (Flow-matching)
  • Reward: D (Dense Reward), S (Sparse Reward)
  • Model Type: MB (Model-based), MF (Model-free)
  • Environment: Sim. (Simulation), Real (Real-world)
  • Task: MT (Multi-task), ST (Single-task)
  • Policy: On-Policy, Off-Policy, Hybrid (mixed on/off-policy), Test-time (inference-time adaptation)

Offline RL-VLA

Method Date Publication Sim. Real Base VLA Model Action Reward Algorithm Policy Type Project
Q-Transformer 2023.10 CoRL23πŸ”— βœ“ βœ— Transformer AR S CQL Off-Policy MF πŸ”—
PAC 2024.02 ICML24πŸ”— βœ“ βœ“ Perceiver-Actor-Critic AR S AC Off-Policy MF πŸ”—
GeRM(Quadruped Robot) 2024.03 IROS24πŸ”— βœ“ βœ— Transformer-MoE AR S CQL Off-Policy MF πŸ”—
MoRE(Quadruped Robot) 2025.03 ICRA25πŸ”— βœ— βœ“ MLLM-MoE AR S CQL Off-Policy MF -
ReinboT 2025.05 ICML25πŸ”— βœ“ βœ“ ReinboT AR D DT + RTG Off-Policy MF πŸ”—
CO-RFT 2025.08 - βœ— βœ“ RoboVLMs AR D Cal-QL + TD3 Off-Policy MF -
ARFM 2025.09 AAAI26πŸ”— βœ“ βœ“ Ο€β‚€ Flow D ARFM Off-Policy MF -
$Ο€^*_{0.6}$ 2025.11 - βœ— βœ“ $Ο€_{0.6}$ Flow D RECAP Off-Policy MF πŸ”—
NORA-1.5 2025.11 - βœ“ βœ“ NORA-1.5 AR / Flow D DPO Off-Policy MB πŸ”—
GigaBrain-0.5M* 2026.2 - βœ— βœ“ GigaBrain-0.5 Flow D RAMP Off-Policy MB πŸ”—
ARM 2026.4 - βœ— βœ“ GR00T N1.5 Flow D AW-BC Off-Policy MF πŸ”—

Online RL-VLA

Method Date Publication Sim. Real Base VLA Model Action Reward Algorithm Policy Type Project
FLaRe 2024.09 ICRA25πŸ”— βœ“ (ST) βœ“ (ST) SPOC AR S PPO On-Policy MF πŸ”—
PA-RL 2024.12 ICLR25 WorkshopπŸ”— βœ“ (ST) βœ“ (ST) OpenVLA AR S PA-RL Off-Policy MF πŸ”—
RLDG 2024.12 RSS25πŸ”— βœ— βœ“ (ST) OpenVLA / Octo AR / Diffusion S RLPD Off-Policy MF πŸ”—
iRe-VLA 2025.01 ICRA25πŸ”— βœ“ (MT) βœ“ (MT) iRe-VLA AR S SACfD + SFT Off-Policy MF -
GRAPE 2025.02 ICRA25 PosterπŸ”— βœ“ (MT) βœ“ (MT) OpenVLA AR D TPO On-Policy MF πŸ”—
SafeVLA 2025.03 NeurIPS25 PosterπŸ”— βœ“ (ST) βœ— SPOC AR S PPO On-Policy MF πŸ”—
RIPT-VLA 2025.05 - βœ“ (MT) βœ— QueST / OpenVLA-OFT AR S LOOP On-Policy MF πŸ”—
VLA-RL 2025.05 - βœ“ (MT) βœ— OpenVLA AR D PPO On-Policy MF πŸ”—
RLVLA 2025.05 NeurIPS25 PosterπŸ”— βœ“ (MT) βœ— OpenVLA AR S PPO / GRPO / DPO Hybrid MF πŸ”—
RFTF 2025.05 - βœ“ (MT) βœ— GR-MG, Seer AR D PPO On-Policy MF -
TGRPO 2025.06 - βœ“ (ST) βœ— OpenVLA AR D GRPO On-Policy MF -
RLRC 2025.06 - βœ“ (MT) βœ— OpenVLA AR S PPO On-Policy MF πŸ”—
ThinkAct 2025.07 NeurIPS25 PosterπŸ”— βœ“ (MT) βœ— MLLM + DiT AR / Diffusion D GRPO (System 2) On-Policy MF πŸ”—
DiffusionRL-VLA 2025.9 - βœ“ βœ— Ο€β‚€ Flow S PPO(DP) + BC(VLA) On-Policy MF -
SimpleVLA-RL 2025.09 ICLR26 PosterπŸ”— βœ“ (MT) βœ“ (ST) OpenVLA-OFT AR S GRPO On-Policy MF πŸ”—
Dual-Actor FT 2025.09 IROS25 Workshop Extended AbstractπŸ”— βœ“ (MT) βœ“ (MT) Octo / SmolVLA Diffusion S QL + BC Off-Policy MF πŸ”—
Generalist 2025.09 NeurIPS25 PosterπŸ”— βœ“ (MT) βœ“ (MT) PaLI 3B AR D REINFORCE On-Policy MF πŸ”—
VLAC 2025.09 - βœ— βœ“ (MT) VLAC AR D PPO On-Policy MF πŸ”—
Robo-Dopamine 2025.12 CVPR26πŸ”— βœ“ (MT) βœ“ (MT) Pi0.5 Flow D PPO On-Policy MF πŸ”—
AC PPO 2025.09 - βœ“ (ST) βœ— Octo-small AR S PPO+BC On-Policy MF -
VLA-RFT 2025.10 - βœ“ (MT) βœ— VLA-Adapter Flow D GRPO On-Policy MB πŸ”—
RLinf-VLA 2025.10 - βœ“ (MT) βœ“ (MT) OpenVLA / OpenVLA-OFT AR S PPO / GRPO On-Policy MF πŸ”—
FPO 2025.10 - βœ“ (MT) βœ— Ο€β‚€ Flow S FPO On-Policy MF -
ReSA 2025.10 - βœ“ (MT) βœ— OpenVLA AR D PPO + SFT On-Policy MF -
Ο€_RL 2025.10 - βœ“ (MT) βœ— Ο€β‚€ / Ο€β‚€.β‚… Flow S PPO / GRPO On-Policy MF πŸ”—
PLD 2025.10 ICLR26 PosterπŸ”— βœ“ (MT) βœ“ (MT) OpenVLA / Ο€β‚€ / Octo AR / Flow S Cal-QL + SAC Off-Policy MF πŸ”—
DeepThinkVLA 2025.10 - βœ“ (MT) βœ— Ο€β‚€-Fast AR S GRPO On-Policy MF πŸ”—
World-Env 2025.11 - βœ“ (ST) βœ“ (ST) OpenVLA-OFT AR D PPO On-Policy MB πŸ”—
RobustVLA 2025.11 - βœ“ (MT) βœ— OpenVLA-OFT AR D PPO On-Policy MF -
WMPO 2025.11 ICLR26 PosterπŸ”— βœ“ (MT) βœ“ (MT) OpenVLA-OFT AR S GRPO On-Policy MB πŸ”—
ProphRL 2025.11 - βœ“ (ST) βœ“ (ST) VLA-Adapter / Ο€0.5 / OpenVLA-OFT(flow action) Flow S FA-GRPO On-Policy MB πŸ”—
EVOLVE-VLA 2025.12 - βœ“ (MT) βœ— OpenVLA-OFT AR D GRPO On-Policy MB(VLAC) πŸ”—
SOP 2026.1 - βœ— βœ“ (MT) Ο€0.5 Flow S HG-DAgger / RECAP Off-Policy MF πŸ”—
Green-VLA 2026.1 - βœ“ (MT) βœ“ (MT) Green-VLA Flow S IQL + actor-critic Off-Policy MF πŸ”—
SA-VLA 2026.1 - βœ“ (MT) βœ— Ο€0.5 Flow D PPO On-Policy MF πŸ”—
E2HiL 2026.1 - βœ— βœ“ (MT) Octo Diffusion S RLPD Off-Policy MF πŸ”—
World-Gymnast 2026.2 ICLR26 WorkshopπŸ”— βœ“ (MT) βœ“ (MT) OpenVLA-OFT AR S GRPO On-Policy MB πŸ”—
RL-VLA3 2026.2 ICLR26 WorkshopπŸ”— βœ“ (MT) βœ— Ο€0 / Ο€0.5 / GR00T N1.5 / OpenVLA-OFT Flow / AR S PPO / GRPO On-Policy MF β€”
World-VLA-Loop 2026.2 - βœ“ (ST) βœ“ (ST) OpenVLA-OFT AR S GRPO On-Policy MB πŸ”—
RISE 2026.2 - βœ— βœ“ (ST) Ο€0.5 Flow D RISE On-Policy MB πŸ”—
WoVR 2026.2 - βœ“ (MT) βœ“ (MT) OpenVLA-OFT AR S GRPO On-Policy MB πŸ”—
ALOE 2026.2 - βœ— βœ“ (ST) Ο€β‚€.β‚… Flow S AWR(Advantage-Weighted Regression) Off-Policy MF πŸ”—
TwinRL-VLA 2026.2 - βœ— βœ“ (ST) Octo Diffusion S Actor-Critic Off-Policy MF β€”
RL-Co 2026.3 - βœ“ (ST) βœ“ (ST) OpenVLA / Ο€0.5 AR / Flow D ReinFlow / GRPO On-Policy MF β€”
Ο€_StepNFT 2026.3 - βœ“ (MT) βœ— Ο€β‚€ / Ο€β‚€.β‚… Flow S NFT On-Policy MF πŸ”—
ROBOMETER 2026.3 - βœ— βœ“ (MT) Ο€β‚€ Flow D DSRL Off-Policy MF πŸ”—
AtomVLA 2026.3 - βœ“ (MT) βœ“ (ST) AtomVLA Flow D GRPO On-Policy MB β€”
NS-VLA 2026.3 - βœ“ (MT) βœ— NS-VLA AR D GRPO On-Policy MF πŸ”—
Gen3D-RL-VLA 2026.03 - βœ“(MT) βœ“(MT) Ο€β‚€.β‚… Flow S PPO On-Policy MB -
Simple Recipe Works 2026.03 - βœ“(MT) βœ— OpenVLA-OFT / Ο€β‚€ / Ο€β‚€-Fast AR / Flow S PPO On-Policy MF πŸ”—
RoboAlign 2026.03 - βœ“(MT) βœ“(MT) MLLM + Diffusion AR D GRPO On-Policy MF -
SmoothVLA 2026.03 - βœ“(MT) βœ— OpenVLA-OFT AR D GRPO On-Policy MF -
AcceRL 2026.03 - βœ“(MT) βœ— OpenVLA-OFT AR S PPO On-Policy MB πŸ”—
VLA-MBPO 2026.03 - βœ“(MT) βœ“(MT) Ο€β‚€ / OpenVLA Flow / AR S MBPO On-Policy MB -
VLA-OPD 2026.03 - βœ“(MT) βœ— OpenVLA-OFT AR D OPD On-Policy MF -
OmniVLA-RL 2026.04 - βœ“(MT) βœ— OmniVLA-RL (MoT) Flow D Flow-GSPO On-Policy MF -
VLAJS 2026.04 - βœ“(MT) βœ“(ST) VLA-guided RL agent AR S PPO On-Policy MF -
DAERT 2026.04 - βœ“(MT) βœ— Ο€β‚€ / OpenVLA AR / Flow S Diversity-aware RL On-Policy MF -
RL Token 2026.04 - βœ— βœ“(ST) Ο€β‚€.β‚… Flow D RLPD Off-Policy MF πŸ”—
LaST-R1 2026.04 - βœ“(MT) βœ— LaST-R1 AR D LAPO On-Policy MF -

Offline + Online RL-VLA

Method Date Publication Sim. Real Base VLA Model Action Reward Algorithm Policy Type Project
ConRFT 2025.4 RSS26πŸ”— βœ— βœ“(MT) Octo-small Diffusion S Cal-QL + BC Off-Policy MF πŸ”—
SRPO 2025.11 - βœ“(MT) βœ“(MT) OpenVLA* / Ο€β‚€ / Ο€β‚€-Fast AR / Flow D SRPO Hybrid MF (MB-Reward but MF-RL) πŸ”—
DLR 2025.11 - βœ“ βœ— Ο€β‚€ / OpenVLA Flow / AR S PPO(MLP) + SFT(VLA) On-Policy MF -
GR-RL 2025.12 - βœ— βœ“ GR-3 Flow S TD3 / DSRL Off-Policy MF πŸ”—
STARE-VLA 2025.12 - βœ“ βœ— OpenVLA / Ο€β‚€.β‚… AR / Flow D PPO / TPO / SFT On-Policy MF πŸ”—
IG-RFT 2026.2 - βœ— βœ“ Ο€β‚€.β‚… Flow D IG-AWR off-policy MF β€”
POCO 2026.04 - βœ“(MT) βœ“(MT) Ο€β‚€ / Octo Flow / Diffusion D POCO (EM + Clipped) Off-Policy MF πŸ”—

Test-time RL-VLA

Method Date Publication Sim. Real Base VLA Model Action Reward Algorithm Policy Type Project
V-GPS 2024.10 CoRL25πŸ”— βœ“(MT) βœ“(MT) Octo / RT-1 / OpenVLA AR / Diffusion D Cal-QL Test-time MF πŸ”—
Hume 2025.5 CVPR26πŸ”—/ βœ“(MT) βœ“(MT) Hume Flow S Value Guidance Test-time MF πŸ”—
DSRL 2025.6 CoRL25πŸ”— βœ“(MT) βœ“(MT) DP / Ο€β‚€ Diffusion / Flow S Diffusion Steering Test-time MF πŸ”—
VLA-Reasoner 2025.9 ICRA26πŸ”— βœ“(ST) βœ“(ST) OpenVLA / SpatialVLA / Ο€β‚€-Fast AR / Diffusion D MCTS Test-time MB πŸ”—
RoVer 2025.10 - βœ“(MT) βœ“(MT) OpenVLA / Ο€β‚€ / GR00T-N1.5 AR / Flow D PRM Verifier Test-time MF -
VLAPS 2025.11 CoRL25 WorkshopπŸ”— βœ“(ST) βœ— Octo Diffusion S MCTS Test-time MB πŸ”—
VLA-Pilot 2025.11 - βœ— βœ“(ST) DiVLA / RDT AR / Diffusion D Value GuidanceT Test-time MB(MLLM) πŸ”—
TACO 2025.12 - βœ“ βœ“(ST) Ο€β‚€ / OpenVLA et al. Flow S CNF estimation Test-time MF πŸ”—
TT-VLA 2026.1 - βœ“(ST) βœ“(ST) Nora / OpenVLA / TraceVLA AR D PPO (Value-free) Test-time MF -
VLS 2026.2 - βœ“(MT) βœ“(MT) OpenVLA / Ο€β‚€ / Ο€β‚€.β‚… Flow D gradient-based steer Test-time MB(VLM) πŸ”—
FASTER 2026.04 - βœ“(ST) βœ— Ο€β‚€.β‚… Flow D Value-guided Denoising MDP Test-time MF -

Note: The πŸ”— symbol in the Project column indicates papers with available project pages, GitHub repositories, or demo websites.

πŸ”— Useful Resources

🎯 RL-VLA Action Optimization

Different VLA architectures require distinct RL optimization strategies based on their action generation mechanisms:

RL-VLA Action Optimization

  • πŸ”€ Autoregressive VLA: Optimizes actions at the token-level. Each action token is individually optimized through RL, enabling fine-grained control over action sequences but requiring careful handling of sequential dependencies.

  • 🌊 Generative VLA (Diffusion/Flow): Optimizes along the action generation process at the sequence-level. The entire action trajectory is optimized as a cohesive unit through the denoising or flow-matching process, providing holistic action optimization.

  • πŸ”— Dual-system VLA: Optimizes at the bridge-level. RL decides which high-level action proposal to pass to the fast controller, creating a hierarchical optimization approach that complements both token-level and sequence-level methods.

Base VLA Models

Datasets & Benchmarks

  • Open X-Embodiment - Large-scale robotic datasets
  • LIBERO - Benchmark for lifelong robot learning
  • SimplerEnv - Benchmark for real-sim robot learning
  • RoboTwin - Benchmark for bimanual robot learning
  • DeepPHY - Benchmark for physical reasoning

Frameworks & Tools

  • RLinf - Infrastructure for online RL fine-tuning of VLAs
  • RLinfv0.2 - Infrastructure for real world RL

🀝 Contributing

We welcome contributions to this awesome list! Please feel free to:

  1. Add new papers: Submit a PR with new RL-VLA papers following the existing format
  2. Update information: Correct any errors or update paper information
  3. Suggest improvements: Propose better organization or additional sections

Contribution Guidelines

  • Ensure papers are relevant to RL-VLA research
  • Include paper links, project pages (if available), and key details
  • Follow the existing table format for consistency
  • Add a brief description for new paradigms or significant methodological contributions

πŸ“„ Citation

If you find this repository useful, please consider citing:

@article{pine2025rlvla,
  title={A Survey on Reinforcement Learning of Vision-Language-Action Models for Robotic Manipulation},
  author={Haoyuan Deng, Zhenyu Wu, Haichao Liu, Wenkai Guo, Yuquan Xue, Ziyu Shan, Chuanrui Zhang, Bofang Jia, Yuan Ling, Guanxing Lu, and Ziwei Wang},
  journal={TechRxiv},
  year={2025},
  doi={10.36227/techrxiv.176531955.54563920/v1},
  note={Preprint}
}

⭐ Star History

Star this repository if you find it helpful!

Star History Chart

About

A Survey on Reinforcement Learning of Vision-Language-Action Models for Robotic Manipulation

Resources

Readme
Activity

Stars

837 stars

Watchers

10 watching

Forks

25 forks
Report repository

Releases

Packages

Contributors