RawVLA: Embodied Neural Image Signal Processor for Robotic Manipulation

Shuhong Liu^1,2, Heng Zhou², Lingfeng Qian², Yuhao Fang², Xianbao Hou², Qianyu Zhou¹, Lin Gu⁴, Wei Sui², Jianfei Yang³, Ziteng Cui^1,5

¹The University of Tokyo   ²D-Robotics   ³NTU   ⁴Tohoku University   ⁵HKUST(GZ)

RAW-VLA is a streaming, illumination-adaptive RAW image frontend for vision-language-action policies. This repository contains the RAW frontend, StarVLA integration, RAWVLA-Bench lighting transforms, and LIBERO and RoboTwin 2.0 evaluation adapters.

Clone the repository with its submodules:

git clone --recurse-submodules https://github.com/ShuhongLL/RawVLA.git
cd RawVLA
git submodule sync --recursive
git submodule update --init --recursive

Several project-maintained submodules are branches of this repository. Their branch and pinned commit are documented in SUBMODULES.md.

Install only the environment needed for your workflow:

The robotwin environment runs the simulator; RAW-VLA training for both benchmarks uses the StarVLA environment. RoboTwin π0/π0.5 training recipes are in configs/robotwin.

Standalone OpenVLA-OFT and GR00T have separate, optional environments; see ENVIRONMENTS.md for their installation and validation.

Use this environment for StarVLA/RAW-VLA training, LIBERO evaluation, and the StarVLA policy server used alongside RoboTwin:

conda env create -f environment-libero.yml
conda activate starvla-libero
python -m pip install --no-deps --no-build-isolation -e ./third_party/LIBERO
python -m pip install --no-deps -e ./third_party/openvla-oft
python -m pip install --no-deps -e ./starVLA
export LIBERO_CONFIG_PATH="$PWD/.local/libero"
python scripts/setup/configure_libero.py

The last command writes ignored, machine-local simulator paths to .local/libero/config.yaml. Versioned RAW-VLA training recipes are in configs/libero.

Install the simulator for evaluation in its own environment. The asset script downloads the official assets unless you set ROBOTWIN_ASSETS_SOURCE to an existing asset directory:

conda env create -f environment-robotwin2.yml
conda activate robotwin
bash scripts/setup/setup_robotwin_assets.sh
ROBOTWIN_PYTHON=python bash third_party/RoboTwin/script/_install.sh

Evaluation with a StarVLA policy server also uses the starvla-libero environment above. See ENVIRONMENTS.md for interpreter selection, host-specific graphics requirements, and validation commands.

Download the public StarVLA LIBERO checkpoints and their base models with:

python tools/checkpoints/download_starvla_libero.py
# Add --all for every listed model family.
python tools/checkpoints/validate_starvla_libero.py

Set STARVLA_ROOT=/custom/path/to/starVLA when the StarVLA checkout is not at the repository default.

The train and test resources are hosted separately: paired training trajectories are on Hugging Face, while the frozen test/evaluation protocols are JSON manifests in this GitHub repository. No pre-rendered test trajectories are included in the Hugging Face dataset.

The paired RAW/RGB training trajectories are available directly from RawVLA-Bench on Hugging Face. It contains 2,403 successful trajectories: 1,771 from LIBERO and 632 from RoboTwin 2.0.

hf download ToferFish/RawVLA-Bench \
  --repo-type dataset \
  --local-dir benchmark_data/RawVLA-Bench

After downloading, the train files are located at:

• LIBERO: benchmark_data/RawVLA-Bench/data/libero/raw/ and benchmark_data/RawVLA-Bench/data/libero/rgb/.
• RoboTwin 2.0: benchmark_data/RawVLA-Bench/data/robotwin2/raw/ and benchmark_data/RawVLA-Bench/data/robotwin2/rgb/.

Optional data regeneration is documented in replay/README.md.

The test protocols are recorded in these frozen simulator rollout manifests:

• LIBERO full evaluation: 40 tasks × 50 initial states × 5 lighting domains = 10,000 rollouts.
• LIBERO smaller evaluation: 40 tasks × 10 initial states × 5 lighting domains = 2,000 rollouts.
• RoboTwin 2.0 evaluation: 13 tasks × 50 seeds × 5 lighting domains = 3,250 rollouts.

The manifests specify task, seed, and lighting choices. The LIBERO runner reads its manifest directly. The RoboTwin JSON documents the 3,250 benchmark cases, but the RoboTwin runner shown below uses its own task list and easy/hard settings; it does not execute those JSON entries automatically. The example command below launches an evaluation, not the exact frozen 3,250-rollout protocol. During evaluation, the simulator renders observations and the policy produces actions.

The paper's main RawVLA-Bench training recipes are:

The LIBERO recipes inherit rawvla_libero.yaml; the RoboTwin recipes inherit rawvla_robotwin.yaml. Both use the shared rawvla.yaml defaults. For RoboTwin, prepare the frozen backbones with the backbone guide and set RAWVLA_MODEL_ROOT and RAWVLA_TOKENIZER_PATH as described in the RoboTwin training guide. Additional experimental LIBERO recipes are in configs/libero/optional.

For example, train Qwen3-OFT on the published LIBERO trajectories:

export RAWVLA_ROOT="$PWD"
export LIBERO_CONFIG_PATH="$RAWVLA_ROOT/.local/libero"
conda activate starvla-libero
cd starVLA
accelerate launch --num_processes 1 \
  starVLA/training/train_starvla.py \
  --config_yaml ../configs/libero/qwen3_oft.yaml \
  --datasets.vla_data.cache_root "$RAWVLA_ROOT/benchmark_data/RawVLA-Bench/data/libero/raw"

Each child config uses a relative extends entry. The training loader merges the base first and the selected backbone config second; command-line dotlist overrides remain highest priority. The VLA backbone is frozen and RAW-VLA is initialized from scratch.

The main public runners derive paths from the checkout and accept overrides via environment variables:

PY=python bash experiments/simulation/libero/scripts/run_libero_bench.sh

DATA_ROOT=/path/to/data \
CKPT_ROOT=/path/to/checkpoints \
bash scripts/evaluation/run_robotwin2_starvla_rgb_eval.sh one adjust_bottle

Run scripts with --help where supported and start with one task/episode before launching full benchmark matrices.

These synthetic studies vary image-processing conditions such as exposure, sensor noise, bit depth, color response, and tone. They cover both LIBERO and RoboTwin 2.0, including the FastWAM baseline, and are separate from the simulator-lighting tests in RawVLA-Bench. See the ISP perturbation guide for environments, settings, and evaluation commands.

Physical-robot entry points are documented in the real-robot guide. LIBERO and RoboTwin 2.0 are simulator integrations.

If you use RawVLA or RawVLA-Bench, please cite the paper:

@article{liu2026rawvla,
  title={RawVLA: Embodied Neural Image Signal Processor for Robotic Manipulation},
  author={Liu, Shuhong and Zhou, Heng and Qian, Lingfeng and Fang, Yuhao and Hou, Xianbao and Zhou, Qianyu and Gu, Lin and Sui, Wei and Yang, Jianfei and Cui, Ziteng},
  journal={arXiv preprint arXiv:2609.37530},
  year={2026},
  url={https://arxiv.org/abs/2609.37530}
}