VTLA

Vision–Tactile–Language–Action Model for Open-World Generalization

English | 简体中文

Warning

VTLA is an experimental research code release. This repository does not currently publish a paper, dataset, pretrained checkpoint, benchmark result, or claim of end-to-end hardware validation.

VTLA extends LeRobot with a Piper-centered vision-and-touch stack. It records robot, RGB, and tactile observations with host timestamps and adds two complementary levels of tactile feedback:

1. Policy-level tactile perception routes tactile visualization images through dedicated encoders in ACT, Pi0, and Pi0.5.
2. Action-level tactile correction applies a bounded residual refiner to compatible state/force datasets and base-policy actions.

Architecture

flowchart LR
    RGB[Top and wrist RGB] --> Base[Base policy]
    TI[Tactile visualization] --> TE[Tactile image encoder]
    TE --> Base
    S[Robot state] --> Base
    Base --> A[Base action chunk]
    F[Scalar tactile force] --> R[Bounded residual refiner]
    S --> R
    A --> R
    R --> Safe[Clipping, rate limits, and robot command]

Loading

The repository also contains:

• a Piper follower and hand-guided Piper leader integration;
• Orbbec V2 RGB capture and a proprietary SightTac adapter;
• DreamTacVLA Stage 1/Stage 2 research code;
• an isolated Python 3.11 FTP-1 worker with sync, async, and FTP-1 RTC serving modes.

Current support

Capability	ACT	Pi0	Pi0.5	DreamTacVLA	FTP-1
Dedicated tactile image input	Yes	Yes	Yes	Native force-field path	Native tactile path
Bounded tactile residual refiner	Sync only	Sync or generic RTC	Sync or generic RTC	No	No
Piper-oriented workflow	Experimental	Experimental	Experimental	Required contract	Required contract
Released weights or reference metrics	No	No	No	No	No

FTP-1 RTC is implemented inside the FTP-1 worker and is not the same interface as LeRobot's generic RTC processor.

Installation boundary

VTLA is not published as a separate Python package. Clone this repository:

git clone https://github.com/CanyonChen/VTLA.git
cd VTLA

Then follow the official LeRobot source-install guide for Python, CUDA, system packages, and optional dependencies. Skip the guide's LeRobot clone step and run its source-install commands from this VTLA checkout.

Important

pip install lerobot installs the official upstream package and does not contain VTLA's Piper, tactile, DreamTacVLA, or FTP-1 changes.

Proprietary SightTac dependency

The SightTac SDK is proprietary and is not distributed in this repository because it is covered by a commercial confidentiality agreement. VTLA does not provide its version, binaries, internal API documentation, directory layout, or vendor installation procedure. Users must obtain lawful access directly from the vendor and configure it under the vendor's terms. Workflows that instantiate a SightTac device are unavailable when that dependency is absent.

Documentation

• Documentation index
• Getting started
• Architecture
• Hardware and safety
• Data contract
• Policies
• Workflows
• FTP-1
• Troubleshooting
• Limitations

For unchanged LeRobot datasets, robots, policies, and general concepts, use the upstream LeRobot documentation.

Known limitation: tactile refiner datasets

The Piper recorder keeps SightTac scalar force channels as independent dense features, while lerobot-train-tactile-refiner currently resolves force_keys from observation.state. A directly recorded Piper dataset therefore does not yet satisfy the refiner trainer's input contract without an explicit compatible dataset transformation. The documentation does not present this path as turnkey.

Upstream and licenses

This codebase is derived from LeRobot 0.6.1. See UPSTREAM.md for the relationship and THIRD_PARTY_NOTICES.md for third-party attribution. The repository is distributed under Apache-2.0; bundled third-party components remain subject to their own license files.

Contributions are described in CONTRIBUTING.md.