T-WLA

Visual planning and tactile correction share a history-aware visual memory. Input thumbnails are illustrative.

How can a robot respond to changing contact before its next visual update?

Tactile feedback reveals local contact changes, but the appropriate response also depends on the task, scene, and ongoing motion. T-WLA is a world–language–action architecture that connects a visual policy and a lightweight tactile editor through shared visual memory.

• Plan with visual history. At each RGB update, retrieve past visual information and combine it with the current observation, instruction, state, and field of view to produce visual memory and a base action chunk.
• Correct asynchronously with touch. Between RGB updates, combine recent tactile context with the cached visual memory and base actions to predict residual corrections, without rerunning the visual branch.
• Learn from future interaction. Future-image and future-pressure prediction complement action supervision during training. Neither prediction head runs during action inference.

corrected actions = original base actions + masked tactile residuals

Each tactile update revises only valid, unexecuted actions relative to the original plan. A new RGB update replaces the visual memory and base chunk.

The paper evaluates tactile-conditioned hand-motion prediction on EgoTouch and Tachin, visual-only prediction on GigaHands, OakInk2, and TACO, and four real-robot tasks: egg transfer, single-sheet pickup, paper-cup nesting, and paper-bag dragging. Reported results include a 30% reduction in EgoTouch finger-joint MPJPE relative to the strongest reported baseline and 50.0% average robot success with 30 demonstrations per task. See the project page for qualitative comparisons and robot videos.

This repository provides the visual foundation model, tactile encoder pretraining, tactile-editor training, predictive objectives, baseline implementations, and robot hardware tools.

Use Python ≥ 3.10 and a compatible CUDA/PyTorch environment for training. The tactile package requires PyTorch ≥ 2.4. From the repository root:

git clone https://github.com/kaichen-z/TWLA.git
cd TWLA
python -m pip install -r T-WLA_foundation_model/requirements.txt
python -m pip install -e './tactile_editor[test]'

For tactile-only development, the editable package is sufficient. Its unit tests run on CPU without downloading model weights:

python -m pytest tactile_editor/tests

Download the visual policy, RGB world head, and H8 encoder from the repository root using the Hugging Face CLI:

hf download anon-submission-0518/T-WLA-FD T-WLA-FD.pt \
  --local-dir checkpoints
hf download anon-submission-0518/T-WLA-WM T-WLA-WM.pt \
  --local-dir checkpoints
hf download MIT-Media-Lab/egotouch-tactile-encoder-p1-v1 \
  mano-aligned-v1/h8/best.pt mano-aligned-v1/h8/resolved_config.json \
  --local-dir checkpoints/egotouch-p1

The H8 release provides a formal P1 encoder checkpoint, distinct from the same repository's standalone safetensors exports. Match its history length, coordinate convention, and normalization to your cache; see checkpoint compatibility.

The encoder initializes tactile feature extraction; it is not a trained residual editor. Train the editor with the workflow below. No final W3 editor download is currently linked in this release.

Set external data, output, and pretrained-asset locations:

export VLA_DATA=/path/to/prepared_data
export VLA_RUNS=/path/to/training_outputs
export VLA_WEIGHTS=/path/to/third_party_weights

Use absolute paths for external inputs. Downloaded weights do not replace prepared data or frozen visual-prediction caches. Future-image supervision requires valid jpeg_root frame stores; the raw-video fallback reuses the current frame and is not suitable for future-image training. Cache formats and additional asset requirements are described in the tactile guide and reproduction notes.

The released workflow uses staged optimization:

During editor training, the visual controller supplies frozen cached memory and base actions. The RGB predictor stays frozen; its tactile adapter and the pressure predictor are trained.

Edit the visual configuration, then run:

cd T-WLA_foundation_model
torchrun --nproc_per_node=4 train.py --config configs/train.json \
  --init-world-from ../checkpoints/T-WLA-WM.pt
cd ..

--init-world-from initializes only the RGB world head. To initialize the visual controller as well, add --init-from /path/to/visual_checkpoint.pt. Use --resume /path/to/run_checkpoint.pt to continue an existing run. Adjust the GPU count and training budget to your setup.

Use a compatible pretrained H8 checkpoint from Pretrained Weights, or train your own. For pretraining, edit cache_root, output_dir, and the OpenTouch initialization checkpoint in the encoder configuration, then run from the repository root:

python -m egotouch_formal.train \
  --config tactile_editor/configs/encoder_p1_h8.json

The resulting formal P1 best.pt is used by the EgoTouch editor's --encoder-checkpoint argument. Native Tachin uses its own sensor encoder and does not take this argument.

With a prepared EgoTouch cache and matching checkpoints, run from the repository root:

python tactile_editor/scripts/train_paired30.py \
  --root /path/to/paired30_cache --variant w3 \
  --encoder-checkpoint /path/to/compatible_encoder_h8/best.pt \
  --world-checkpoint checkpoints/T-WLA-WM.pt \
  --world-config /path/to/matching_world_config.json \
  --lambda-rgb 0.1 --lambda-tactile 0.0001 \
  --output-dir /path/to/runs/w3 --validation-only

The auxiliary coefficients are example settings. Use --smoke-steps 2 for a short, data-dependent check. Keep --validation-only during model selection; omitting it also evaluates the selected checkpoint on the test cache.

python tactile_editor/scripts/train_tachin.py \
  --cache /path/to/tachin_cache --model twla \
  --world-checkpoint checkpoints/T-WLA-WM.pt \
  --world-config /path/to/matching_world_config.json \
  --lambda-rgb 0.1 --lambda-tactile 0.0001 \
  --output /path/to/runs/tachin_w3

Run Python from T-WLA_foundation_model/ with a matching visual checkpoint and configuration:

from inference.policy import VLAPolicy, to_canvas, fov_from_intrinsics

policy = VLAPolicy(
    checkpoint="../checkpoints/T-WLA-FD.pt",
    instruction="Left hand: None. Right hand: pick up the cup.",
)

# Supply an RGB image, state/mask, camera intrinsics, and episode identifier.
base_actions, visual_memory = policy.act(
    to_canvas(rgb), state, state_mask,
    fov_from_intrinsics(fx, fy, cx, cy), episode_id,
    steps=16, return_mf=True,
)

The supplied configuration returns a 16 × 192 action chunk and 8 × 960 visual memory. state and state_mask each have 212 dimensions; actions are returned in physical coordinates. Visual history resets when the episode identifier changes or policy.reset() is called.

For tactile correction, policy_context(...) converts the visual output into the editor's normalized input space using explicit training statistics and action-validity masks. The intended execution cycle is:

1. RGB update: generate and cache visual memory and the base chunk.
2. Tactile update: encode recent touch and predict a masked residual for the remaining actions, using the original base chunk as reference.
3. Next RGB update: replace the cached memory and chunk; tactile updates never modify the visual memory bank.

Apply robot-specific action conversion, limits, and safety checks before execution. Calibration, teleoperation, recording, and device interfaces are in robot_hardware/; the library interfaces are not a complete asynchronous robot scheduler.

For visual-policy diagnostics:

cd T-WLA_foundation_model
torchrun --nproc_per_node=2 eval_test.py --config configs/train.json \
  --checkpoint ../checkpoints/T-WLA-FD.pt --out test.json
cd ..

Tactile evaluation measures wrist translation error, wrist rotation geodesic error, and wrist-relative finger-joint MPJPE. Paper comparisons use a common prediction horizon, per-sequence averages, and an equal-weight average across EgoTouch's five environments.

The physical metric utilities implement this strict horizon and aggregation protocol. The default EgoTouch runner and visual evaluator report separate diagnostic reductions, so their output should not be substituted directly for paper-table values. See evaluation and timing notes before reproducing comparisons.

Training saves artifacts to the output directory selected in the configuration or command:

Retain the checkpoint, configuration, and matching cache/statistics identity together. A --smoke-steps run writes a smoke report and is not a completed training run.

The core implementation includes visual planning, tactile encoder pretraining, frozen-visual editor training, predictive losses, baseline editors, evaluation utilities, and hardware tools. Visual-policy, RGB-world-head, and tactile-encoder weights are linked above.

• Data: provide prepared datasets, caches, split definitions, and normalization statistics; raw-recording preprocessing is external.
• Editor weights: the training pipeline produces editor checkpoints; this release does not yet link a final pretrained W3 editor.
• Extensions: joint visual–tactile fine-tuning is a proposed extension, not part of the released offline training workflow. Robot adaptation and a complete asynchronous deployment/timing pipeline require additional integration.

Implementation-specific details are collected in Reproduction Notes.

This work builds on SmolVLA/LeRobot, SmolVLM2, EgoTouch/OpenTouch, SANA, DC-AE, VITRA, and MemoryVLA. Please follow each upstream model and dataset's access and licensing terms. See dependencies and attribution for details.

Project Page · Method Overview · Robot Demonstrations