RoboSTAR: Next-Scale Autoregressive Sign Language Motion Translation for Humanoid Robots

Yujia Zeng^* Chensheng Peng^* Yuxin Chen^* Alex Shao Nathan Jew Masayoshi Tomizuka
UC Berkeley
^*Equal contribution

RoboSTAR translates English text or audio into continuous American Sign Language (ASL) motion.
This release contains the text-to-sign-motion component.

Source_GT_SOKE_Ours.mp4

Installation

git clone https://github.com/zyjOrz/RoboSTAR.git
cd RoboSTAR
conda create -n robostar python=3.10 -y
conda activate robostar
pip install -e .

Inference

Predicted-duration inference:

python -m robostar.infer \
  --model Ivystream/RoboSTAR \
  --text "A person explains the plan." \
  --length-mode predicted \
  --output-dir outputs/predicted

For a stable demonstration duration, specify seconds explicitly:

python -m robostar.infer \
  --model Ivystream/RoboSTAR \
  --text "A person explains the plan." \
  --length-mode seconds \
  --length-value 6.0 \
  --output-dir outputs/six_seconds

Other supported modes are tokens and frames.

Data preparation

python -m robostar.prepare_data \
  --train-manifest data/train_raw.jsonl \
  --val-manifest data/val_raw.jsonl \
  --test-manifest data/test_raw.jsonl \
  --output data/prepared

Training

1. Train the FSQ tokenizer

torchrun --standalone --nproc_per_node=8 -m robostar.train_tokenizer \
  --config configs/tokenizer.yaml \
  --prepared-root data/prepared \
  --output experiments/robostar_tokenizer

2. Export motion tokens

torchrun --standalone --nproc_per_node=8 -m robostar.export_tokens \
  --model experiments/robostar_tokenizer/best.pt \
  --prepared-root data/prepared \
  --output data/tokens

3. Build the optional retrieval memory

The isolated-word source used in our experiments is akasheroor/American-Sign-Language-Dataset.

python -m robostar.retrieval build \
  --word-token-jsonl data/word_tokens/train_source_tokens.jsonl \
  --output data/retrieval/word2code.json

4. Build coarse-to-fine caches

python -m robostar.build_cache \
  --token-root data/tokens \
  --tokenizer-model experiments/robostar_tokenizer/best.pt \
  --base-model google/mt5-large \
  --retrieval data/retrieval/word2code.json \
  --output data/cache

5. Train RoboSTAR

torchrun --standalone --nproc_per_node=8 -m robostar.train_generator \
  --config configs/robostar_mt5_large.yaml \
  --cache-root data/cache \
  --output experiments/robostar_mt5_large

Evaluation

python -m robostar.evaluate \
  --predictions outputs/predictions \
  --manifest data/prepared/test.jsonl

🔗 Citation

If you find our work useful, please consider citing:

@article{robostar,
  title   = {RoboSTAR: ...},
  author  = {...},
  journal = {...},
  year    = {2026}
}

🙏 Acknowledgements

• SOKE: Our work follows SOKE in adopting its motion representation and sign retrieval formulation.

• How2Sign Dataset: Our work uses the How2Sign dataset for training and evaluation.

• HandMDM: Our human-mesh rendering and visualization pipeline is inspired by HandMDM.