UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models

⚙️ Setup

1) Create and activate environment

conda create -n unilact python=3.10 -y
conda activate unilact

2) Clone the repo and install dependencies

git clone https://github.com/manishgovind/uniact-vla.git
cd UniLACT
pip install -r requirements.txt

3) (Optional) Set project root

export PROJECT_UNILACT_ROOT=/path/to/UniLACT

---

🚀 Training (3 Stages)

UniLACT training consists of three stages:

1. Stage 1: Unified latent action learning (UniLARN)
2. Stage 2: Unified latent pretraining
3. Stage 3: Action fine-tuning

Training is driven by YAML configs under unilact/configs/train/ and configs under unilarn/configs/train.

---

Stage 1 — UniLARN (Unified Latent Action Learning)

# (Update this command to your UniLARN entrypoint/config if different)
cd ${PROJECT_UNILACT_ROOT}/unilarn
accelerate launch --main_process_port <master_port> train_unilarn.py --config_path "${PROJECT_UNILACT_ROOT}/unilarn/configs/train/train_unilarn_on_calvin.yaml"

---

Stage 2 — Unified latent Pretraining

# (Update this command to your pretraining config if different)
cd ${PROJECT_UNILACT_ROOT}/unilact/train
accelerate launch --main_process_port <master_port>  train_unilact.py --config_path "${PROJECT_UNILACT_ROOT}/unilact/configs/train/pretrain_unilact_on_calvin.yaml"

---

Stage 3 — Fine-tuning

cd ${PROJECT_UNILACT_ROOT}/unilact/train
accelerate launch --main_process_port <master_port> train_unilact.py   --config_path "${PROJECT_UNILACT_ROOT}/unilact/configs/train/finetune_unilact_on_calvin.yaml"

---

Evaluation on CALVIN (ABC→D) Benchmark

Install the CALVIN benchmark in the same conda environment (unilact) by following the official CALVIN repository instructions.

conda activate unilact
export PROJECT_UNILACT_ROOT=/path/to/UniLACT
cd ${PROJECT_UNILACT_ROOT}/scripts
bash evaluate_unilact_on_calvin.sh

---

⏳ To-Do

• Training Data preparation
• Support for OXE-pretraining
• Release pretrained and finetuned model checkpoints

🙏 Acknowledgements

This project builds on top of Moto, CALVIN. We thank the authors for the ir open-sourced work.

📝 Citation

If you find our work useful, please cite:

@article{govind2026unilactdepthawarergblatent,
  title= {UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models},
  author= {Manish Kumar Govind and Dominick Reilly and Pu Wang and Srijan Das},
  journal={arXiv preprint arXiv:2602.20231},
  year={2026}
}