🦾 VLA Evaluation Framework

This repository serves as the official artifact for the paper A Systematic Evaluation of Vision Language Action Models Across Workspace and Data Regimes. It provides a comprehensive suite of tools for dataset generation, model fine-tuning, and simulation-based evaluation of Vision-Language-Action (VLA) models.

The goal of this repository is to ensure full reproducibility of the experiments reported in the paper. The codebase follows a client-server architecture to decouple simulation from model inference, allowing for flexible evaluation of heavy VLA models.

---

High Camera View
high_1.mp4	high_2.mp4	high_3.mp4
Front Camera View
front_1.mp4	front_2.mp4	front_3.mp4

📂 Repository Structure

The repository allows for versioned experimentation. The folders v1 and v2 correspond to the specific experimental iterations discussed in the paper. Both versions have identical directory structures, differing only in configuration choices and model checkpoints.

Root Directory

.
├── v1/                          # Version 1 experiments (See details below)
├── v2/                          # Version 2 experiments (Identical structure to v1)
├── Isaac-GR00T/                 # Simulator Submodule (pinned to paper version)
├── groot_service.py             # Server: Main entrypoint for the VLA model
├── install.md                   # Setup: Environment & dependency instructions
├── datasets/                    # Data
│   └── v1_test_dataset/         # Example evaluation dataset (not included in the repository)
├── models/                      # Fine-tuned Weights
│   └── v1_gr00t/                # Example fine-tuned model (not included in the repository)
└── README.md

v1 (and v2) Directory Layout

Below is the detailed structure for the experiment folders.

v1
├── config/                      # Simulation configurations
│   ├── cameras.py               # Camera intrinsics/extrinsics
│   ├── trossen_robots.py        # Robot specifications
│   └── ...
├── data_creation/               # Dataset generation pipeline
│   ├── create_data.sh           # ENTRYPOINT: Script to generate datasets
│   ├── multi_gpu_controller.sh  # Parallel generation for large scale data
│   └── ...
├── inference/                   # Evaluation pipeline
│   ├── run_inference.py         # ENTRYPOINT: Run single-model evaluation
│   ├── inference_service_adapters/
│   │   ├── groot.py             # Adapter for GR00T model
│   │   ├── pi0.py               # Adapter for Pi0 model
│   │   └── sample.py            # Template for adding new models
│   └── ...
├── usd/                         # Universal Scene Description assets
└── utils/                       # Shared utility scripts

---

🛠️ Installation & Setup

1. Clone & Submodules

This repository relies on Isaac-GR00T as a core simulation backend. It is included as a Git submodule and pinned to the exact commit used in our experiments to guarantee deterministic behavior.

# Clone the repository
git clone git@github.com:purwar-lab/vla.git
cd vla

# Initialize and update submodules
git submodule update --init --recursive

2. Python Environments

⚠️ Important: We use separate environments for simulation and inference to avoid dependency conflicts (e.g., Isaac Sim vs. PyTorch versions).

Please refer to install.md for exact environment specifications. Do not deviate from the package versions listed there, as this may break physics determinism or model loading.

---

🧱 Dataset Creation

We generate synthetic datasets for "Pick and Place" and other manipulation tasks.

• Navigate to: v1/data_creation/
• See v1/data_creation/README.md for generation scripts and configuration details.

---

🎯 Model Fine-Tuning

Training code is decoupled by model architecture to handle specific dependency requirements.

• Locate your model: Go to the specific model folder (e.g., v1/models/gr00t).
• Follow local instructions: Each model folder contains a README with training commands.

---

🚀 Inference & Evaluation

We utilize a Client-Server architecture for evaluation. This allows the simulator (Client) to run in a lightweight environment while the VLA model (Server) runs in a heavy GPU environment.

Step 1: Start the Inference Server

This service loads the model weights and listens for observations from the simulator.

python groot_service.py \
  --model_path models/v1_gr00t/checkpoint-70000 \
  --port 4545

• --model_path: Path to your fine-tuned checkpoint.
• --port: The socket port for communication.

Step 2: Run the Simulation Client

In a separate terminal (and potentially a separate environment), launch the evaluator.

python v1/inference/run_inference.py \
  --inference_service_name groot \
  --inference_server_port 4545 \
  --hdf5_file_or_folder_path datasets/v1_test_dataset \
  --num_episodes 100 \
  --enable_cameras \
  --headless | tee v1_infer.log

Argument	Description
inference_service_name	Matches the adapter name (e.g., groot, pi0).
hdf5_file_or_folder_path	The dataset containing initial states/prompts.
enable_cameras	Renders visual observations for the VLA.
headless	Runs without GUI.

---

🔌 Adding New Models

The pipeline is model-agnostic. To benchmark a new architecture:

1. Create an Adapter: Write a Python wrapper that standardizes inputs/outputs.
2. Save: Place it in v1/inference/inference_service_adapters/.
3. Reference: Use sample.py as a template or look at groot.py for a complex example.

Bulk Evaluation: To evaluate multiple models sequentially, use the multi-model runner: "" python v1/inference/run_multi_model_inference.py ""

---

📊 Outputs & Metrics

The evaluation script generates v1_infer.log that contains metrics containing:

• ✅ Episode Success Rates (SR)
• ⏱️ Runtime Statistics
• 📉 Task-specific failure modes

These metrics correspond directly to the tables in the paper.

---

📌 Notes

• Determinism: While we pin submodules, results may vary slightly based on GPU hardware (floating point non-associativity).
• Headless Mode: We strongly recommend headless mode for large-scale evaluation (100+ episodes) to reduce rendering overhead.