• 2026-07-28: We released the TriWorldBench code and valid set with 100 episodes and testset with 500 episodes.
• 2026-07-28: We launched the TriWorldBench Challenge.

• Overview
• Getting Started
  • Environment Setup
  • Dataset Preparation
    • Quick Test Dataset
    • Prepare Your Own Test Dataset
  • Inference
  • Evaluation
• Leaderboard & Submission
• Acknowledgement

TriWorldBench is a unified benchmark designed to systematically evaluate embodied world models from three synchronized robot views: head, left wrist, and right wrist. It reports 19 evaluation signals across six dimensions: tri-view consistency, task alignment, physical and 3D coherence, motion quality, temporal consistency, and visual quality. Tri-view consistency is the central dimension. It combines reference-anchored signals, which compare every generated camera with its corresponding view from a consistent ground-truth triplet, with direct semantic judgments of head-wrist and joint tri-view agreement. The remaining dimensions test whether the prediction follows the task, preserves plausible interaction and geometry, exhibits the expected motion in each camera, remains stable over time, and produces usable images. Trajectory-derived STATE annotations connect these evaluations by identifying the action phase, active arm, and expected moving views. TriWorldBench retains per-view, head-wrist pair, and joint tri-view results, additionally reports visual-quality scores grounded by task and consistency performance, and names its final overall aggregate TWB-Score. Together, these dimensions track progress toward embodied world models that generate one consistent robot world across multiple cameras.

a. Create the conda virtual environment

conda create -n triworldbench python=3.10.0 -y
conda activate triworldbench

cd Triworld
pip install --upgrade pip setuptools wheel
pip install -r requirements.txt
pip install --no-deps -r requirements_no_deps.txt

System dependencies: ffmpeg, ffprobe, CUDA-capable GPU(s).

b. Download model weights

Follow DOWNLOAD_LINKS.md and place all checkpoints under:

./weights/

Set weights_root in config/config.yaml to this directory.

TORCH_HOME is set automatically to {weights_root}/torch_home during evaluation (ResNet34 for photometric_smoothness).

We provide a 100-episode validation set derived from RoboTwin2.0, the corresponding gt_dataset, and processed STATE annotations. Users only need to prepare inference results on the validation set, then update a small amount of configuration before running evaluation.

The validation dataset is released at TriWorldBench/Dataset. Download val_dataset from this Hugging Face dataset page.

After extraction, the validation bundle contains:

val_dataset/
├── test_dataset/
│   ├── data/
│   ├── first_frame/
│   └── instructions/
├── STATE/
│   └── episode*.json
└── gt_dataset/
    └── episode*/

gt_dataset and STATE already use the format required by this evaluation project. The matching VQA question annotations for this validation set are included in this repository under metrics/VQA/qa_val/; they are used by the VQA metric to score task and view-specific question answering. Users only need to run inference on the inputs from test_dataset, then point the evaluation config to the resulting outputs for a quick validation run. See the Inference and Evaluation sections for how to use these paths.

If you want to build your own test dataset, you can generate or download RoboTwin2.0 data from TianxingChen/RoboTwin2.0. See test_data/data_readme.md for how to prepare gt_dataset, STATE, and matching VQA QA files for evaluation.

a. Run inference and place generated videos

Use the inputs from the quick val_dataset/test_dataset bundle, or from your own prepared test dataset, to run your model inference. Place per-method source data under input/<your_method>/:

input/<your_method>/
└── episodeN/
    ├── head.mp4
    ├── left.mp4
    └── right.mp4

Example: input/<your_method>/episode1/{head,left,right}.mp4

b. Preprocess inference outputs

Convert raw MP4 inputs into GT-aligned frame trees.

Before preprocessing, place the corresponding gt_dataset under the default test_data/gt_dataset, or set gt_dataset in config/config.yaml to your custom GT path.

cd Triworld
./run_preprocess.sh <your_method>
# optional: --limit 10 --workers 8 --overwrite --dry-run

Output location and naming:

generate/<your_method>_<input_ts>_<run_ts>/

• input_ts — batch id for the same inference input (reuse with --input-ts YYYYMMDD_HHMMSS)
• run_ts — timestamp of this preprocess run

A pointer file is written:

generate/<your_method>_<input_ts>_latest.path   # text file containing latest run dir name

Set eval_input in config to this directory (or the _latest alias).

Edit config/config.yaml:

eval_input: generate/<your_method>_latest  # preprocessed data
metrics: all                                       # or [0, 3, 12]
output_dir: output
parallel_per_gpu: 4
metric_parallel_per_gpu:
  background_consistency: 1
  semantic_alignment: 1
  subject_consistency: 2
  trajectory_accuracy: 4
gpu_list: "0,1,2,3,4,5,6,7"
python: python
weights_root: weights
gt_dataset: test_data/gt_dataset
STATE: test_data/STATE
hyperparameters: config/hyperparameters.yaml
vlm_reference_db: test_data/vlm_data_base
vlm_max_new_tokens: 512
episode_weights: false    # false=equal mean; true=frame-bucket weights
resume: true               # skip completed episodes / finished metrics

python: python assumes that commands are launched after conda activate triworldbench; you can also set it to the absolute Python path of your TriWorldBench environment.

Configurable fields:

Metric index reference:

conda activate triworldbench
cd Triworld
./run_eval.sh
# dry run: ./run_eval.sh --dry-run
# custom config: CONFIG=config/config.yaml ./run_eval.sh

Final CSV results are written under:

output/<your_method>_<input_ts>_<run_ts>/
└── final_results/
    ├── episode_metrics.csv
    └── summary_metrics.csv

episode_metrics.csv reports each metric score per episode. summary_metrics.csv reports aggregate scores for each metric across the evaluated episodes.

Reference resource usage for val100:

On our 100-episode validation run with NVIDIA A100 40GB x 8 GPUs, the full evaluation finished within 2 hours. Peak GPU memory was about 160 GB in total, with the highest single-GPU peak around 23 GB. Peak system memory was about 170 GB. GPU utilization may reach 100% during model-heavy metrics, so adjust gpu_list, parallel_per_gpu, and metric_parallel_per_gpu if running on fewer or smaller GPUs.

Submit your results on the TriWorldBench leaderboard.

We acknowledge RoboTwin 2.0 for providing the dataset and simulation platform support that enables embodied task evaluation.

For video quality evaluation, TriWorldBench also benefits from the open-source evaluation community, including WorldArena, VBench, EWMBench, WorldScore, EvalCrafter, and JEDI.