Control Dataset: VLA or IL?

A controlled dataset for testing whether finetuning turns VLA models into fancy imitation learners.

📄 Blog: VLA or IL? A Controlled Dataset for Testing Whether Finetuning Turns Your VLA into a Fancy Imitation Learner

Repository Structure

.
├── generate_control_dataset.py   # Generates 40 controlled BDDL variants (2x2 design)
├── render_examples.py            # Renders scene images for each variant
├── control_dataset/
│   ├── original_seen/            # Baseline: original position + seen prompt
│   ├── original_unseen/          # Tests language grounding: original position + unseen prompt
│   ├── shuffled_seen/            # Tests spatial generalization: shuffled position + seen prompt
│   └── shuffled_unseen/          # Tests both: shuffled position + unseen prompt
└── examples/
    ├── original_seen/
    ├── original_unseen/
    ├── shuffled_seen/
    └── shuffled_unseen/

Usage

Generate the dataset

python generate_control_dataset.py \
    --bddl_dir /path/to/libero/bddl_files/libero_object \
    --output_dir ./control_dataset \
    --seed 42