SEPA-Eval

Self-Evolving Physical AI Evaluation

SEPA-Eval turns a static VLA benchmark into a living one. It records rollout failures, clusters them, mutates the worst-case scenarios into harder variants, validates them through a critic ensemble, and automatically promotes them into the active benchmark distribution — closing the loop between evaluation and continual learning.

Architecture · Installation · Quick Start · CLI Reference · Demo · Paper

---

The Problem with Static Benchmarks

Problem	What Happens
Saturation	Frontier models exceed 95%+ SR on LIBERO; the benchmark stops discriminating
Misalignment	Binary success rate misses fragile success, unsafe contact, and brittle grasp recovery
Silent drift	Scores look meaningful long after they stop predicting real deployment behavior

SEPA-Eval solves all three by making the benchmark itself a first-class output of the evaluation process.

---

Key Capabilities

• Structured trace recording — every episode emits observations, actions, contact events, success labels, and detected failure moments to a crash-safe SQLite + msgpack store
• Failure mining — heuristic detectors (timeout, out-of-reach, grasp, contact dynamics, recovery, pose, distractor, language) feed into DBSCAN clustering over compact trace embeddings
• Scenario mutation — pose perturbation, material swap, distractor injection, instruction paraphrase, and horizon extension operators generate harder task candidates from failure seeds
• Critic ensemble — semantic critic (VLM judge), safety critic (force/collision heuristics), and robustness critic score each candidate independently
• Promotion pipeline — async gate execution with configurable timeouts; evidence recorded per task; human review queue for borderline decisions
• Hard-case export — failed episodes exported as jsonl or LeRobot-style parquet datasets, ready for AlphaBrain's continual learning pipeline
• Self-evolution loop — a single orchestrator command runs the full Evaluate → Diagnose → Generate → Validate → Promote → Report cycle

---

Architecture

The closed-loop pipeline runs six stages on every evolution cycle:

1. Evaluate — run VLA policies on the current benchmark distribution (LIBERO / RoboCasa) via AlphaBrain's WebSocket eval harness; record structured rollout traces
2. Diagnose — classify failures by type and cluster traces with DBSCAN to extract reproducible failure seeds
3. Generate — apply mutation operators to failure seeds to produce harder task candidates
4. Validate — score candidates through the critic ensemble; defer on timeout rather than hard-fail
5. Promote — admit candidates that clear all gates into the live benchmark distribution; log evidence per task
6. Report — surface capability frontiers, saturation signals, and model comparison across evolved benchmark slices

Package Structure

AlphaBrain/sepa_eval/
  hooks/          BenchmarkAdapter protocol + LiberoHook, RobocasaHook
  memory/         EvalMemory (SQLite WAL + msgpack trace store), schema
  mining/         FailureStepDetector, DBSCAN clustering, seed extraction, LLM summarizer
  mutation/       PosePerturbation, MaterialSwap, DistractorAdd, InstructionParaphrase, HorizonExtension
  critics/        Semantic critic (VLM), safety critic, robustness critic
  promotion/      Validation pipeline, promotion gates, human review queue
  orchestrator/   Self-evolution loop
  exporter/       Hard-case dataset export (jsonl + LeRobot parquet)
  registry/       models.yaml-backed model registry + DB sync
  reporting/      Markdown report generation, task-model heatmaps
  configs/        mutation.yaml, orchestrator defaults, model configs
  tests/          Unit and integration tests

---

Installation

SEPA-Eval package only

The sepa_eval package lives inside AlphaBrain/. Install from that subdirectory.

python -m venv .venv && source .venv/bin/activate
pip install -e ./AlphaBrain
pip install -e "./AlphaBrain[dev]"   # adds black, ruff, pre-commit

All runtime dependencies (msgpack, scikit-learn, jinja2, pyyaml, openai, requests, pyarrow, Pillow, numpy) are declared in pyproject.toml and installed automatically. After installation, the sepa-eval CLI is available directly on your PATH.

Optional extra	Required for
openai	Instruction paraphrase mutation + GPT-4o-mini fallback critic
pyarrow	LeRobot-style parquet export (graceful JSONL fallback if absent)
pytest	Running the test suite

Full benchmark setup (LIBERO + RoboCasa)

Running SEPA-Eval against real simulators requires three separate Python environments and several dataset downloads. See SETUP.md for the complete step-by-step guide covering:

• AlphaBrain conda environment
• LIBERO simulator installation and data preparation
• RoboCasa Tabletop environment (robosuite 1.5.1 + tabletop fork + assets)
• RoboCasa 365 environment
• .env variable reference with correct path conventions
• Verification commands and common pitfalls

---

Quick Start

All commands run from the AlphaBrain/ directory.

All commands use the sepa-eval CLI installed with the package, or equivalently python -m sepa_eval.

1. Check memory status

sepa-eval --memory-dir ./eval_memory status

2. Register / sync models

sepa-eval --memory-dir ./eval_memory sync-models

3. Run the evolution loop

sepa-eval --memory-dir ./eval_memory run

4. Generate a Markdown report

sepa-eval --memory-dir ./eval_memory report --output ./eval_memory/report.md

5. Review pending promotion candidates

sepa-eval --memory-dir ./eval_memory review list
sepa-eval --memory-dir ./eval_memory review approve <task_id>

---

CLI Reference

Command	Description
run	Run the full closed-loop evolution cycle
eval	Register a model and run CI-style evaluation checks
promote	Run the promotion pipeline on pending candidates
report	Generate a Markdown report with heatmaps
export-hard-cases	Export failed episodes as a training dataset
diff	Compare task-level success rates between two models
sync-models	Sync configs/models.yaml into the database
prune	Remove old trace files while preserving DB rows
review list	List candidates awaiting human review
review approve	Promote a candidate manually
status	Show memory statistics and evolution metrics

Default storage root: SEPA_MEMORY_DIR env var, or ./eval_memory/.

---

Wiring into an AlphaBrain Eval Run

from sepa_eval.hooks.libero_trace_hook import run_libero_episode_with_trace
from sepa_eval.memory.eval_memory import EvalMemory
from sepa_eval.memory.schema import TraceIdentity

memory = EvalMemory(
    db_path="../eval_memory/eval.db",
    memory_dir="../eval_memory/traces",
)

trace = run_libero_episode_with_trace(
    env=libero_env,
    policy_fn=policy_fn,
    memory=memory,
    identity=TraceIdentity(
        trace_id="trace-001",
        eval_run_id="run-001",
        benchmark="libero_spatial",
        task_id="pick_red_cup",
        task_instruction="pick up the red cup",
        model_id="alphabrain-v2",
        model_version="v2.1",
    ),
)

RoboCasa: replace libero_trace_hook with robocasa_trace_hook and run_robocasa_episode_with_trace.

---

Customer Demo Workflow

A deterministic, simulator-free walkthrough using synthetic traces.

1. Generate synthetic traces

cd AlphaBrain
python ../demo/generate_libero_traces.py --output-dir ../demo/demo_eval_memory

2. Run the guided demo

cd demo
./run_customer_demo.sh               # interactive, pauses between stages
./run_customer_demo.sh --no-pause    # fully automatic
./run_customer_demo.sh --regenerate  # rebuild traces first

3. Open the HTML dashboard

python demo/render_report_html.py \
  --input  demo/output/report.md \
  --output demo/output/report.html
open demo/output/report.html

Artifacts produced: demo/output/report.md, demo/output/report.html, demo/demo_eval_memory/ (SQLite + msgpack traces).

---

Development and Testing

# Run all SEPA-Eval tests (from repo root)
pytest AlphaBrain/sepa_eval/tests -v

# Lint
ruff check AlphaBrain/sepa_eval/
black --check AlphaBrain/

The test suite has 86 tests (1 intentionally skipped — Phase 5 live-simulator e2e). Coverage includes: memory and trace persistence, critics, mutation operators, failure classification and clustering, promotion pipeline, exporter, registry, orchestrator evolution loop, reporting and heatmaps, LIBERO and RoboCasa benchmark hooks, and integration-level flows. Production code passes ruff with zero violations.

---

Current Status

Implemented and test-covered (86 tests, ruff clean):

• Trace hooks and benchmark adapters (LIBERO, RoboCasa)
• Persistent memory (SQLite WAL + msgpack) with crash-safe writes
• Failure classification, clustering, and seed extraction
• Full mutation operator suite (5 operators)
• Promotion gates + async pipeline with evidence recording
• Orchestrator loop, metrics output, and review CLI
• Reporting with task×model heatmaps, hard-case export, and model registry
• sepa-eval CLI entry point; all runtime deps declared in pyproject.toml

Pending / in progress:

• /judge endpoint wiring in server_policy.py (needed for local VLM critic)
• Long-term storage migration beyond SQLite (ANN indexing for large trace volumes)
• Simulator-specific validation (LIBERO state replay, RoboCasa material override fidelity)
• Full end-to-end evolution loop verified against a live simulator

See TODOS.md for the detailed implementation checklist.

---

Citation

If you use SEPA-Eval in your research, please cite:

@article{alchedata2026sepa,
  title   = {Self-Evolving Agents for Physical AI Evaluation},
  author  = {Alchedata},
  year    = {2026},
  note    = {White paper. \url{https://github.com/alchedata/sepa-eval}}
}

---

Repository Layout

.
├── AlphaBrain/              AlphaBrain VLA framework + sepa_eval package
│   └── sepa_eval/           SEPA-Eval implementation
├── demo/                    Synthetic trace generator and HTML dashboard renderer
├── paper/                   ACM acmart white paper source
├── assets/                  Diagrams and logos
├── PRD_SEPA_VLA_Eval.md     Product and system specification
└── TODOS.md                 Implementation status tracker

---

License

SEPA-Eval is released under the MIT License.

---

Alchedata — DATA INFRA 2.0 for Physical AI

Closed-loop eval → data → better checkpoints, automatically.