feat(benchmark): add replayable video relation evals

[fengweit]

1. This PR and proposed structure

This PR adds a replayable spatiotemporal video-relation evaluator with a strict public/private boundary:

video
  -> sampled observations
  -> canonical spatial facts
  -> sample-aware intervals
  -> public questions + private answers/evidence
  -> isolated candidate predictions
  -> exact report + annotated evidence viewer

Responsibilities are kept explicit:

contracts       models.py, utilities.py
teacher         video_adapter.py, yoloe_adapter.py, relations.py, intervals.py
generation      generation.py, observation_io.py, replay.py
release         bundles.py, ports.py
candidate       temporal_memory_answerer.py, candidate_worker.py
evaluation      runner.py, scoring.py
review          evidence_viewer.py
acceptance      demo.py, reproduce_reference.sh, test_*.py

Interviewer reproduction

From a PR checkout:

gh pr checkout 2989
./dimos/benchmark/spatiotemporal/reproduce_reference.sh

The script materializes LFS assets, installs dependencies, extracts YOLO-E weights, installs the prompt dependency when needed, runs pytest/Ruff/mypy, executes the real CPU demo, validates summary gates, and prints the evidence-viewer path.

After setup, the fast rerun is:

./dimos/benchmark/spatiotemporal/reproduce_reference.sh --skip-setup --skip-tests
open .artifacts/spatiotemporal-video-qa/evidence-viewer/index.html

The reviewer path is: reproduce → inspect evidence → verify public/private separation → verify deterministic replay → understand limitations.

Scope and interpretation

This is an evaluator contribution, not a new visual-QA model.

• YOLO-E outputs are teacher pseudo-labels, not ground truth.
• The included TemporalMemory candidate is timestamp-scripted and not visually grounded.
• Its score is explicitly labeled not_model_quality.
• Candidate execution runs in a subprocess whose DB, JSONL, structured logs, inputs, and results stay under an ephemeral candidate root.
• The evidence viewer exposes teacher defects rather than hiding them.

Reference-video defects are documented: the same robot changes tracker ID from 2 to 3, an oven/cabinet column is labeled refrigerator, and the robot is missed in several sampled frames.

Concrete generated examples and proof

The reference video produces real schema-v2 examples such as:

frame 0:   1 left-of 2
  public:  Did 1 ever appear left of 2?
  oracle:  true, evidence frame [0]

negative control:
  public:  Did 1 ever appear right of 2?
  oracle:  false, evidence schedule [0, 150, 300, 450, 600]

frames 0 → 300:
  public:  Did "1 left of 2" happen before "1 above 3"?
  oracle:  true, backed by the frame-0 and frame-300 intervals

The committed reproduction script now verifies and prints representative examples. Its proof gates check:

• 5 sampled frames → 8 observations → 3 facts → 3 intervals → 18 questions
• spatial balance: 3 true / 3 false
• temporal balance: 6 true / 6 false
• public question records contain no expected answer or evidence fields
• candidate summary explicitly reports visually_grounded=false and not_model_quality
• viewer exists with 18 oracle rows and 5 annotated frames
• logical SHA-256 and full summary SHA-256 are stable

This proves the trusted reference candidate's explicit-input boundary and evaluator repeatability, not an OS sandbox or production model accuracy. Untrusted candidates require a container/worker with repository and oracle paths unmounted.

Evidence viewer UI

The evaluator now generates a self-contained monitor/inspection surface rather than a raw HTML table. It shows:

• the full video → sample → track → relation → question → score pipeline;
• one concrete frame → fact → public test → private oracle proof chain;
• a four-gate robot-motion verdict covering linked relation evidence, strict temporal order, identity continuity, and detection coverage;
• a clickable relation timeline with evidence gaps left visible;
• five aligned annotated evidence frames with full-resolution lightbox;
• future-project mappings for locomotion/patrol, manipulation, long-horizon memory, and release regression gates; and
• searchable/filterable spatial and temporal oracle cases with evidence links.

For the reference video the UI reports the precise, non-overclaimed result: relation-order eval ready; motion claim needs review. All 3/3 intervals are oracle-referenced and strictly ordered, but the robot label spans IDs 2 and 3 and is detected in only 2/5 evidence frames.

2. Further directions

The evaluator can become a quality-gated self-improvement data engine:

new videos
  -> generated challenge episodes
  -> quality/disagreement gates
  -> failure clustering
  -> human review/correction
  -> training export
  -> candidate N+1
  -> frozen holdout evaluation

Proposed next components:

1. Episode admission reports for confidence, track continuity, label consistency, contradictions, evidence coverage, and replay determinism.
2. A three-set registry: frozen regression set, reviewed improvement set, and unreviewed challenge pool.
3. Failure clustering across spatial, temporal, identity, label, missed-detection, and long-horizon-memory errors.
4. Accept/correct/reject review workflow built on the evidence viewer.
5. Provenance-preserving training export for reviewed examples only.
6. Promotion policy requiring frozen-set improvement, no critical-family regression, and passing privacy/latency/resource gates.

Automatically generated episodes may enter the challenge pool immediately, but not a trusted benchmark or training set without quality gates.

3. Production path

Productionization is proposed in stages:

1. Repeatable local and CI evaluation: pin assets/dependencies/schema and publish summary, manifests, and viewer.
2. Versioned episode service: immutable source hashes, teacher provenance, idempotent generation, retention, and private-evidence access controls.
3. Quality-gated curation: automatic admission metrics, uncertainty review, and reviewer corrections stored separately from teacher output.
4. Continuous candidate evaluation: frozen holdouts plus a separate new-video challenge stream, with family accuracy, latency, memory, and regression tracking.
5. Controlled self-improvement: train from reviewed failures, compare on untouched holdouts, require approval, and support rollback.

Correctness and safety

• strict frozen Pydantic v2 contracts with extra="forbid"
• schema-v2 episode-scoped question IDs
• canonical inverse relations to avoid duplicate weighting
• relation-flip-safe negative controls
• zero-detection samples preserved in interval schedules
• temporal cases require every cross-interval comparison to be strict and unanimous
• exactly one private answer per public question
• candidate receives no bundle, oracle, fact, interval, or evidence records
• subprocess isolation prevents candidate DB/JSONL/structured-log escape
• symlink/path/hard-link protections and full encoded-video decode verification
• deterministic, escaped, failure-clean atomic evidence viewer

Verification

• 127 passed
• Ruff format: passed
• Ruff check: passed
• package mypy: passed (17 source files)
• independent read-only verification: PASS
• Greptile security review: PASS

Real macOS CPU demo, repeated with byte-identical summaries:

• 750 decoded frames / 5 sampled frames
• 8 observations / 4 native tracker IDs
• 3 canonical facts / 3 intervals
• 18 questions: 6 spatial, 12 temporal
• 18/18 valid smoke-candidate predictions; 0 missing, 0 invalid
• 5 annotated evidence frames
• logical bundle SHA-256: 07b7b99844b01badbbdecd65227144cdee7d7ef1e6d6671f2f022bc7498eaaf0
• summary SHA-256: 9a679fe7b1689d5d988f567c87c8be636b7e35d55079c377f58dda79a0e1948c

Visual walkthrough

1. From video frames to relationships

The first evaluator deliberately uses simple, inspectable 2D image-coordinate relationships. Tracked boxes become spatial facts; the ordering of accepted facts becomes temporal before/after tests.

These are teacher pseudo-labels, not human ground truth or full 3D scene understanding.

2. Evidence-review UI

The generated review surface connects the pipeline, one concrete proof chain, honest motion-quality gates, and the sampled relation timeline. It keeps evidence gaps and teacher defects visible instead of converting them into stronger claims.

[greptile-apps]

Greptile Summary

This PR adds a replayable video-relation evaluation pipeline.

• Samples videos into canonical spatial and temporal facts.
• Generates public questions with private oracle evidence.
• Runs the reference candidate in an isolated subprocess.
• Produces deterministic bundles, scoring reports, and an evidence viewer.

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.

Important Files Changed

Filename	Overview
dimos/benchmark/spatiotemporal/demo.py	Builds the evaluation flow and runs candidate work in a temporary directory.
dimos/benchmark/spatiotemporal/candidate_worker.py	Validates child paths and executes the public-only candidate entry point.
dimos/benchmark/spatiotemporal/test_demo.py	Covers candidate log isolation and temporary-root cleanup.

_{Reviews (9): Last reviewed commit: "Merge branch 'main' into feat/replayable..." | Re-trigger Greptile}