diff --git a/.gitignore b/.gitignore
index ef119951f5..4883f96a6f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -45,6 +45,7 @@ node_modules
# Ignore build artifacts
dist/
build/
+/.artifacts/
# Rust build artifacts
**/target/
diff --git a/dimos/benchmark/spatiotemporal/README.md b/dimos/benchmark/spatiotemporal/README.md
new file mode 100644
index 0000000000..b1427b2928
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/README.md
@@ -0,0 +1,594 @@
+# Replayable Spatiotemporal Video-Relation Evals
+
+This package turns a video into a deterministic evaluation bundle for spatial and temporal relationship questions.
+
+The core contribution is the evaluator, not a new vision model:
+
+- a real video is decoded and sparsely sampled;
+- YOLO-E produces teacher pseudo-labels and persistent object identities;
+- geometry produces canonical spatial facts and sampled-frame intervals;
+- deterministic generation produces answer-free public questions and private oracle evidence;
+- the same saved observations replay to byte-stable logical bundles under different output roots;
+- a candidate receives only the video and public questions;
+- an evaluator joins predictions to private answers and emits exact diagnostics; and
+- an evaluator-only HTML viewer makes every pseudo-label claim inspectable.
+
+## 1. This PR: reproducible evaluator foundation
+
+### Interviewer fast path
+
+#### Prerequisites
+
+A fresh checkout needs:
+
+- `git` and `git-lfs`;
+- access to the DimOS LFS endpoint for `assets/simple_demo.mp4` and the YOLO-E archive;
+- `uv`; and
+- `gh` only when using the `gh pr checkout` command below.
+
+The reference path runs on macOS or Linux and supports CPU execution. GitHub repository access does not necessarily grant access to the separately configured DimOS LFS endpoint.
+
+#### Reproduce the reference episode
+
+From a PR checkout, one script materializes assets, installs the project and prompt dependency, runs every focused quality gate, executes the real CPU demo, and prints the result locations:
+
+```bash
+gh pr checkout 2989
+./dimos/benchmark/spatiotemporal/reproduce_reference.sh
+```
+
+If Git reports that the branch is already assigned to another worktree, enter the path shown by Git and run the same script there. If setup and tests already passed, rerun only the real demo with:
+
+```bash
+./dimos/benchmark/spatiotemporal/reproduce_reference.sh --skip-setup --skip-tests
+```
+
+A successful run prints both:
+
+```text
+SUMMARY_GATES=PASS
+Reproduction complete.
+```
+
+Then open:
+
+```bash
+open .artifacts/spatiotemporal-video-qa/evidence-viewer/index.html
+```
+
+On Linux, open the same `index.html` in a browser.
+
+#### Try another video
+
+After the initial setup, reuse the pipeline with a different video:
+
+```bash
+uv run python -m dimos.benchmark.spatiotemporal.demo \
+ --source-video /path/to/video.mp4 \
+ --output-root .artifacts/my-video-eval \
+ --duration-s 25 \
+ --frame-stride 150
+```
+
+Use a regular 15–30-second video that OpenCV can decode. Useful episodes contain multiple visible, spatially separated objects and at least one relationship change. Arbitrary videos are not guaranteed to yield accepted relations or questions because YOLO-E labels and tracks remain teacher pseudo-labels.
+
+Open the custom result at:
+
+```text
+.artifacts/my-video-eval/evidence-viewer/index.html
+```
+
+#### Common setup failures
+
+- **LFS authentication or pointer-sized assets:** obtain access to the configured DimOS LFS endpoint, then rerun `git lfs pull --include='assets/simple_demo.mp4,data/.lfs/models_yoloe.tar.gz'`. The reproduction script rejects either asset if it remains an LFS pointer.
+- **No relations or questions from a custom video:** choose a clearer clip or adjust its duration and sampling stride. Inspect detection labels, confidence, identity continuity, and object separation rather than weakening privacy or determinism checks.
+
+A reviewer should establish the feature in this order:
+
+1. **Reproduce:** run the script and confirm the summary gates pass.
+2. **See:** inspect the annotated evidence viewer rather than trusting aggregate scores.
+3. **Verify privacy:** compare `bundle-a/public/` with `bundle-a/oracle/`.
+4. **Verify replay:** confirm `bundle-a` and `bundle-b` have the same logical digest.
+5. **Understand scope:** treat YOLO-E as a pseudo-label teacher and the bundled candidate as a plumbing smoke test.
+
+### What the PR proposes
+
+The PR establishes a reusable evaluator contract:
+
+```text
+video
+ -> sampled observations
+ -> canonical relation facts
+ -> sample-aware intervals
+ -> public questions + private answers/evidence
+ -> isolated candidate predictions
+ -> exact report + inspectable evidence
+```
+
+The package is intentionally divided by responsibility:
+
+```text
+contracts models.py, utilities.py
+teacher video_adapter.py, yoloe_adapter.py, relations.py, intervals.py
+generation generation.py, observation_io.py, replay.py
+release bundles.py, ports.py
+candidate temporal_memory_answerer.py, candidate_worker.py
+evaluation runner.py, scoring.py
+review evidence_viewer.py
+acceptance demo.py, reproduce_reference.sh, test_*.py
+```
+
+This PR stops at a deterministic and inspectable evaluator. It does not hide teacher defects, automatically promote generated pseudo-labels to ground truth, or claim that the smoke-candidate score measures visual intelligence.
+
+### Concrete examples from the verified reference video
+
+The examples below are copied from the generated schema-v2 artifacts, not invented documentation fixtures.
+
+#### Spatial positive and negative controls
+
+At sampled frame `0`, the teacher derives:
+
+```json
+{
+ "frame_id": 0,
+ "timestamp_s": 0.0,
+ "subject_id": "1",
+ "predicate": "left-of",
+ "object_id": "2",
+ "evidence_frame_ids": [0]
+}
+```
+
+That one accepted fact produces a public positive question and a private answer:
+
+```json
+{"text":"Did 1 ever appear left of 2?","predicate":"left-of","object_ids":["1","2"]}
+{"expected":true,"evidence_frame_ids":[0]}
+```
+
+It also produces the opposite negative control. The public question remains answer-free; the private oracle cites the complete sample schedule to support episode-level absence:
+
+```json
+{"text":"Did 1 ever appear right of 2?","predicate":"right-of","object_ids":["1","2"]}
+{"expected":false,"evidence_frame_ids":[0,150,300,450,600]}
+```
+
+Across the reference run, 3 accepted canonical spatial facts produce 6 spatial questions: 3 positive and 3 negative.
+
+#### Temporal ordering
+
+The private intervals include:
+
+```text
+frame 0 / 0.000 s: "1 left of 2"
+frame 300 / 10.001 s: "1 above 3"
+frame 600 / 20.002 s: "1 left of 8"
+```
+
+From those intervals, generation emits paired temporal checks. For example:
+
+```json
+{"text":"Did \"1 left of 2\" happen before \"1 above 3\"?","predicate":"before"}
+{"expected":true,"evidence_interval_ids":["interval@frame-0","interval@frame-300"]}
+
+{"text":"Did \"1 left of 2\" happen after \"1 above 3\"?","predicate":"after"}
+{"expected":false,"evidence_interval_ids":["interval@frame-0","interval@frame-300"]}
+```
+
+The readable aliases above shorten stable SHA-256 interval IDs only for presentation; generated artifacts contain the full IDs. With 3 unanimously ordered relation pairs, the reference run produces 12 balanced temporal questions: before/after positives and their inverse negative controls.
+
+#### Public/private proof
+
+Candidate-visible `public/questions.jsonl` contains only question semantics:
+
+```json
+{
+ "episode_id": "office_robot_25s",
+ "question_kind": "spatial",
+ "predicate": "left-of",
+ "object_ids": ["1", "2"],
+ "reference_ids": [],
+ "text": "Did 1 ever appear left of 2?"
+}
+```
+
+It does not contain `expected`, `evidence_frame_ids`, `evidence_interval_ids`, observations, facts, or intervals. The parent evaluator retains those fields under `oracle/`; it never serializes them into the reference candidate's explicit input or temporary root, and it uses them for scoring after candidate execution.
+
+### Proof matrix
+
+| Claim | Reproducible artifact | Reference evidence |
+|---|---|---|
+| A video automatically becomes an eval | `observations.jsonl`, `bundle-a/` | 5 sampled frames → 8 observations → 3 facts → 3 intervals → 18 questions |
+| Spatial cases are balanced | public questions + private answers | 6 spatial questions: 3 true, 3 false |
+| Temporal cases are strict and balanced | interval and answer JSONL | frames 0, 300, 600 → 12 temporal questions: 6 true, 6 false |
+| Reference candidate input excludes answers | candidate subprocess and bundle layout | only copied video + public questions are serialized into candidate root |
+| Replay is deterministic | `bundle-a`, `bundle-b`, `summary.json` | equal manifests and logical SHA-256 `07b7b998…eaaf0` |
+| Results are inspectable | `evidence-viewer/index.html` | 18 oracle rows linked to 5 annotated evidence frames |
+| The smoke score is not overclaimed | `summary.json` | `visually_grounded=false`, `score_interpretation=not_model_quality` |
+| Teacher defects remain visible | evidence viewer and README | robot ID changes `2`→`3`, missed detections, oven/cabinet mislabeled `refrigerator` |
+
+The reproduction script verifies generated counts, balanced expected values, absence of answer/evidence fields from public question records, summary gates, and representative examples. This proves the trusted reference candidate's explicit-input boundary, evaluator mechanics, and repeatability. It does not provide an OS security sandbox: an untrusted candidate must run in a container or worker with repository and oracle paths unmounted. It also intentionally does not prove that the current pseudo-label teacher or smoke candidate is production-accurate.
+
+## 2. Further directions: from videos to gated self-improvement
+
+The evaluator can become the data engine for continuous improvement:
+
+```text
+new videos
+ -> generated challenge episodes
+ -> quality and disagreement gates
+ -> candidate failure clustering
+ -> human review/correction
+ -> training-data export
+ -> candidate N+1
+ -> frozen holdout evaluation
+```
+
+The next proposed components are:
+
+1. **Episode admission report:** detector confidence, track continuity, label consistency, relation contradictions, evidence coverage, and replay determinism.
+2. **Three-set registry:** a frozen reviewed regression set, a reviewed improvement set, and an unreviewed challenge pool.
+3. **Failure miner:** cluster spatial, temporal, identity, label, missing-detection, and long-horizon-memory failures.
+4. **Review workflow:** accept, correct, or reject generated episodes using the existing evidence viewer as the starting interface.
+5. **Training export:** emit only reviewed examples, with provenance back to video, teacher version, schema, and evidence.
+6. **Promotion policy:** replace a candidate only when frozen accuracy improves, critical families do not regress, and privacy, latency, and resource gates pass.
+
+Automatically generated data may enter the challenge pool immediately, but it must not enter a trusted benchmark or training set without explicit quality gates.
+
+## 3. Production path
+
+Productionization should happen in explicit stages:
+
+### Stage A — repeatable local and CI evaluation
+
+- Pin model weights, dependencies, prompts, schema, and reference assets.
+- Run package tests and a small reviewed fixture in CI.
+- Publish manifests, summary, and viewer as build artifacts.
+- Preserve candidate subprocess and public/private filesystem isolation.
+
+### Stage B — versioned episode service
+
+- Store immutable source-video hashes and schema-versioned bundles.
+- Register teacher version, prompts, thresholds, and sampling schedule.
+- Make generation idempotent by episode key.
+- Add retention and access policies for private video and oracle evidence.
+
+### Stage C — quality-gated curation
+
+- Compute automatic admission metrics.
+- Route uncertain identity, label, or relation cases to review.
+- Record reviewer corrections separately from teacher output.
+- Promote only reviewed episodes into frozen evaluation sets.
+
+### Stage D — continuous candidate evaluation
+
+- Evaluate every candidate against the same frozen set.
+- Run new videos as a separate challenge stream.
+- Track per-family accuracy, missing or invalid answers, latency, memory, and regressions.
+- Keep oracle data inaccessible to candidate containers and services.
+
+### Stage E — controlled self-improvement
+
+- Select high-value reviewed failures for training.
+- Train or tune a new candidate with complete data provenance.
+- Compare against the incumbent on untouched frozen holdouts.
+- Require policy approval before promotion and support immediate rollback.
+
+Production readiness therefore means more than deploying the demo: it requires trusted data admission, immutable versioning, observability, privacy controls, candidate promotion gates, and rollback.
+
+# Detailed implementation reference
+
+## What this demonstrates
+
+The reference demo exercises real:
+
+- Git-LFS video input;
+- OpenCV decode, trim, encode, reopen, and full-frame verification;
+- YOLO-E 11s prompt inference;
+- tracker identity normalization;
+- spatial relation and interval derivation;
+- public/private bundle generation and validation;
+- root-independent deterministic replay;
+- candidate isolation;
+- TemporalMemory ingestion/query/cleanup plumbing; and
+- exact scoring and evidence rendering.
+
+The demo does **not** claim that YOLO-E outputs are human ground truth. They are teacher pseudo-labels and must be reviewed in `evidence-viewer/index.html`.
+
+The included TemporalMemory candidate is also **not** a model-quality baseline. It uses a timestamp-scripted VLM fixture and a deterministic public-question hash for yes/no responses. Its score proves that candidate plumbing, isolation, readiness, prediction parsing, and scoring run end to end without cloud credentials. Replace it with a real candidate before drawing model-quality conclusions.
+
+## Evaluation boundary
+
+```text
+TEACHER / EVALUATOR CANDIDATE
+
+source video copied source video
+ | |
+OpenCV sampling v
+ | TemporalMemory smoke fixture
+YOLO-E pseudo-labels |
+ | v
+canonical spatial facts public predictions
+ |
+sampled-frame intervals
+ |
+ +--> public questions ----------------------+
+ |
+ +--> private oracle answers/evidence
+ |
+ v
+ exact evaluator report
+```
+
+The candidate function receives only:
+
+1. a copied video in a temporary candidate directory;
+2. public `Question` records;
+3. that temporary directory; and
+4. the public duration.
+
+It never receives an `EvaluationBundle`, observations, facts, intervals, evidence IDs, or expected answers. The candidate directory is outside teacher output and is deleted after execution.
+
+## Semantics
+
+### Spatial facts
+
+Image-plane predicates are closed to:
+
+- `left-of` / `right-of`;
+- `above` / `below`.
+
+Replay stores one canonical representation per physical relation:
+
+- horizontal facts use `left-of`;
+- vertical facts use `above`.
+
+This prevents inverse duplicates such as `A left-of B` and `B right-of A` from receiving double weight.
+
+Spatial questions use episode-level existential wording, for example:
+
+```text
+Did object_1 ever appear left of object_2?
+```
+
+For a stable physical relation, generation emits:
+
+- one positive question for the observed canonical relation; and
+- one negative control for the opposite predicate only when that opposite relation never occurred in the episode.
+
+If a relation flips during the episode, both observed directions become positives and contradictory negatives are omitted. Negative evidence covers the complete sampled-frame schedule.
+
+### Temporal facts
+
+Facts coalesce when they occur at adjacent **sample positions**, not when raw source frame IDs differ by one. For a stride of 150, frames 0 and 150 are adjacent samples. An intervening sampled frame with no relation breaks the interval, including a zero-detection frame.
+
+Temporal questions are generated only from strict, unanimous ordering proofs. Touching, overlapping, containing, or contradictory intervals produce no temporal claim.
+
+Public temporal text is human-readable:
+
+```text
+Did "object_1 left of object_2" happen before "object_3 above object_4"?
+```
+
+Stable relation IDs remain in structured `reference_ids`; private interval IDs remain in oracle evidence.
+
+### Stable contracts
+
+All records are strict frozen Pydantic v2 models with `extra="forbid"` and no implicit coercion.
+
+Current schema:
+
+```text
+spatiotemporal-video-qa/v2
+```
+
+Question IDs include episode identity. Relation, interval, question, and bundle IDs are SHA-256 values over canonical semantic preimages. Changing a preimage, enum value, or schema version is an explicit migration and requires regenerated artifacts.
+
+## Package map
+
+| File | Responsibility |
+|---|---|
+| `models.py` | Strict records and closed enums |
+| `utilities.py` | Canonical JSON and stable IDs |
+| `relations.py` | Image-plane geometry |
+| `intervals.py` | Sample-schedule-aware interval construction |
+| `generation.py` | Balanced spatial and strict temporal cases |
+| `observation_io.py` | Canonical observation JSONL |
+| `bundles.py` | Public/private artifacts and manifests |
+| `replay.py` | Observation-to-bundle replay |
+| `video_adapter.py` | OpenCV sampling, including zero-detection samples |
+| `yoloe_adapter.py` | YOLO-E normalization and identity statistics |
+| `evidence_viewer.py` | Private HTML and annotated evidence frames |
+| `temporal_memory_answerer.py` | Public-only candidate lifecycle adapter |
+| `runner.py` / `scoring.py` | Parsing, scoring, and diagnostics |
+| `demo.py` | Real end-to-end acceptance command |
+
+## Prerequisites
+
+Run from the repository root.
+
+### Materialize LFS assets
+
+```bash
+git lfs install --local
+git lfs pull --include='assets/simple_demo.mp4,data/.lfs/models_yoloe.tar.gz'
+python3 - <<'PY'
+from pathlib import Path
+for name in ('assets/simple_demo.mp4', 'data/.lfs/models_yoloe.tar.gz'):
+ size = Path(name).stat().st_size
+ print(name, size)
+ assert size > 1024, f'{name} is still an LFS pointer'
+PY
+```
+
+### Extract weights and install the YOLO-E prompt dependency
+
+```bash
+uv run python -c "from dimos.utils.data import get_data; print(get_data('models_yoloe'))"
+uv pip install 'git+https://github.com/ultralytics/CLIP.git'
+```
+
+The first prompt-mode run may download `mobileclip_blt.ts` into the Ultralytics cache.
+
+## Run the real demo
+
+```bash
+uv run python -m dimos.benchmark.spatiotemporal.demo \
+ --source-video assets/simple_demo.mp4 \
+ --output-root .artifacts/spatiotemporal-video-qa \
+ --duration-s 25 \
+ --frame-stride 150
+```
+
+The command uses CPU for reproducibility. Duration must be 15–30 seconds and stride must be positive.
+
+## Output
+
+```text
+.artifacts/spatiotemporal-video-qa/
+├── office_robot_25s.mp4
+├── observations.jsonl
+├── summary.json
+├── evidence-viewer/
+│ ├── index.html
+│ └── frames/frame_*.jpg
+├── bundle-a/
+│ ├── public/
+│ └── oracle/
+└── bundle-b/
+ ├── public/
+ └── oracle/
+```
+
+`bundle-a` and `bundle-b` are independent writes of the same logical teacher data. Their manifests and logical digest must match.
+
+The viewer is evaluator-only because it contains expected answers and private evidence. It is generated atomically, escapes untrusted text, rejects symlinked path components, preserves an existing viewer on failure, and never writes into the candidate directory.
+
+## Verified reference run
+
+The schema-v2 acceptance run on macOS CPU completed twice with byte-identical `summary.json` files:
+
+- 750 decoded frames over 25 seconds;
+- 5 sampled frames per independent detector run;
+- 8 YOLO-E observations with 4 native tracker IDs and no fallback IDs;
+- 3 canonical relation facts and 3 sampled-frame intervals;
+- 18 questions and answers: 6 spatial and 12 temporal;
+- 5 annotated evidence frames;
+- detector observations equal across two fresh YOLO-E instances;
+- bundle logical SHA-256 `07b7b99844b01badbbdecd65227144cdee7d7ef1e6d6671f2f022bc7498eaaf0`;
+- full-summary SHA-256 `9a679fe7b1689d5d988f567c87c8be636b7e35d55079c377f58dda79a0e1948c`; and
+- 18/18 valid smoke-baseline predictions with zero missing or invalid responses.
+
+Visual inspection confirmed that the boxes are geometrically aligned where detections exist. It also exposed the intended pseudo-label caveats: the same physical robot changes tracker identity from `2` to `3`, the teacher calls a built-in oven/cabinet column a `refrigerator`, and the visible robot is not detected in every sampled frame. These labels are suitable for demonstrating evaluator mechanics, not for claiming a clean human-annotated benchmark.
+
+## Five-minute reviewer walkthrough
+
+1. Open `evidence-viewer/index.html` and confirm the header metrics match the reference run: 5 evidence frames, 4 track IDs, 3 relation intervals, 6 spatial and 12 temporal questions, and a balanced 9/9 oracle split.
+2. Follow **One generated proof chain** from frame → derived relation → answer-free public question → private expected answer and evidence. This is the shortest proof of what the evaluator creates.
+3. Read **Robot-motion verification** before claiming motion quality. A trustworthy motion claim requires all four checks:
+ - relation events are linked to evidence frames;
+ - relation intervals have strict temporal order;
+ - the robot keeps one identity across those events; and
+ - robot detections cover the evidence schedule sufficiently.
+4. For this reference clip, confirm the honest verdict: relation ordering is evaluable, but the full robot-motion claim needs review because the robot changes ID `2` → `3` and appears in only 2/5 evidence frames.
+5. Use **What changed, and when** to inspect the sampled sequence: `1 left of 2` at 0.000 s, `1 above 3` at 10.001 s, and `1 left of 8` at 20.002 s. The 5.001 s and 15.002 s gaps remain visible and are not interpolated.
+6. Inspect all five annotated frames. Confirm box placement, label semantics, confidence, identity continuity, and the relation chip shown for each accepted event. Click any frame for the full-resolution lightbox.
+7. Filter **Questions and private oracle** by spatial, temporal, true, or false. Follow evidence chips back to frames and verify both positive cases and inverse negative controls.
+8. Read **Reusable motion-evaluation patterns** to connect the same contract to locomotion/patrol, manipulation, long-horizon memory, and release regression gates.
+9. Inspect `bundle-a/public/questions.jsonl`: it must contain no expected answers or evidence. Keep `bundle-a/oracle/answers.jsonl` private.
+10. Compare `bundle-a` and `bundle-b` manifests and the logical digest in `summary.json` to verify deterministic replay.
+11. Confirm `summary.json` labels the candidate as a non-visual plumbing smoke test.
+
+The viewer intentionally distinguishes **relation-order evidence** from a stronger **continuous robot-motion claim**. The current evaluator can prove sampled spatial predicates and strict before/after ordering. Production trajectory claims additionally need stable identity, higher detection coverage, denser sampling, and task-specific motion predicates or pose data.
+
+## Quality gates
+
+```bash
+uv run pytest dimos/benchmark/spatiotemporal -q
+uv run ruff format --check dimos/benchmark/spatiotemporal
+uv run ruff check dimos/benchmark/spatiotemporal
+uv run --group lint mypy dimos/benchmark/spatiotemporal
+```
+
+Inspect the real-run summary:
+
+```bash
+python3 - <<'PY'
+import json
+from pathlib import Path
+s = json.loads(Path('.artifacts/spatiotemporal-video-qa/summary.json').read_text())
+assert s['video']['duration_s'] == 25
+assert s['video']['frames'] == 750
+assert s['teacher']['detector_repeat_equal'] is True
+assert s['teacher']['relation_facts'] > 0
+assert s['teacher']['relation_intervals'] > 0
+assert s['teacher']['spatial_questions'] > 0
+assert s['teacher']['temporal_questions'] > 0
+assert s['candidate']['candidate_used_oracle'] is False
+assert s['candidate']['baseline_kind'] == 'public_only_plumbing_smoke_test'
+assert s['candidate']['visually_grounded'] is False
+assert s['candidate']['readiness']['ready'] is True
+assert s['candidate']['questions_answered'] == s['teacher']['questions']
+assert s['candidate']['status_counts']['missing'] == 0
+assert s['candidate']['status_counts']['invalid'] == 0
+assert s['review']['question_count'] == s['teacher']['questions']
+assert Path('.artifacts/spatiotemporal-video-qa', s['review']['index_path']).is_file()
+print('SUMMARY_GATES=PASS')
+PY
+```
+
+Verify complete-run determinism:
+
+```bash
+cp .artifacts/spatiotemporal-video-qa/summary.json /tmp/stqa-summary-first.json
+uv run python -m dimos.benchmark.spatiotemporal.demo \
+ --source-video assets/simple_demo.mp4 \
+ --output-root .artifacts/spatiotemporal-video-qa \
+ --duration-s 25 \
+ --frame-stride 150
+cmp /tmp/stqa-summary-first.json .artifacts/spatiotemporal-video-qa/summary.json
+shasum -a 256 .artifacts/spatiotemporal-video-qa/summary.json
+```
+
+`cmp` must be silent and exit zero.
+
+## Safety properties covered by tests
+
+- strict and frozen records;
+- duplicate-key and noncanonical JSON rejection;
+- episode-safe question IDs;
+- sample-schedule validation and zero-detection frames;
+- inverse-relation canonicalization;
+- relation-flip order independence;
+- public/private manifest binding;
+- exactly one oracle answer per public question;
+- root-independent replay;
+- candidate signature and filesystem isolation;
+- source/target path and hard-link protection;
+- symlinked output and ancestor rejection;
+- full encoded-video decode verification;
+- deterministic, escaped evidence HTML;
+- temporal interval-to-frame evidence resolution; and
+- failure-clean atomic viewer replacement.
+
+## Extending the evaluator
+
+A production candidate should implement the public `CandidateAnswerer` lifecycle:
+
+1. ingest the public video;
+2. report readiness;
+3. answer each public `Question`; and
+4. release resources.
+
+Do not pass private bundle records into candidate code. Do not use oracle answers to tune predictions during a scored run. Keep teacher generation replayable from saved observations so evaluator changes can be compared independently from detector changes.
+
+## Known limitations
+
+- YOLO-E labels and tracks are pseudo-labels and can be semantically wrong.
+- Relations are 2D image-plane geometry, not metric 3D scene relations.
+- Sparse sampling can miss short-lived events.
+- Episode-level negative questions require the complete sampled-frame schedule.
+- The reference candidate score is not a visual-model benchmark.
+- The reference episode ID is fixed in the demo; a dataset CLI should expose it explicitly.
+
+If a custom video fails, improve the video, prompts, tracking, confidence threshold, sampling schedule, or versioned benchmark contract. Do not weaken privacy, determinism, or path-safety checks to force a pass.
diff --git a/dimos/benchmark/spatiotemporal/bundles.py b/dimos/benchmark/spatiotemporal/bundles.py
new file mode 100644
index 0000000000..be1354cd06
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/bundles.py
@@ -0,0 +1,280 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Deterministic public/private evaluation bundle storage."""
+
+from collections.abc import Sequence
+from dataclasses import dataclass
+from hashlib import sha256
+from pathlib import Path
+from typing import TypeVar
+
+from pydantic import BaseModel
+
+from dimos.benchmark.spatiotemporal.models import (
+ BundleArtifact,
+ ObjectObservation,
+ OracleAnswer,
+ OracleBundleManifest,
+ PublicBundleManifest,
+ Question,
+ RelationFact,
+ RelationInterval,
+)
+from dimos.benchmark.spatiotemporal.utilities import (
+ SCHEMA_VERSION,
+ canonical_json_bytes,
+ canonical_model_json,
+ stable_id,
+)
+
+RecordT = TypeVar("RecordT", bound=BaseModel)
+
+
+@dataclass(frozen=True)
+class EvaluationBundle:
+ """A fully validated public bundle and its private oracle records."""
+
+ public_manifest: PublicBundleManifest
+ oracle_manifest: OracleBundleManifest
+ questions: tuple[Question, ...]
+ observations: tuple[ObjectObservation, ...]
+ relation_facts: tuple[RelationFact, ...]
+ relation_intervals: tuple[RelationInterval, ...]
+ answers: tuple[OracleAnswer, ...]
+
+
+def _write_jsonl(root: Path, relative_path: str, records: Sequence[BaseModel]) -> BundleArtifact:
+ path = _safe_path(root, relative_path)
+ content = b"".join(f"{canonical_model_json(record)}\n".encode() for record in records)
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_bytes(content)
+ return BundleArtifact(
+ path=path.parent.name + "/" + path.name,
+ sha256=sha256(content).hexdigest(),
+ record_count=len(records),
+ )
+
+
+def _safe_path(root: Path, relative_path: str) -> Path:
+ path = root
+ if path.is_symlink():
+ raise ValueError(f"bundle path contains a symlink: {relative_path}")
+ for part in Path(relative_path).parts:
+ path /= part
+ if path.is_symlink():
+ raise ValueError(f"bundle path contains a symlink: {relative_path}")
+ return path
+
+
+def _read_jsonl(root: Path, artifact: BundleArtifact, model: type[RecordT]) -> tuple[RecordT, ...]:
+ path = _safe_path(root, artifact.path)
+ return tuple(model.model_validate_json(line) for line in path.read_bytes().splitlines())
+
+
+def _verify_artifact(root: Path, artifact: BundleArtifact) -> None:
+ content = _safe_path(root, artifact.path).read_bytes()
+ if sha256(content).hexdigest() != artifact.sha256:
+ raise ValueError(f"artifact digest mismatch: {artifact.path}")
+ record_count = len(content.splitlines())
+ if record_count != artifact.record_count:
+ raise ValueError(f"artifact record count mismatch: {artifact.path}")
+
+
+def _verify_episode_metadata(root: Path, manifest: PublicBundleManifest) -> None:
+ expected = (
+ canonical_json_bytes(
+ {
+ "bundle_id": manifest.bundle_id,
+ "episode_id": manifest.episode_id,
+ "schema_version": manifest.schema_version,
+ "source_video_sha256": manifest.source_video_sha256,
+ }
+ )
+ + b"\n"
+ )
+ if (root / manifest.episode.path).read_bytes() != expected:
+ raise ValueError("episode metadata does not match the public manifest")
+
+
+def _validate_records(
+ *,
+ episode_id: str,
+ questions: Sequence[Question],
+ observations: Sequence[ObjectObservation],
+ relation_facts: Sequence[RelationFact],
+ relation_intervals: Sequence[RelationInterval],
+ answers: Sequence[OracleAnswer],
+) -> None:
+ question_ids = [question.question_id for question in questions]
+ if len(set(question_ids)) != len(question_ids):
+ raise ValueError("duplicate question ID")
+ answer_ids = [answer.question_id for answer in answers]
+ if len(set(answer_ids)) != len(answer_ids):
+ raise ValueError("duplicate oracle answer reference")
+ if any(answer_id not in set(question_ids) for answer_id in answer_ids):
+ raise ValueError("oracle answer contains a foreign question ID")
+ if any(question_id not in set(answer_ids) for question_id in question_ids):
+ raise ValueError("public question is missing oracle answer")
+ relation_ids = {fact.relation_id for fact in relation_facts} | {
+ interval.relation_id for interval in relation_intervals
+ }
+ if any(
+ reference_id not in relation_ids
+ for question in questions
+ for reference_id in question.reference_ids
+ ):
+ raise ValueError("question contains a foreign relation reference")
+ interval_id_list = [interval.interval_id for interval in relation_intervals]
+ interval_ids = set(interval_id_list)
+ if len(interval_ids) != len(interval_id_list):
+ raise ValueError("duplicate interval ID")
+ if any(
+ interval_id not in interval_ids
+ for answer in answers
+ for interval_id in answer.evidence_interval_ids
+ ):
+ raise ValueError("oracle answer contains a foreign interval reference")
+ if (
+ any(question.episode_id != episode_id for question in questions)
+ or any(observation.episode_id != episode_id for observation in observations)
+ or any(fact.episode_id != episode_id for fact in relation_facts)
+ or any(interval.episode_id != episode_id for interval in relation_intervals)
+ ):
+ raise ValueError("bundle contains a record from a foreign episode")
+
+
+def write_bundle(
+ root: Path,
+ *,
+ episode_id: str,
+ source_video_sha256: str,
+ questions: Sequence[Question],
+ observations: Sequence[ObjectObservation],
+ relation_facts: Sequence[RelationFact],
+ relation_intervals: Sequence[RelationInterval],
+ answers: Sequence[OracleAnswer],
+) -> None:
+ """Write one deterministic public release and its private oracle companion."""
+ _validate_records(
+ episode_id=episode_id,
+ questions=questions,
+ observations=observations,
+ relation_facts=relation_facts,
+ relation_intervals=relation_intervals,
+ answers=answers,
+ )
+
+ root.mkdir(parents=True, exist_ok=True)
+ bundle_id = stable_id(
+ "bundle",
+ {
+ "episode_id": episode_id,
+ "schema_version": SCHEMA_VERSION,
+ "source_video_sha256": source_video_sha256,
+ },
+ )
+ episode_path = _safe_path(root, "public/episode.json")
+ episode_content = (
+ canonical_json_bytes(
+ {
+ "bundle_id": bundle_id,
+ "episode_id": episode_id,
+ "schema_version": SCHEMA_VERSION,
+ "source_video_sha256": source_video_sha256,
+ }
+ )
+ + b"\n"
+ )
+ episode_path.parent.mkdir(parents=True, exist_ok=True)
+ episode_path.write_bytes(episode_content)
+
+ public_manifest = PublicBundleManifest(
+ schema_version=SCHEMA_VERSION,
+ bundle_id=bundle_id,
+ episode_id=episode_id,
+ source_video_sha256=source_video_sha256,
+ episode=BundleArtifact(
+ path="public/episode.json",
+ sha256=sha256(episode_content).hexdigest(),
+ record_count=1,
+ ),
+ questions=_write_jsonl(root, "public/questions.jsonl", questions),
+ )
+ public_manifest_bytes = f"{canonical_model_json(public_manifest)}\n".encode()
+ _safe_path(root, "public/manifest.json").write_bytes(public_manifest_bytes)
+
+ oracle_manifest = OracleBundleManifest(
+ schema_version=SCHEMA_VERSION,
+ bundle_id=bundle_id,
+ episode_id=episode_id,
+ source_video_sha256=source_video_sha256,
+ public_manifest_sha256=sha256(public_manifest_bytes).hexdigest(),
+ observations=_write_jsonl(root, "oracle/observations.jsonl", observations),
+ relation_facts=_write_jsonl(root, "oracle/relation_facts.jsonl", relation_facts),
+ relation_intervals=_write_jsonl(
+ root, "oracle/relation_intervals.jsonl", relation_intervals
+ ),
+ answers=_write_jsonl(root, "oracle/answers.jsonl", answers),
+ )
+ _safe_path(root, "oracle/manifest.json").write_text(
+ f"{canonical_model_json(oracle_manifest)}\n",
+ encoding="utf-8",
+ )
+
+
+def load_bundle(root: Path) -> EvaluationBundle:
+ """Load one public release together with its private oracle companion."""
+ public_manifest_path = _safe_path(root, "public/manifest.json")
+ oracle_manifest_path = _safe_path(root, "oracle/manifest.json")
+ public_manifest = PublicBundleManifest.model_validate_json(public_manifest_path.read_bytes())
+ oracle_manifest = OracleBundleManifest.model_validate_json(oracle_manifest_path.read_bytes())
+ public_manifest_bytes = public_manifest_path.read_bytes()
+ if sha256(public_manifest_bytes).hexdigest() != oracle_manifest.public_manifest_sha256:
+ raise ValueError("oracle manifest does not match the public manifest digest")
+ if (
+ oracle_manifest.bundle_id != public_manifest.bundle_id
+ or oracle_manifest.episode_id != public_manifest.episode_id
+ or oracle_manifest.source_video_sha256 != public_manifest.source_video_sha256
+ ):
+ raise ValueError("public and oracle manifest identities differ")
+ artifacts = (
+ public_manifest.episode,
+ public_manifest.questions,
+ oracle_manifest.observations,
+ oracle_manifest.relation_facts,
+ oracle_manifest.relation_intervals,
+ oracle_manifest.answers,
+ )
+ for artifact in artifacts:
+ _verify_artifact(root, artifact)
+ _verify_episode_metadata(root, public_manifest)
+ bundle = EvaluationBundle(
+ public_manifest=public_manifest,
+ oracle_manifest=oracle_manifest,
+ questions=_read_jsonl(root, public_manifest.questions, Question),
+ observations=_read_jsonl(root, oracle_manifest.observations, ObjectObservation),
+ relation_facts=_read_jsonl(root, oracle_manifest.relation_facts, RelationFact),
+ relation_intervals=_read_jsonl(root, oracle_manifest.relation_intervals, RelationInterval),
+ answers=_read_jsonl(root, oracle_manifest.answers, OracleAnswer),
+ )
+ _validate_records(
+ episode_id=public_manifest.episode_id,
+ questions=bundle.questions,
+ observations=bundle.observations,
+ relation_facts=bundle.relation_facts,
+ relation_intervals=bundle.relation_intervals,
+ answers=bundle.answers,
+ )
+ return bundle
diff --git a/dimos/benchmark/spatiotemporal/candidate_worker.py b/dimos/benchmark/spatiotemporal/candidate_worker.py
new file mode 100644
index 0000000000..eccb7b8307
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/candidate_worker.py
@@ -0,0 +1,74 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Subprocess entry point for the filesystem-isolated public candidate."""
+
+import argparse
+import json
+from pathlib import Path
+
+
+def _candidate_child(root: Path, path: Path, *, must_exist: bool) -> Path:
+ resolved_root = root.resolve(strict=True)
+ resolved = path.resolve(strict=must_exist)
+ if not resolved.is_relative_to(resolved_root):
+ raise ValueError(f"candidate path escapes temporary root: {path}")
+ if path.is_symlink():
+ raise ValueError(f"candidate path must not be a symlink: {path}")
+ return resolved
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--video", type=Path, required=True)
+ parser.add_argument("--questions", type=Path, required=True)
+ parser.add_argument("--output-root", type=Path, required=True)
+ parser.add_argument("--duration-s", type=int, required=True)
+ parser.add_argument("--result", type=Path, required=True)
+ args = parser.parse_args()
+
+ output_root = args.output_root.resolve(strict=True)
+ video = _candidate_child(output_root, args.video, must_exist=True)
+ questions_path = _candidate_child(output_root, args.questions, must_exist=True)
+ result_path = _candidate_child(output_root, args.result, must_exist=False)
+
+ from dimos.benchmark.spatiotemporal.demo import (
+ _run_temporal_memory_candidate_in_process,
+ )
+ from dimos.benchmark.spatiotemporal.models import Question
+
+ questions = tuple(
+ Question.model_validate_json(line)
+ for line in questions_path.read_text(encoding="utf-8").splitlines()
+ if line
+ )
+ answers, runtime = _run_temporal_memory_candidate_in_process(
+ video,
+ questions,
+ output_root,
+ args.duration_s,
+ )
+ result_path.write_text(
+ json.dumps(
+ {"answers": answers, "runtime": runtime},
+ sort_keys=True,
+ separators=(",", ":"),
+ )
+ + "\n",
+ encoding="utf-8",
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/dimos/benchmark/spatiotemporal/demo.py b/dimos/benchmark/spatiotemporal/demo.py
new file mode 100644
index 0000000000..4ecdf2dd4b
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/demo.py
@@ -0,0 +1,543 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""One-command real-video spatiotemporal QA demonstration."""
+
+from __future__ import annotations
+
+import argparse
+from collections import Counter
+from collections.abc import Sequence
+from dataclasses import asdict
+from hashlib import sha256
+import json
+import math
+import os
+from pathlib import Path
+import shutil
+import subprocess
+import sys
+from tempfile import TemporaryDirectory
+from typing import Any, cast
+
+import cv2
+
+from dimos.benchmark.spatiotemporal.bundles import EvaluationBundle, load_bundle
+from dimos.benchmark.spatiotemporal.evidence_viewer import write_evidence_viewer
+from dimos.benchmark.spatiotemporal.models import ObjectObservation, Question, QuestionId
+from dimos.benchmark.spatiotemporal.observation_io import write_observations
+from dimos.benchmark.spatiotemporal.replay import replay_observations
+from dimos.benchmark.spatiotemporal.runner import build_evaluation_report
+from dimos.benchmark.spatiotemporal.temporal_memory_answerer import TemporalMemoryAnswerer
+from dimos.benchmark.spatiotemporal.utilities import canonical_model_json
+from dimos.benchmark.spatiotemporal.video_adapter import OpenCVVideoSampler
+from dimos.benchmark.spatiotemporal.yoloe_adapter import (
+ YoloeAdapterStatistics,
+ YoloeObservationDetector,
+)
+from dimos.models.vl.base import VlModel
+from dimos.msgs.sensor_msgs.Image import Image, ImageFormat
+from dimos.perception.detection.detectors.yoloe import Yoloe2DDetector, YoloePromptMode
+from dimos.perception.experimental.temporal_memory.temporal_memory import TemporalMemory
+
+DEFAULT_PROMPTS = (
+ "quadruped robot",
+ "chair",
+ "whiteboard",
+ "table",
+ "trash can",
+ "refrigerator",
+ "door",
+ "cart",
+)
+
+
+class TimestampScriptedPublicVlModel(VlModel):
+ """Deterministic TemporalMemory plumbing fixture with no private inputs."""
+
+ def query(self, image: Image, query: str, **kwargs: Any) -> str:
+ if "new_entities" in query and "entities_present" in query:
+ timestamp_s = float(image.ts or 0.0)
+ entities = [{"id": "R1", "type": "appliance", "descriptor": "white refrigerator"}]
+ if timestamp_s < 10.0:
+ entities.append({"id": "Q1", "type": "robot", "descriptor": "quadruped robot"})
+ else:
+ entities.append({"id": "C1", "type": "furniture", "descriptor": "office chair"})
+ moving_id = "Q1" if timestamp_s < 10.0 else "C1"
+ relation_type = "left_of" if timestamp_s < 10.0 else "right_of"
+ return json.dumps(
+ {
+ "window": {"start_s": max(0.0, timestamp_s - 4.0), "end_s": timestamp_s},
+ "caption": "Robot lab scene with changing robot and chair visibility.",
+ "entities_present": [{"id": entity["id"]} for entity in entities],
+ "new_entities": entities,
+ "relations": [
+ {
+ "type": relation_type,
+ "subject": moving_id,
+ "object": "R1",
+ "confidence": 0.7,
+ }
+ ],
+ "on_screen_text": [],
+ }
+ )
+ if "rolling summary" in query.lower():
+ return "A quadruped robot appeared before an office chair near a refrigerator."
+ marker = "**Question:** "
+ public_question = query.split(marker, 1)[1].splitlines()[0] if marker in query else query
+ return "yes" if int(sha256(public_question.encode()).hexdigest(), 16) % 2 == 0 else "no"
+
+ def query_batch(self, images: list[Image], query: str, **kwargs: Any) -> list[str]:
+ return [self.query(image, query, **kwargs) for image in images]
+
+ def stop(self) -> None:
+ return None
+
+
+def _assert_no_symlink_components(path: Path) -> None:
+ current = Path(path.anchor) if path.is_absolute() else Path.cwd()
+ parts = path.parts[1:] if path.is_absolute() else path.parts
+ for part in parts:
+ current /= part
+ if current.is_symlink():
+ raise ValueError(f"output path contains a symlink: {current}")
+
+
+def _prepare_output_root(output_root: Path) -> Path:
+ _assert_no_symlink_components(output_root.absolute())
+ if output_root.exists() and not output_root.is_dir():
+ raise ValueError("output root must be a directory")
+ output_root.mkdir(parents=True, exist_ok=True)
+ return output_root.resolve()
+
+
+def _safe_child(output_root: Path, name: str) -> Path:
+ if Path(name).name != name:
+ raise ValueError(f"unsafe artifact name: {name}")
+ path = output_root / name
+ _assert_no_symlink_components(path)
+ return path
+
+
+def _reset_directory(path: Path) -> None:
+ if path.is_symlink():
+ raise ValueError(f"refusing to replace symlinked directory: {path}")
+ if path.exists():
+ if not path.is_dir():
+ raise ValueError(f"expected directory artifact: {path}")
+ shutil.rmtree(path)
+ path.mkdir(parents=True)
+
+
+def _trim_video(source: Path, target: Path, duration_s: float) -> tuple[float, int]:
+ if source.is_symlink() or not source.is_file():
+ raise ValueError(f"source video must be a regular file: {source}")
+ if target.is_symlink() or target.is_dir():
+ raise ValueError(f"target video must be a regular file path: {target}")
+ if source.resolve() == target.resolve(strict=False) or (
+ target.exists() and os.path.samefile(source, target)
+ ):
+ raise ValueError("source and target video paths must differ")
+ capture = cv2.VideoCapture(str(source))
+ written = 0
+ try:
+ if not capture.isOpened():
+ raise RuntimeError(f"failed to open source video: {source}")
+ fps = capture.get(cv2.CAP_PROP_FPS)
+ width = int(capture.get(cv2.CAP_PROP_FRAME_WIDTH))
+ height = int(capture.get(cv2.CAP_PROP_FRAME_HEIGHT))
+ if not math.isfinite(fps) or fps <= 0.0 or width <= 0 or height <= 0:
+ raise ValueError("source video has invalid metadata")
+ frame_limit = round(duration_s * fps)
+ writer = cv2.VideoWriter(
+ str(target),
+ cv2.VideoWriter_fourcc(*"mp4v"), # type: ignore[attr-defined]
+ fps,
+ (width, height),
+ )
+ try:
+ if not writer.isOpened():
+ raise RuntimeError(f"failed to open target video writer: {target}")
+ while written < frame_limit:
+ decoded, frame = capture.read()
+ if not decoded:
+ break
+ writer.write(frame)
+ written += 1
+ finally:
+ writer.release()
+ except Exception:
+ if target.exists() and not target.is_symlink() and target.is_file():
+ target.unlink()
+ raise
+ finally:
+ capture.release()
+ if written != frame_limit:
+ if target.exists() and not target.is_symlink() and target.is_file():
+ target.unlink()
+ raise RuntimeError(f"expected {frame_limit} frames, wrote {written}")
+ verification = cv2.VideoCapture(str(target))
+ decoded_frames = 0
+ try:
+ if not verification.isOpened():
+ raise RuntimeError(f"failed to reopen encoded video: {target}")
+ while True:
+ decoded, _ = verification.read()
+ if not decoded:
+ break
+ decoded_frames += 1
+ except Exception:
+ if target.exists() and not target.is_symlink() and target.is_file():
+ target.unlink()
+ raise
+ finally:
+ verification.release()
+ if decoded_frames != frame_limit:
+ if target.exists() and not target.is_symlink() and target.is_file():
+ target.unlink()
+ raise RuntimeError(
+ f"encoded video contains {decoded_frames} decodable frames; expected {frame_limit}"
+ )
+ return fps, written
+
+
+def _detector_factory() -> Yoloe2DDetector:
+ return Yoloe2DDetector(
+ prompt_mode=YoloePromptMode.PROMPT,
+ conf=0.15,
+ max_area_ratio=0.8,
+ device="cpu",
+ )
+
+
+def _sample_observations(
+ video: Path, frame_stride: int
+) -> tuple[
+ tuple[ObjectObservation, ...],
+ YoloeAdapterStatistics,
+ tuple[tuple[int, float], ...],
+]:
+ detector = YoloeObservationDetector(prompts=DEFAULT_PROMPTS, detector_factory=_detector_factory)
+ sampler = OpenCVVideoSampler(detector=detector, frame_stride=frame_stride)
+ try:
+ observations = sampler.sample(video, "office_robot_25s")
+ return observations, detector.statistics, sampler.sample_schedule
+ finally:
+ sampler.close()
+
+
+def _teacher_run(
+ video: Path, output_root: Path, frame_stride: int
+) -> tuple[EvaluationBundle, dict[str, Any]]:
+ observations_path = _safe_child(output_root, "observations.jsonl")
+ if observations_path.is_dir():
+ raise ValueError("observation artifact path must not be a directory")
+ first_root = _safe_child(output_root, "bundle-a")
+ second_root = _safe_child(output_root, "bundle-b")
+ _reset_directory(first_root)
+ _reset_directory(second_root)
+ observations, detector_statistics, sample_schedule = _sample_observations(video, frame_stride)
+ repeated_observations, repeated_statistics, repeated_schedule = _sample_observations(
+ video, frame_stride
+ )
+ if (
+ observations != repeated_observations
+ or detector_statistics != repeated_statistics
+ or sample_schedule != repeated_schedule
+ ):
+ raise RuntimeError("YOLO-E observations differ across independent detector runs")
+ write_observations(observations_path, observations)
+ source_video_sha256 = sha256(video.read_bytes()).hexdigest()
+ first = replay_observations(
+ observations_path,
+ first_root,
+ source_video_sha256,
+ sample_schedule=sample_schedule,
+ )
+ second = replay_observations(
+ observations_path,
+ second_root,
+ source_video_sha256,
+ sample_schedule=sample_schedule,
+ )
+ if first.logical_sha256 != second.logical_sha256:
+ raise RuntimeError("bundle logical hashes differ across output roots")
+ first_bundle = load_bundle(first_root)
+ second_bundle = load_bundle(second_root)
+ if first_bundle.public_manifest != second_bundle.public_manifest:
+ raise RuntimeError("public manifests differ across output roots")
+ if first_bundle.oracle_manifest != second_bundle.oracle_manifest:
+ raise RuntimeError("oracle manifests differ across output roots")
+ summary = {
+ "answers": len(first_bundle.answers),
+ "bundle_id": first_bundle.public_manifest.bundle_id,
+ "detector_repeat_equal": True,
+ "detector_statistics": asdict(detector_statistics),
+ "labels": dict(sorted(Counter(observation.label for observation in observations).items())),
+ "logical_sha256": first.logical_sha256,
+ "observations": len(observations),
+ "questions": len(first_bundle.questions),
+ "relation_facts": len(first_bundle.relation_facts),
+ "relation_intervals": len(first_bundle.relation_intervals),
+ "sampled_frames": len(sample_schedule),
+ "source_video_sha256": source_video_sha256,
+ "spatial_questions": sum(
+ question.question_kind.value == "spatial" for question in first_bundle.questions
+ ),
+ "temporal_questions": sum(
+ question.question_kind.value == "temporal" for question in first_bundle.questions
+ ),
+ "unique_object_ids": sorted({observation.object_id for observation in observations}),
+ }
+ return first_bundle, summary
+
+
+def _build_temporal_memory(database_root: Path, duration_s: int) -> TemporalMemory:
+ """Construct TemporalMemory with persistent paths selected by the worker environment."""
+ return TemporalMemory(
+ vlm=TimestampScriptedPublicVlModel(),
+ db_dir=str(database_root),
+ new_memory=True,
+ fps=1.0,
+ window_s=5.0,
+ stride_s=5.0,
+ max_frames_per_window=3,
+ max_buffer_frames=max(30, duration_s),
+ summary_interval_s=30.0,
+ enable_distance_estimation=False,
+ visualize=False,
+ use_clip_filtering=False,
+ )
+
+
+def _run_temporal_memory_candidate_in_process(
+ video: Path,
+ questions: Sequence[Question],
+ output_root: Path,
+ duration_s: int,
+) -> tuple[dict[QuestionId, str | bool | None], dict[str, Any]]:
+ """Run a public-only candidate and return predictions without oracle access."""
+ database_root = _safe_child(output_root, "temporal-memory")
+ _reset_directory(database_root)
+ temporal_memory = _build_temporal_memory(database_root, duration_s)
+
+ def ingest(path: Path) -> int:
+ temporal_memory._accumulator.set_start_time(0.0)
+ capture = cv2.VideoCapture(str(path))
+ try:
+ if not capture.isOpened():
+ raise RuntimeError(f"TemporalMemory failed to open video: {path}")
+ fps = capture.get(cv2.CAP_PROP_FPS)
+ if not math.isfinite(fps) or fps <= 0.0:
+ raise ValueError("TemporalMemory video FPS must be finite and positive")
+ ingested = 0
+ for second in range(duration_s):
+ capture.set(cv2.CAP_PROP_POS_FRAMES, round(second * fps))
+ decoded, frame = capture.read()
+ if not decoded:
+ raise RuntimeError(f"TemporalMemory failed to decode second {second}")
+ image = Image.from_numpy(
+ cv2.cvtColor(frame, cv2.COLOR_BGR2RGB),
+ format=ImageFormat.RGB,
+ frame_id=str(round(second * fps)),
+ ts=float(second),
+ )
+ temporal_memory._accumulator.add_frame(image, float(second))
+ ingested += 1
+ if ingested % 5 == 0:
+ temporal_memory._analyze_window()
+ return ingested
+ finally:
+ capture.release()
+
+ answerer = TemporalMemoryAnswerer(
+ ingest=ingest,
+ query=temporal_memory.query,
+ cleanup=temporal_memory.stop,
+ )
+ try:
+ readiness = answerer.ingest_video(video)
+ state = temporal_memory.get_state()
+ candidate_answers = {
+ question.question_id: answerer.answer(question) for question in questions
+ }
+ runtime = {
+ "baseline_kind": "public_only_plumbing_smoke_test",
+ "no_answers": sum(answer == "no" for answer in candidate_answers.values()),
+ "questions_answered": len(candidate_answers),
+ "readiness": readiness.model_dump(mode="json"),
+ "score_interpretation": "not_model_quality",
+ "temporal_memory_state": state,
+ "visually_grounded": False,
+ "vlm_mode": "timestamp_scripted_temporal_memory_plumbing",
+ "yes_answers": sum(answer == "yes" for answer in candidate_answers.values()),
+ }
+ return candidate_answers, runtime
+ finally:
+ answerer.close()
+
+
+def _run_temporal_memory_candidate(
+ video: Path,
+ questions: Sequence[Question],
+ output_root: Path,
+ duration_s: int,
+) -> tuple[dict[QuestionId, str | bool | None], dict[str, Any]]:
+ """Run the public-only candidate in a subprocess with isolated logs and storage."""
+ questions_path = _safe_child(output_root, "candidate-questions.jsonl")
+ result_path = _safe_child(output_root, "candidate-result.json")
+ questions_path.write_text(
+ "".join(f"{canonical_model_json(question)}\n" for question in questions),
+ encoding="utf-8",
+ )
+ child_env = os.environ.copy()
+ child_env["DIMOS_RUN_LOG_DIR"] = str(_safe_child(output_root, "run-logs"))
+ command = [
+ sys.executable,
+ "-m",
+ "dimos.benchmark.spatiotemporal.candidate_worker",
+ "--video",
+ str(video),
+ "--questions",
+ str(questions_path),
+ "--output-root",
+ str(output_root),
+ "--duration-s",
+ str(duration_s),
+ "--result",
+ str(result_path),
+ ]
+ subprocess.run(command, check=True, cwd=Path.cwd(), env=child_env)
+ payload = json.loads(result_path.read_text(encoding="utf-8"))
+ if not isinstance(payload, dict):
+ raise ValueError("candidate worker result must be an object")
+ raw_answers = payload.get("answers")
+ runtime = payload.get("runtime")
+ if not isinstance(raw_answers, dict) or not isinstance(runtime, dict):
+ raise ValueError("candidate worker result has invalid answers or runtime")
+ expected_ids = {question.question_id for question in questions}
+ if set(raw_answers) != expected_ids:
+ raise ValueError("candidate worker answers do not match public questions")
+ for answer in raw_answers.values():
+ if answer is not None and not isinstance(answer, str | bool):
+ raise ValueError("candidate worker returned an invalid answer type")
+ return cast("dict[QuestionId, str | bool | None]", raw_answers), cast("dict[str, Any]", runtime)
+
+
+def _score_candidate(
+ bundle: EvaluationBundle,
+ candidate_answers: dict[QuestionId, str | bool | None],
+ runtime: dict[str, Any],
+) -> dict[str, Any]:
+ """Score public-only predictions in the evaluator's private boundary."""
+ oracle_by_question = {answer.question_id: answer for answer in bundle.answers}
+ report = build_evaluation_report(
+ bundle.questions,
+ oracle_by_question,
+ candidate_answers,
+ source_video_sha256=bundle.public_manifest.source_video_sha256,
+ )
+ return {
+ **runtime,
+ "by_family": {family.value: asdict(score) for family, score in report.by_family.items()},
+ "candidate_used_oracle": False,
+ "overall": asdict(report.overall),
+ "status_counts": {status.value: count for status, count in report.status_counts.items()},
+ }
+
+
+def run_demo(
+ source_video: Path,
+ output_root: Path,
+ *,
+ duration_s: int = 25,
+ frame_stride: int = 150,
+) -> dict[str, Any]:
+ """Run the real video, YOLO-E, bundle, replay, and TemporalMemory gates."""
+ if not 15 <= duration_s <= 30:
+ raise ValueError("duration must be between 15 and 30 seconds")
+ if frame_stride < 1:
+ raise ValueError("frame stride must be positive")
+ if source_video.is_symlink() or not source_video.is_file():
+ raise ValueError(f"source video must be a regular file: {source_video}")
+ output_root = _prepare_output_root(output_root)
+ video = _safe_child(output_root, "office_robot_25s.mp4")
+ summary_path = _safe_child(output_root, "summary.json")
+ if summary_path.is_dir():
+ raise ValueError("summary artifact path must not be a directory")
+ fps, frames = _trim_video(source_video, video, float(duration_s))
+ bundle, teacher = _teacher_run(video, output_root, frame_stride)
+ with TemporaryDirectory(prefix="dimos-stqa-candidate-") as candidate_directory:
+ candidate_root = Path(candidate_directory).resolve()
+ candidate_video = candidate_root / video.name
+ shutil.copyfile(video, candidate_video)
+ predictions, runtime = _run_temporal_memory_candidate(
+ candidate_video,
+ bundle.questions,
+ candidate_root,
+ duration_s,
+ )
+ candidate = _score_candidate(bundle, predictions, runtime)
+ viewer_result = write_evidence_viewer(
+ video,
+ bundle,
+ _safe_child(output_root, "evidence-viewer"),
+ )
+ summary = {
+ "candidate": candidate,
+ "review": {
+ "evidence_frame_count": viewer_result.evidence_frame_count,
+ "index_path": f"evidence-viewer/{viewer_result.index_path}",
+ "question_count": viewer_result.question_count,
+ },
+ "teacher": teacher,
+ "video": {
+ "duration_s": duration_s,
+ "fps": fps,
+ "frames": frames,
+ "path": str(video),
+ },
+ }
+ summary_path.write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n")
+ return summary
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--source-video", type=Path, default=Path("assets/simple_demo.mp4"))
+ parser.add_argument(
+ "--output-root",
+ type=Path,
+ default=Path(".artifacts/spatiotemporal-video-qa"),
+ )
+ parser.add_argument("--duration-s", type=int, default=25)
+ parser.add_argument("--frame-stride", type=int, default=150)
+ args = parser.parse_args()
+ print(
+ json.dumps(
+ run_demo(
+ args.source_video,
+ args.output_root,
+ duration_s=args.duration_s,
+ frame_stride=args.frame_stride,
+ ),
+ indent=2,
+ sort_keys=True,
+ )
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/dimos/benchmark/spatiotemporal/evidence_viewer.py b/dimos/benchmark/spatiotemporal/evidence_viewer.py
new file mode 100644
index 0000000000..a587a2b14c
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/evidence_viewer.py
@@ -0,0 +1,469 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Evaluator-only HTML and annotated-frame evidence viewer."""
+
+from collections import defaultdict
+from dataclasses import dataclass
+from hashlib import sha256
+from html import escape
+from itertools import pairwise
+from pathlib import Path
+import shutil
+from tempfile import mkdtemp
+from typing import Protocol
+
+import cv2
+import numpy as np
+
+from dimos.benchmark.spatiotemporal.models import (
+ ObjectObservation,
+ OracleAnswer,
+ Question,
+ RelationInterval,
+)
+
+
+class EvidenceBundle(Protocol):
+ """Private evaluator records required to render evidence."""
+
+ @property
+ def questions(self) -> tuple[Question, ...]: ...
+
+ @property
+ def answers(self) -> tuple[OracleAnswer, ...]: ...
+
+ @property
+ def observations(self) -> tuple[ObjectObservation, ...]: ...
+
+ @property
+ def relation_intervals(self) -> tuple[RelationInterval, ...]: ...
+
+
+@dataclass(frozen=True)
+class EvidenceViewerResult:
+ """Root-independent description of generated viewer artifacts."""
+
+ index_path: str
+ evidence_frame_count: int
+ question_count: int
+
+
+def _assert_no_symlink_components(path: Path) -> None:
+ absolute = path.absolute()
+ for component in reversed((absolute, *absolute.parents)):
+ if component.is_symlink():
+ raise ValueError(f"viewer path contains a symlink: {component}")
+
+
+def _validate_output_root(root: Path) -> None:
+ _assert_no_symlink_components(root)
+ if root.exists() and not root.is_dir():
+ raise ValueError(f"viewer output must be a directory: {root}")
+ if root.exists():
+ for path in root.rglob("*"):
+ if path.is_symlink():
+ raise ValueError(f"viewer output contains a symlink: {path}")
+
+
+def _create_staging_root(root: Path) -> Path:
+ parent = root.parent
+ _assert_no_symlink_components(parent)
+ parent.mkdir(parents=True, exist_ok=True)
+ _assert_no_symlink_components(parent)
+ staging = Path(mkdtemp(prefix=f".{root.name}.tmp-", dir=parent))
+ (staging / "frames").mkdir()
+ return staging
+
+
+def _replace_output_atomically(staging: Path, output_root: Path) -> None:
+ backup: Path | None = None
+ if output_root.exists():
+ backup = Path(mkdtemp(prefix=f".{output_root.name}.old-", dir=output_root.parent))
+ backup.rmdir()
+ output_root.rename(backup)
+ try:
+ staging.rename(output_root)
+ except Exception:
+ if backup is not None and backup.exists() and not output_root.exists():
+ backup.rename(output_root)
+ raise
+ if backup is not None:
+ shutil.rmtree(backup)
+
+
+def _frame_evidence(bundle: EvidenceBundle) -> tuple[dict[str, tuple[int, ...]], tuple[int, ...]]:
+ intervals = {interval.interval_id: interval for interval in bundle.relation_intervals}
+ evidence_by_question: dict[str, tuple[int, ...]] = {}
+ all_frames: set[int] = set()
+ for answer in bundle.answers:
+ frames = set(answer.evidence_frame_ids)
+ for interval_id in answer.evidence_interval_ids:
+ try:
+ frames.update(intervals[interval_id].evidence_frame_ids)
+ except KeyError as error:
+ raise ValueError(f"unknown evidence interval: {interval_id}") from error
+ ordered = tuple(sorted(frames))
+ evidence_by_question[answer.question_id] = ordered
+ all_frames.update(ordered)
+ return evidence_by_question, tuple(sorted(all_frames))
+
+
+def _object_color(object_id: str) -> tuple[int, int, int]:
+ digest = sha256(object_id.encode("utf-8")).digest()
+ return tuple(64 + component % 160 for component in digest[:3]) # type: ignore[return-value]
+
+
+def _annotate_frame(frame: np.ndarray, observations: tuple[ObjectObservation, ...]) -> np.ndarray:
+ annotated = frame.copy()
+ height, width = annotated.shape[:2]
+ for observation in sorted(observations, key=lambda item: item.object_id):
+ x_min = min(width - 1, max(0, round(observation.box.x_min * width)))
+ y_min = min(height - 1, max(0, round(observation.box.y_min * height)))
+ x_max = min(width - 1, max(0, round(observation.box.x_max * width)))
+ y_max = min(height - 1, max(0, round(observation.box.y_max * height)))
+ color = _object_color(observation.object_id)
+ cv2.rectangle(annotated, (x_min, y_min), (x_max, y_max), color, 2)
+ label = f"{observation.object_id}: {observation.label} {observation.confidence:.2f}"
+ cv2.putText(
+ annotated,
+ label,
+ (x_min, max(12, y_min - 4)),
+ cv2.FONT_HERSHEY_SIMPLEX,
+ 0.4,
+ color,
+ 1,
+ cv2.LINE_AA,
+ )
+ return annotated
+
+
+def _write_evidence_frames(
+ video_path: Path,
+ output_root: Path,
+ frame_ids: tuple[int, ...],
+ observations: tuple[ObjectObservation, ...],
+) -> None:
+ if not frame_ids:
+ return
+ observations_by_frame: dict[int, list[ObjectObservation]] = defaultdict(list)
+ for observation in observations:
+ observations_by_frame[observation.frame_id].append(observation)
+
+ wanted = set(frame_ids)
+ capture = cv2.VideoCapture(str(video_path))
+ written: set[int] = set()
+ try:
+ if not capture.isOpened():
+ raise RuntimeError(f"failed to open evidence video: {video_path}")
+ frame_id = 0
+ while frame_id <= frame_ids[-1]:
+ decoded, frame = capture.read()
+ if not decoded or frame is None:
+ raise RuntimeError(f"failed to decode evidence frame {frame_id}")
+ if frame_id in wanted:
+ annotated = _annotate_frame(frame, tuple(observations_by_frame.get(frame_id, ())))
+ encoded, content = cv2.imencode(".jpg", annotated, [cv2.IMWRITE_JPEG_QUALITY, 90])
+ if not encoded:
+ raise RuntimeError(f"failed to encode evidence frame {frame_id}")
+ (output_root / "frames" / f"frame_{frame_id:06d}.jpg").write_bytes(
+ content.tobytes()
+ )
+ written.add(frame_id)
+ frame_id += 1
+ finally:
+ capture.release()
+ if written != wanted:
+ missing = sorted(wanted - written)
+ raise RuntimeError(f"missing evidence frames: {missing}")
+
+
+def _render_html(
+ bundle: EvidenceBundle,
+ evidence_by_question: dict[str, tuple[int, ...]],
+) -> str:
+ answers = {answer.question_id: answer for answer in bundle.answers}
+ observations_by_frame: dict[int, list[ObjectObservation]] = defaultdict(list)
+ for observation in bundle.observations:
+ observations_by_frame[observation.frame_id].append(observation)
+
+ questions = sorted(bundle.questions, key=lambda item: item.question_id)
+ evidence_frame_ids = sorted(
+ {frame_id for frame_ids in evidence_by_question.values() for frame_id in frame_ids}
+ )
+ episode_id = questions[0].episode_id if questions else "unknown"
+ object_ids = sorted({observation.object_id for observation in bundle.observations})
+ spatial_count = sum(question.question_kind.value == "spatial" for question in questions)
+ temporal_count = len(questions) - spatial_count
+ positive_count = sum(answer.expected for answer in bundle.answers)
+
+ timestamps_by_frame: dict[int, float] = {}
+ for observation in bundle.observations:
+ timestamps_by_frame.setdefault(observation.frame_id, observation.timestamp_s)
+ for interval in bundle.relation_intervals:
+ timestamps_by_frame.setdefault(interval.start_frame_id, interval.start_timestamp_s)
+ timestamps_by_frame.setdefault(interval.end_frame_id, interval.end_timestamp_s)
+
+ relations_by_frame: dict[int, list[str]] = defaultdict(list)
+ for interval in bundle.relation_intervals:
+ relation = (
+ f"{interval.subject_id} {interval.predicate.value.replace('-', ' ')} "
+ f"{interval.object_id}"
+ )
+ for frame_id in interval.evidence_frame_ids:
+ relations_by_frame[frame_id].append(relation)
+
+ robot_observations = [
+ observation
+ for observation in bundle.observations
+ if "robot" in observation.label.casefold()
+ ]
+ robot_ids = sorted({observation.object_id for observation in robot_observations})
+ robot_frames = sorted({observation.frame_id for observation in robot_observations})
+ robot_coverage = len(robot_frames) / len(evidence_frame_ids) if evidence_frame_ids else 0.0
+ referenced_interval_ids = {
+ interval_id for answer in bundle.answers for interval_id in answer.evidence_interval_ids
+ }
+ displayed_frames = set(evidence_frame_ids)
+ linked_intervals = [
+ interval
+ for interval in bundle.relation_intervals
+ if interval.interval_id in referenced_interval_ids
+ and bool(displayed_frames.intersection(interval.evidence_frame_ids))
+ ]
+ relation_status = (
+ "pass"
+ if linked_intervals and len(linked_intervals) == len(bundle.relation_intervals)
+ else "review"
+ )
+ ordered_intervals = sorted(
+ linked_intervals,
+ key=lambda interval: (interval.start_timestamp_s, interval.interval_id),
+ )
+ strict_order = len(ordered_intervals) > 1 and all(
+ left.end_frame_id < right.start_frame_id and left.end_timestamp_s < right.start_timestamp_s
+ for left, right in pairwise(ordered_intervals)
+ )
+ proof_question = next(
+ (
+ question
+ for question in questions
+ if question.question_kind.value == "spatial"
+ and answers[question.question_id].expected
+ and evidence_by_question.get(question.question_id)
+ ),
+ None,
+ )
+ if proof_question is not None and len(proof_question.object_ids) == 2:
+ proof_frame = evidence_by_question[proof_question.question_id][0]
+ subject_id, object_id = proof_question.object_ids
+ proof_relation = (
+ f"{subject_id} {proof_question.predicate.value.replace('-', ' ')} {object_id}"
+ )
+ proof_chain = (
+ f"
01 · evidenceframe {proof_frame}
"
+ f"02 · derived fact{escape(proof_relation)}
"
+ f"03 · public test{escape(proof_question.text)}
"
+ f"04 · private oracletrue · evidence f{proof_frame}
"
+ )
+ else:
+ proof_chain = "No positive spatial proof is available.
"
+
+ rows: list[str] = []
+ for question in questions:
+ try:
+ answer = answers[question.question_id]
+ except KeyError as error:
+ raise ValueError(
+ f"question is missing oracle answer: {question.question_id}"
+ ) from error
+ frame_ids = evidence_by_question.get(question.question_id, ())
+ links = " ".join(
+ f'f{frame_id}'
+ for frame_id in frame_ids
+ )
+ expected = str(answer.expected).lower()
+ rows.append(
+ f''
+ f'| '
+ f"{question.question_kind.value} | "
+ f"{escape(question.text)}"
+ f'…{escape(question.question_id[-10:])} | '
+ f'{expected} | '
+ f"{links or 'none'} | "
+ "
"
+ )
+
+ timeline: list[str] = []
+ cards: list[str] = []
+ for frame_id in evidence_frame_ids:
+ observations = sorted(observations_by_frame[frame_id], key=lambda item: item.object_id)
+ timestamp_s = timestamps_by_frame.get(frame_id)
+ timestamp = f"{timestamp_s:.3f}s" if timestamp_s is not None else "time unknown"
+ relations = (
+ "".join(
+ f'{escape(relation)}'
+ for relation in sorted(relations_by_frame[frame_id])
+ )
+ or 'no accepted relation'
+ )
+ timeline.append(
+ f'"
+ )
+ labels = (
+ ", ".join(
+ f"#{observation.object_id} {observation.label} ({observation.confidence:.2f})"
+ for observation in observations
+ )
+ or "No accepted detections"
+ )
+ cards.append(
+ f''
+ f''
+ f"{timestamp}frame {frame_id}
"
+ f"{escape(labels)}
{relations}
"
+ ""
+ )
+
+ identity_status = "pass" if len(robot_ids) == 1 else "review"
+ identity_detail = (
+ f"one robot track ({robot_ids[0]})"
+ if len(robot_ids) == 1
+ else (
+ f"robot label spans IDs {', '.join(robot_ids)}"
+ if robot_ids
+ else "no robot-labeled detection"
+ )
+ )
+ coverage_status = "pass" if robot_coverage >= 0.8 else "review"
+ if relation_status == "review" or not strict_order:
+ motion_verdict = "Motion evidence incomplete; review required"
+ elif identity_status == "review" or coverage_status == "review":
+ motion_verdict = "Relation-order eval ready; motion claim needs review"
+ else:
+ motion_verdict = "Motion evidence ready for review"
+
+ return (
+ """
+
+
+
+
+Video → Eval evidence
+
+
+
+
+Evaluator-only review surface
Video → relationship eval evidence
Trace how sampled robot-video observations become spatial and temporal tests, then inspect the exact frames supporting each oracle answer.
EPISODE"""
+ + escape(episode_id)
+ + """
+
+"""
+ + str(len(evidence_frame_ids))
+ + """evidence frames
"""
+ + str(len(object_ids))
+ + """tracked IDs
"""
+ + str(len(bundle.relation_intervals))
+ + """relation intervals
"""
+ + str(spatial_count)
+ + """ / """
+ + str(temporal_count)
+ + """spatial / temporal
"""
+ + str(positive_count)
+ + """ / """
+ + str(len(bundle.answers) - positive_count)
+ + """true / false
+What the system does
Turns observed relations into replayable tests
01Video
02Sample
03Track
04Relations
05Questions
06Score + evidence
+One generated proof chain
From a sampled frame to a scored evaluation
"""
+ + proof_chain
+ + """
+Robot-motion verification
"""
+ + escape(motion_verdict)
+ + """
"""
+ + relation_status
+ + """Relation events derived
"""
+ + f"{len(linked_intervals)}/{len(bundle.relation_intervals)}"
+ + """ intervals are referenced by oracle answers and linked to displayed frames.
"""
+ + ("pass" if strict_order else "review")
+ + """Temporal order
Strict non-overlap across the displayed relation sequence.
"""
+ + identity_status
+ + """Robot identity continuity
"""
+ + escape(identity_detail)
+ + """.
"""
+ + coverage_status
+ + """Robot detection coverage
"""
+ + f"{len(robot_frames)}/{len(evidence_frame_ids)} evidence frames ({robot_coverage:.0%})"
+ + """.
+Relation timeline
What changed, and when
Each stop is a sampled evidence frame. Relation labels come from private accepted intervals; gaps remain visible rather than interpolated.
"""
+ + "".join(timeline)
+ + """
+Future-project leverage
Reusable motion-evaluation patterns
The same contracts can evaluate new videos without changing candidate/oracle separation.
Locomotion & patrolVerify before/after ordering around tracked landmarks and objects.
ManipulationTurn above/below and left/right object transitions into regression cases.
Long-horizon memoryTest whether a candidate recalls relation events separated in time.
Model releasesReplay frozen bundles and block spatial or temporal regressions.
+Visual evidence
Annotated sampled frames
Click a frame to inspect full resolution. Colors are stable per object ID.
"""
+ + "".join(cards)
+ + """
+Generated evaluation
Questions and private oracle
Filter by family or expected value; evidence links jump back to the supporting frames.
| Family | Question | Expected | Evidence |
"""
+ + "".join(rows)
+ + """
+
+"""
+ )
+
+
+def write_evidence_viewer(
+ video_path: Path,
+ bundle: EvidenceBundle,
+ output_root: Path,
+) -> EvidenceViewerResult:
+ """Write an evaluator-only HTML report linked to annotated private evidence."""
+ _assert_no_symlink_components(video_path)
+ if not video_path.is_file():
+ raise ValueError(f"evidence video must be a regular file: {video_path}")
+ _validate_output_root(output_root)
+ if video_path.resolve().is_relative_to(output_root.resolve(strict=False)):
+ raise ValueError("evidence video cannot be inside viewer output")
+
+ evidence_by_question, frame_ids = _frame_evidence(bundle)
+ html = _render_html(bundle, evidence_by_question)
+ staging = _create_staging_root(output_root)
+ try:
+ _write_evidence_frames(video_path, staging, frame_ids, bundle.observations)
+ (staging / "index.html").write_text(html, encoding="utf-8")
+ _replace_output_atomically(staging, output_root)
+ finally:
+ if staging.exists():
+ shutil.rmtree(staging)
+ return EvidenceViewerResult(
+ index_path="index.html",
+ evidence_frame_count=len(frame_ids),
+ question_count=len(bundle.questions),
+ )
diff --git a/dimos/benchmark/spatiotemporal/generation.py b/dimos/benchmark/spatiotemporal/generation.py
new file mode 100644
index 0000000000..e5a993df18
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/generation.py
@@ -0,0 +1,228 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Deterministic public question generation from accepted relation facts."""
+
+from collections.abc import Sequence
+from itertools import combinations, product
+
+from dimos.benchmark.spatiotemporal.models import (
+ OracleAnswer,
+ Question,
+ QuestionKind,
+ RelationFact,
+ RelationInterval,
+ SpatialPredicate,
+ TemporalPredicate,
+)
+from dimos.benchmark.spatiotemporal.utilities import stable_question_id
+
+_OPPOSITE_SPATIAL_PREDICATE = {
+ SpatialPredicate.LEFT_OF: SpatialPredicate.RIGHT_OF,
+ SpatialPredicate.RIGHT_OF: SpatialPredicate.LEFT_OF,
+ SpatialPredicate.ABOVE: SpatialPredicate.BELOW,
+ SpatialPredicate.BELOW: SpatialPredicate.ABOVE,
+}
+
+
+def _canonical_spatial_relation(
+ subject_id: str, predicate: SpatialPredicate, object_id: str
+) -> tuple[str, SpatialPredicate, str]:
+ if predicate is SpatialPredicate.RIGHT_OF:
+ return object_id, SpatialPredicate.LEFT_OF, subject_id
+ if predicate is SpatialPredicate.BELOW:
+ return object_id, SpatialPredicate.ABOVE, subject_id
+ return subject_id, predicate, object_id
+
+
+def _describe_relation(interval: RelationInterval) -> str:
+ return (
+ f"{interval.subject_id} {interval.predicate.value.replace('-', ' ')} {interval.object_id}"
+ )
+
+
+def generate_spatial_question_cases(
+ facts: Sequence[RelationFact],
+ *,
+ sample_frame_ids: Sequence[int] | None = None,
+) -> tuple[tuple[Question, OracleAnswer], ...]:
+ """Generate episode-level positives and only globally absent negative controls."""
+ if len({fact.episode_id for fact in facts}) > 1:
+ raise ValueError("spatial questions must be generated from one episode")
+
+ grouped: dict[tuple[str, str, SpatialPredicate, str], set[int]] = {}
+ for fact in facts:
+ subject_id, predicate, object_id = _canonical_spatial_relation(
+ fact.subject_id, fact.predicate, fact.object_id
+ )
+ key = (fact.episode_id, subject_id, predicate, object_id)
+ grouped.setdefault(key, set()).update(fact.evidence_frame_ids)
+ episode_evidence = tuple(
+ sorted(
+ set(sample_frame_ids)
+ if sample_frame_ids is not None
+ else {frame_id for evidence in grouped.values() for frame_id in evidence}
+ )
+ )
+
+ cases: dict[str, tuple[Question, OracleAnswer]] = {}
+ for (episode_id, subject_id, accepted_predicate, object_id), evidence in grouped.items():
+ question_specs = [(accepted_predicate, True, tuple(sorted(evidence)))]
+ inverse_key = (episode_id, object_id, accepted_predicate, subject_id)
+ if inverse_key not in grouped:
+ question_specs.append(
+ (
+ _OPPOSITE_SPATIAL_PREDICATE[accepted_predicate],
+ False,
+ episode_evidence,
+ )
+ )
+ for predicate, expected, evidence_frame_ids in question_specs:
+ object_ids = (subject_id, object_id)
+ question_id = stable_question_id(
+ episode_id=episode_id,
+ object_ids=object_ids,
+ predicate=predicate.value,
+ question_kind=QuestionKind.SPATIAL.value,
+ )
+ cases[question_id] = (
+ Question(
+ question_id=question_id,
+ episode_id=episode_id,
+ text=(
+ f"Did {subject_id} ever appear "
+ f"{predicate.value.replace('-', ' ')} {object_id}?"
+ ),
+ question_kind=QuestionKind.SPATIAL,
+ predicate=predicate,
+ object_ids=object_ids,
+ ),
+ OracleAnswer(
+ question_id=question_id,
+ expected=expected,
+ evidence_frame_ids=evidence_frame_ids,
+ ),
+ )
+ return tuple(cases[question_id] for question_id in sorted(cases))
+
+
+def generate_spatial_questions(facts: Sequence[RelationFact]) -> tuple[Question, ...]:
+ """Generate one answer-free public question per accepted spatial relation."""
+ return tuple(
+ question for question, answer in generate_spatial_question_cases(facts) if answer.expected
+ )
+
+
+def generate_temporal_question_cases(
+ intervals: Sequence[RelationInterval],
+) -> tuple[tuple[Question, OracleAnswer], ...]:
+ """Generate balanced temporal cases with private interval-backed truth."""
+ if len({interval.episode_id for interval in intervals}) > 1:
+ raise ValueError("temporal questions must be generated from one episode")
+
+ by_relation: dict[str, list[RelationInterval]] = {}
+ for interval in intervals:
+ by_relation.setdefault(interval.relation_id, []).append(interval)
+
+ proofs: dict[tuple[str, str], list[tuple[RelationInterval, RelationInterval]]] = {}
+ for left_id, right_id in combinations(sorted(by_relation), 2):
+ interval_proofs: list[tuple[RelationInterval, RelationInterval]] = []
+ orientations: set[tuple[str, str]] = set()
+ for left, right in product(by_relation[left_id], by_relation[right_id]):
+ if (
+ left.end_frame_id < right.start_frame_id
+ and left.end_timestamp_s < right.start_timestamp_s
+ ):
+ first, second = left, right
+ elif (
+ right.end_frame_id < left.start_frame_id
+ and right.end_timestamp_s < left.start_timestamp_s
+ ):
+ first, second = right, left
+ else:
+ interval_proofs = []
+ break
+ interval_proofs.append((first, second))
+ orientations.add((first.relation_id, second.relation_id))
+ if interval_proofs and len(orientations) == 1:
+ proofs[(left_id, right_id)] = interval_proofs
+
+ cases: dict[str, tuple[Question, OracleAnswer]] = {}
+ for relation_pair in sorted(proofs):
+ interval_proofs = proofs[relation_pair]
+ first, second = min(
+ interval_proofs,
+ key=lambda proof: (proof[0].interval_id, proof[1].interval_id),
+ )
+ descriptions = {
+ first.relation_id: _describe_relation(first),
+ second.relation_id: _describe_relation(second),
+ }
+ for predicate, references, expected in (
+ (
+ TemporalPredicate.BEFORE,
+ (first.relation_id, second.relation_id),
+ True,
+ ),
+ (
+ TemporalPredicate.AFTER,
+ (second.relation_id, first.relation_id),
+ True,
+ ),
+ (
+ TemporalPredicate.AFTER,
+ (first.relation_id, second.relation_id),
+ False,
+ ),
+ (
+ TemporalPredicate.BEFORE,
+ (second.relation_id, first.relation_id),
+ False,
+ ),
+ ):
+ question_id = stable_question_id(
+ episode_id=first.episode_id,
+ predicate=predicate.value,
+ question_kind=QuestionKind.TEMPORAL.value,
+ reference_ids=references,
+ )
+ question = Question(
+ question_id=question_id,
+ episode_id=first.episode_id,
+ text=(
+ f'Did "{descriptions[references[0]]}" happen {predicate.value} '
+ f'"{descriptions[references[1]]}"?'
+ ),
+ question_kind=QuestionKind.TEMPORAL,
+ predicate=predicate,
+ object_ids=(),
+ reference_ids=references,
+ )
+ cases[question_id] = (
+ question,
+ OracleAnswer(
+ question_id=question_id,
+ expected=expected,
+ evidence_interval_ids=(first.interval_id, second.interval_id),
+ ),
+ )
+
+ return tuple(cases[question_id] for question_id in sorted(cases))
+
+
+def generate_temporal_questions(
+ intervals: Sequence[RelationInterval],
+) -> tuple[Question, ...]:
+ """Generate public questions for unambiguous strict interval orderings."""
+ return tuple(question for question, _ in generate_temporal_question_cases(intervals))
diff --git a/dimos/benchmark/spatiotemporal/intervals.py b/dimos/benchmark/spatiotemporal/intervals.py
new file mode 100644
index 0000000000..bfe9a69199
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/intervals.py
@@ -0,0 +1,169 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Deterministic construction of relation intervals."""
+
+from collections.abc import Sequence
+from itertools import pairwise
+
+from dimos.benchmark.spatiotemporal.models import (
+ RelationFact,
+ RelationId,
+ RelationInterval,
+ TemporalPredicate,
+)
+from dimos.benchmark.spatiotemporal.utilities import SCHEMA_VERSION, JsonValue, stable_id
+
+
+def _validate_sample_schedules(facts: Sequence[RelationFact]) -> None:
+ by_episode: dict[str, dict[int, float]] = {}
+ timestamps_by_episode: dict[str, dict[float, int]] = {}
+ for fact in facts:
+ frames = by_episode.setdefault(fact.episode_id, {})
+ timestamps = timestamps_by_episode.setdefault(fact.episode_id, {})
+ if (fact.frame_id in frames and frames[fact.frame_id] != fact.timestamp_s) or (
+ fact.timestamp_s in timestamps and timestamps[fact.timestamp_s] != fact.frame_id
+ ):
+ raise ValueError("conflicting sample schedule")
+ frames[fact.frame_id] = fact.timestamp_s
+ timestamps[fact.timestamp_s] = fact.frame_id
+
+ for frames in by_episode.values():
+ ordered_samples = sorted(frames.items())
+ if any(
+ current_timestamp >= next_timestamp
+ for (_, current_timestamp), (_, next_timestamp) in pairwise(ordered_samples)
+ ):
+ raise ValueError("conflicting sample schedule")
+
+
+def _build_interval(facts: Sequence[RelationFact]) -> RelationInterval:
+ first = facts[0]
+ last = facts[-1]
+ preimage: dict[str, JsonValue] = {
+ "end_frame_id": last.frame_id,
+ "end_timestamp_s": last.timestamp_s,
+ "episode_id": first.episode_id,
+ "object_id": first.object_id,
+ "predicate": first.predicate.value,
+ "relation_id": first.relation_id,
+ "schema_version": SCHEMA_VERSION,
+ "start_frame_id": first.frame_id,
+ "start_timestamp_s": first.timestamp_s,
+ "subject_id": first.subject_id,
+ }
+ return RelationInterval(
+ interval_id=stable_id("interval", preimage),
+ relation_id=first.relation_id,
+ episode_id=first.episode_id,
+ subject_id=first.subject_id,
+ predicate=first.predicate,
+ object_id=first.object_id,
+ start_frame_id=first.frame_id,
+ end_frame_id=last.frame_id,
+ start_timestamp_s=first.timestamp_s,
+ end_timestamp_s=last.timestamp_s,
+ evidence_frame_ids=tuple(fact.frame_id for fact in facts),
+ )
+
+
+def _sample_positions(
+ facts: Sequence[RelationFact], sample_schedule: Sequence[tuple[int, float]]
+) -> dict[int, int]:
+ if any(
+ current_frame >= next_frame or current_timestamp >= next_timestamp
+ for (current_frame, current_timestamp), (next_frame, next_timestamp) in pairwise(
+ sample_schedule
+ )
+ ):
+ raise ValueError("conflicting sample schedule")
+ schedule_by_frame = dict(sample_schedule)
+ if len(schedule_by_frame) != len(sample_schedule):
+ raise ValueError("conflicting sample schedule")
+ if any(schedule_by_frame.get(fact.frame_id) != fact.timestamp_s for fact in facts):
+ raise ValueError("fact is absent from sample schedule")
+ return {frame_id: index for index, (frame_id, _) in enumerate(sample_schedule)}
+
+
+def build_relation_intervals(
+ facts: Sequence[RelationFact],
+ *,
+ sample_schedule: Sequence[tuple[int, float]] | None = None,
+) -> tuple[RelationInterval, ...]:
+ """Coalesce stable relations across adjacent sampled frames."""
+ _validate_sample_schedules(facts)
+ sample_positions = (
+ _sample_positions(facts, sample_schedule) if sample_schedule is not None else None
+ )
+ ordered = sorted(
+ facts,
+ key=lambda fact: (
+ fact.episode_id,
+ fact.relation_id,
+ fact.frame_id,
+ fact.timestamp_s,
+ ),
+ )
+ groups: list[list[RelationFact]] = []
+ for fact in ordered:
+ previous = groups[-1][-1] if groups else None
+ adjacent = (
+ sample_positions[fact.frame_id] == sample_positions[previous.frame_id] + 1
+ if sample_positions is not None and previous is not None
+ else previous is not None and fact.frame_id == previous.frame_id + 1
+ )
+ if (
+ previous is not None
+ and fact.episode_id == previous.episode_id
+ and fact.relation_id == previous.relation_id
+ and adjacent
+ ):
+ groups[-1].append(fact)
+ else:
+ groups.append([fact])
+ return tuple(_build_interval(group) for group in groups)
+
+
+def derive_temporal_predicate(
+ first_relation_id: RelationId,
+ second_relation_id: RelationId,
+ intervals: Sequence[RelationInterval],
+) -> TemporalPredicate | None:
+ """Return a strict order only when all matching interval evidence agrees."""
+ first = [interval for interval in intervals if interval.relation_id == first_relation_id]
+ second = [interval for interval in intervals if interval.relation_id == second_relation_id]
+ first_episodes = {interval.episode_id for interval in first}
+ second_episodes = {interval.episode_id for interval in second}
+ if not first or not second or first_episodes != second_episodes:
+ return None
+
+ predicates: set[TemporalPredicate] = set()
+ for first_interval in first:
+ for second_interval in second:
+ if first_interval.episode_id != second_interval.episode_id:
+ continue
+ if (
+ first_interval.end_frame_id < second_interval.start_frame_id
+ and first_interval.end_timestamp_s < second_interval.start_timestamp_s
+ ):
+ predicates.add(TemporalPredicate.BEFORE)
+ elif (
+ second_interval.end_frame_id < first_interval.start_frame_id
+ and second_interval.end_timestamp_s < first_interval.start_timestamp_s
+ ):
+ predicates.add(TemporalPredicate.AFTER)
+ else:
+ return None
+
+ return predicates.pop() if len(predicates) == 1 else None
diff --git a/dimos/benchmark/spatiotemporal/models.py b/dimos/benchmark/spatiotemporal/models.py
new file mode 100644
index 0000000000..d5dad60db4
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/models.py
@@ -0,0 +1,396 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Strict data contracts for spatiotemporal video QA."""
+
+from enum import StrEnum
+from pathlib import PurePosixPath
+from typing import Annotated, Self
+import unicodedata
+
+from pydantic import AfterValidator, BaseModel, ConfigDict, Field, model_validator
+
+from dimos.benchmark.spatiotemporal.utilities import (
+ SCHEMA_VERSION,
+ stable_id,
+ stable_question_id,
+)
+
+
+def _require_nfc(value: str) -> str:
+ try:
+ value.encode("utf-8")
+ except UnicodeEncodeError as error:
+ raise ValueError("string fields must contain valid Unicode scalar values") from error
+ if not unicodedata.is_normalized("NFC", value):
+ raise ValueError("string fields must be NFC-normalized")
+ return value
+
+
+NonEmptyString = Annotated[str, Field(min_length=1), AfterValidator(_require_nfc)]
+QuestionId = Annotated[
+ str,
+ Field(pattern=r"^question_[0-9a-f]{64}$"),
+ AfterValidator(_require_nfc),
+]
+RelationId = Annotated[
+ str,
+ Field(pattern=r"^relation_[0-9a-f]{64}$"),
+ AfterValidator(_require_nfc),
+]
+IntervalId = Annotated[
+ str,
+ Field(pattern=r"^interval_[0-9a-f]{64}$"),
+ AfterValidator(_require_nfc),
+]
+BundleId = Annotated[
+ str,
+ Field(pattern=r"^bundle_[0-9a-f]{64}$"),
+ AfterValidator(_require_nfc),
+]
+Sha256Digest = Annotated[str, Field(pattern=r"^[0-9a-f]{64}$")]
+
+
+class SpatialPredicate(StrEnum):
+ """Supported image-plane predicates."""
+
+ LEFT_OF = "left-of"
+ RIGHT_OF = "right-of"
+ ABOVE = "above"
+ BELOW = "below"
+
+
+class TemporalPredicate(StrEnum):
+ """Supported strict interval predicates."""
+
+ BEFORE = "before"
+ AFTER = "after"
+
+
+class QuestionKind(StrEnum):
+ """Supported public question categories."""
+
+ SPATIAL = "spatial"
+ TEMPORAL = "temporal"
+
+
+class PredictionStatus(StrEnum):
+ """Closed set of candidate scoring outcomes."""
+
+ CORRECT = "correct"
+ INCORRECT = "incorrect"
+ MISSING = "missing"
+ INVALID = "invalid"
+
+
+class StrictFrozenModel(BaseModel):
+ """Base for immutable benchmark records with no implicit coercion."""
+
+ model_config = ConfigDict(extra="forbid", frozen=True, strict=True)
+
+
+class BoundingBox2D(StrictFrozenModel):
+ """A normalized image-plane bounding box."""
+
+ x_min: float = Field(ge=0.0, le=1.0, allow_inf_nan=False)
+ y_min: float = Field(ge=0.0, le=1.0, allow_inf_nan=False)
+ x_max: float = Field(ge=0.0, le=1.0, allow_inf_nan=False)
+ y_max: float = Field(ge=0.0, le=1.0, allow_inf_nan=False)
+
+ @model_validator(mode="after")
+ def validate_bounds(self) -> Self:
+ """Reject empty or inverted boxes."""
+ if self.x_min >= self.x_max or self.y_min >= self.y_max:
+ raise ValueError("bounding box minimums must be below maximums")
+ return self
+
+
+class ObjectObservation(StrictFrozenModel):
+ """One object's canonical observation in a sampled video frame."""
+
+ episode_id: NonEmptyString
+ frame_id: int = Field(ge=0)
+ timestamp_s: float = Field(allow_inf_nan=False)
+ object_id: NonEmptyString
+ label: NonEmptyString
+ box: BoundingBox2D
+ confidence: float = Field(ge=0.0, le=1.0, allow_inf_nan=False)
+
+
+class RelationFact(StrictFrozenModel):
+ """One private accepted spatial relation at a sampled frame."""
+
+ relation_id: RelationId
+ episode_id: NonEmptyString
+ frame_id: int = Field(ge=0)
+ timestamp_s: float = Field(allow_inf_nan=False)
+ subject_id: NonEmptyString
+ predicate: SpatialPredicate
+ object_id: NonEmptyString
+ evidence_frame_ids: tuple[int, ...] = Field(min_length=1)
+
+ @model_validator(mode="after")
+ def validate_relation_contract(self) -> Self:
+ if self.subject_id == self.object_id:
+ raise ValueError("relation subject and object IDs must differ")
+ if any(frame_id < 0 for frame_id in self.evidence_frame_ids):
+ raise ValueError("evidence frame IDs must be non-negative")
+ if tuple(sorted(set(self.evidence_frame_ids))) != self.evidence_frame_ids:
+ raise ValueError("evidence frame IDs must be ordered and unique")
+ if self.evidence_frame_ids != (self.frame_id,):
+ raise ValueError("a relation fact must cite exactly its own sample frame")
+ expected_id = stable_id(
+ "relation",
+ {
+ "object_ids": (self.subject_id, self.object_id),
+ "predicate": self.predicate.value,
+ "schema_version": SCHEMA_VERSION,
+ },
+ )
+ if self.relation_id != expected_id:
+ raise ValueError("relation ID does not match its executable contract")
+ return self
+
+
+class RelationInterval(StrictFrozenModel):
+ """One private bounded interval of a stable accepted relation."""
+
+ interval_id: IntervalId
+ relation_id: RelationId
+ episode_id: NonEmptyString
+ subject_id: NonEmptyString
+ predicate: SpatialPredicate
+ object_id: NonEmptyString
+ start_frame_id: int = Field(ge=0)
+ end_frame_id: int = Field(ge=0)
+ start_timestamp_s: float = Field(allow_inf_nan=False)
+ end_timestamp_s: float = Field(allow_inf_nan=False)
+ evidence_frame_ids: tuple[int, ...] = Field(min_length=1)
+
+ @model_validator(mode="after")
+ def validate_interval_contract(self) -> Self:
+ if self.subject_id == self.object_id:
+ raise ValueError("relation subject and object IDs must differ")
+ if self.start_frame_id > self.end_frame_id:
+ raise ValueError("interval frame bounds must be ordered")
+ if self.start_timestamp_s > self.end_timestamp_s:
+ raise ValueError("interval timestamp bounds must be ordered")
+ if (self.start_frame_id == self.end_frame_id) != (
+ self.start_timestamp_s == self.end_timestamp_s
+ ):
+ raise ValueError("interval frame and timestamp bounds must identify the same samples")
+ if tuple(sorted(set(self.evidence_frame_ids))) != self.evidence_frame_ids:
+ raise ValueError("evidence frame IDs must be ordered and unique")
+ if self.evidence_frame_ids[0] != self.start_frame_id:
+ raise ValueError("first evidence frame must equal interval start")
+ if self.evidence_frame_ids[-1] != self.end_frame_id:
+ raise ValueError("last evidence frame must equal interval end")
+ expected_relation_id = stable_id(
+ "relation",
+ {
+ "object_ids": (self.subject_id, self.object_id),
+ "predicate": self.predicate.value,
+ "schema_version": SCHEMA_VERSION,
+ },
+ )
+ if self.relation_id != expected_relation_id:
+ raise ValueError("relation ID does not match interval relation identity")
+ expected_interval_id = stable_id(
+ "interval",
+ {
+ "end_frame_id": self.end_frame_id,
+ "end_timestamp_s": self.end_timestamp_s,
+ "episode_id": self.episode_id,
+ "object_id": self.object_id,
+ "predicate": self.predicate.value,
+ "relation_id": self.relation_id,
+ "schema_version": SCHEMA_VERSION,
+ "start_frame_id": self.start_frame_id,
+ "start_timestamp_s": self.start_timestamp_s,
+ "subject_id": self.subject_id,
+ },
+ )
+ if self.interval_id != expected_interval_id:
+ raise ValueError("interval ID does not match its executable contract")
+ return self
+
+
+class Question(StrictFrozenModel):
+ """A public spatial or temporal question with an answer-free contract."""
+
+ question_id: QuestionId
+ episode_id: NonEmptyString
+ text: NonEmptyString
+ question_kind: QuestionKind
+ predicate: SpatialPredicate | TemporalPredicate
+ object_ids: tuple[NonEmptyString, NonEmptyString] | tuple[()]
+ reference_ids: tuple[RelationId, RelationId] | tuple[()] = ()
+
+ @model_validator(mode="after")
+ def validate_executable_contract(self) -> Self:
+ """Enforce disjoint variants and IDs derived only from public semantics."""
+ if self.question_kind is QuestionKind.SPATIAL:
+ if not isinstance(self.predicate, SpatialPredicate):
+ raise ValueError("spatial questions require a spatial predicate")
+ if len(self.object_ids) != 2 or self.object_ids[0] == self.object_ids[1]:
+ raise ValueError("spatial question object IDs must differ")
+ if self.reference_ids:
+ raise ValueError("spatial questions cannot contain relation references")
+ else:
+ if not isinstance(self.predicate, TemporalPredicate):
+ raise ValueError("temporal questions require a temporal predicate")
+ if self.object_ids:
+ raise ValueError("temporal questions cannot contain object IDs")
+ if len(self.reference_ids) != 2 or self.reference_ids[0] == self.reference_ids[1]:
+ raise ValueError("temporal question relation references must differ")
+ expected_id = stable_question_id(
+ episode_id=self.episode_id,
+ object_ids=self.object_ids,
+ predicate=self.predicate.value,
+ question_kind=self.question_kind.value,
+ reference_ids=self.reference_ids,
+ )
+ if self.question_id != expected_id:
+ raise ValueError("question ID does not match its executable contract")
+ return self
+
+
+class OracleAnswer(StrictFrozenModel):
+ """A private expected answer and its teacher evidence."""
+
+ question_id: QuestionId
+ expected: bool
+ evidence_frame_ids: tuple[int, ...] = ()
+ evidence_interval_ids: tuple[IntervalId, ...] = ()
+
+ @model_validator(mode="after")
+ def validate_evidence(self) -> Self:
+ if any(frame_id < 0 for frame_id in self.evidence_frame_ids):
+ raise ValueError("evidence frame IDs must be non-negative")
+ if tuple(sorted(set(self.evidence_frame_ids))) != self.evidence_frame_ids:
+ raise ValueError("evidence frame IDs must be ordered and unique")
+ if len(set(self.evidence_interval_ids)) != len(self.evidence_interval_ids):
+ raise ValueError("evidence interval IDs must be unique")
+ if not self.evidence_frame_ids and not self.evidence_interval_ids:
+ raise ValueError("oracle answers require private evidence")
+ return self
+
+
+class Prediction(StrictFrozenModel):
+ """One candidate's typed Boolean answer."""
+
+ question_id: QuestionId
+ answer: bool
+
+
+class QuestionResult(StrictFrozenModel):
+ """Exact result for one question."""
+
+ question_id: QuestionId
+ expected: bool
+ predicted: bool
+ correct: bool
+
+
+class BundleArtifact(StrictFrozenModel):
+ """One canonical file referenced by a release manifest."""
+
+ path: NonEmptyString
+ sha256: Sha256Digest
+ record_count: int = Field(ge=0)
+
+ @model_validator(mode="after")
+ def validate_path(self) -> Self:
+ path = PurePosixPath(self.path)
+ if (
+ path.is_absolute()
+ or "\\" in self.path
+ or path.as_posix() != self.path
+ or any(part in {"", ".", ".."} for part in path.parts)
+ ):
+ raise ValueError("artifact path must be a relative canonical path")
+ return self
+
+
+class PublicBundleManifest(StrictFrozenModel):
+ """Public release metadata containing no teacher-derived artifacts."""
+
+ schema_version: NonEmptyString
+ bundle_id: BundleId
+ episode_id: NonEmptyString
+ source_video_sha256: Sha256Digest
+ episode: BundleArtifact
+ questions: BundleArtifact
+
+ @model_validator(mode="after")
+ def validate_manifest(self) -> Self:
+ if self.schema_version != SCHEMA_VERSION:
+ raise ValueError("unsupported schema version")
+ expected_bundle_id = stable_id(
+ "bundle",
+ {
+ "episode_id": self.episode_id,
+ "schema_version": self.schema_version,
+ "source_video_sha256": self.source_video_sha256,
+ },
+ )
+ if self.bundle_id != expected_bundle_id:
+ raise ValueError("bundle ID does not match public release identity")
+ if not self.episode.path.startswith("public/") or not self.questions.path.startswith(
+ "public/"
+ ):
+ raise ValueError("public manifest artifacts must remain under public/")
+ if self.episode.path == self.questions.path:
+ raise ValueError("public manifest artifacts must have distinct paths")
+ return self
+
+
+class OracleBundleManifest(StrictFrozenModel):
+ """Private release metadata bound to an exact public release."""
+
+ schema_version: NonEmptyString
+ bundle_id: BundleId
+ episode_id: NonEmptyString
+ source_video_sha256: Sha256Digest
+ public_manifest_sha256: Sha256Digest
+ observations: BundleArtifact
+ relation_facts: BundleArtifact
+ relation_intervals: BundleArtifact
+ answers: BundleArtifact
+
+ @model_validator(mode="after")
+ def validate_manifest(self) -> Self:
+ if self.schema_version != SCHEMA_VERSION:
+ raise ValueError("unsupported schema version")
+ expected_bundle_id = stable_id(
+ "bundle",
+ {
+ "episode_id": self.episode_id,
+ "schema_version": self.schema_version,
+ "source_video_sha256": self.source_video_sha256,
+ },
+ )
+ if self.bundle_id != expected_bundle_id:
+ raise ValueError("bundle ID does not match oracle release identity")
+ artifacts = (
+ self.observations,
+ self.relation_facts,
+ self.relation_intervals,
+ self.answers,
+ )
+ if any(not artifact.path.startswith("oracle/") for artifact in artifacts):
+ raise ValueError("oracle manifest artifacts must remain under oracle/")
+ if len({artifact.path for artifact in artifacts}) != len(artifacts):
+ raise ValueError("oracle manifest artifacts must have distinct paths")
+ return self
diff --git a/dimos/benchmark/spatiotemporal/observation_io.py b/dimos/benchmark/spatiotemporal/observation_io.py
new file mode 100644
index 0000000000..2ddb3e274b
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/observation_io.py
@@ -0,0 +1,105 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Canonical JSONL persistence for private teacher observations."""
+
+from collections.abc import Iterable
+import json
+from pathlib import Path
+from typing import Any
+
+from pydantic import ValidationError
+
+from dimos.benchmark.spatiotemporal.models import ObjectObservation
+from dimos.benchmark.spatiotemporal.utilities import canonical_model_json
+
+
+def _reject_nonfinite_json(constant: str) -> None:
+ raise ValueError(f"invalid JSON numeric constant: {constant}")
+
+
+def _reject_duplicate_json_fields(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
+ value: dict[str, Any] = {}
+ for key, item in pairs:
+ if key in value:
+ raise ValueError(f"duplicate JSON field: {key}")
+ value[key] = item
+ return value
+
+
+def _identity(observation: ObjectObservation) -> tuple[str, int, str]:
+ return observation.episode_id, observation.frame_id, observation.object_id
+
+
+def _validated_observations(
+ observations: Iterable[ObjectObservation],
+) -> tuple[ObjectObservation, ...]:
+ by_identity: dict[tuple[str, int, str], ObjectObservation] = {}
+ for observation in observations:
+ try:
+ observation = ObjectObservation.model_validate(observation.model_dump())
+ except ValidationError as error:
+ raise ValueError(f"invalid observation: {error}") from error
+ if observation.timestamp_s < 0.0:
+ raise ValueError("observation timestamp must be non-negative")
+
+ identity = _identity(observation)
+ previous = by_identity.get(identity)
+ if previous is not None:
+ qualifier = "duplicate" if previous == observation else "conflicting"
+ raise ValueError(f"{qualifier} observation identity: {identity}")
+ by_identity[identity] = observation
+
+ return tuple(sorted(by_identity.values(), key=_identity))
+
+
+def write_observations(path: Path, observations: Iterable[ObjectObservation]) -> None:
+ """Write observations as deterministic, identity-ordered canonical JSONL."""
+ ordered = _validated_observations(observations)
+ document = "".join(f"{canonical_model_json(observation)}\n" for observation in ordered)
+ path.write_bytes(document.encode("utf-8"))
+
+
+def read_observations(path: Path) -> tuple[ObjectObservation, ...]:
+ """Read and strictly validate canonical teacher observations from JSONL."""
+ try:
+ document = path.read_bytes().decode("utf-8")
+ except UnicodeDecodeError as error:
+ raise ValueError("observation JSONL must be valid UTF-8") from error
+ if document and not document.endswith("\n"):
+ raise ValueError("observation JSONL must end with a canonical newline")
+
+ lines = document[:-1].split("\n") if document else []
+ observations: list[ObjectObservation] = []
+ for line_number, line in enumerate(lines, start=1):
+ try:
+ value = json.loads(
+ line,
+ object_pairs_hook=_reject_duplicate_json_fields,
+ parse_constant=_reject_nonfinite_json,
+ )
+ except (json.JSONDecodeError, ValueError) as error:
+ raise ValueError(f"invalid JSON at line {line_number}: {error}") from error
+ try:
+ observation = ObjectObservation.model_validate(value)
+ except ValidationError as error:
+ raise ValueError(f"invalid observation at line {line_number}: {error}") from error
+ if line != canonical_model_json(observation):
+ raise ValueError(f"non-canonical observation JSON at line {line_number}")
+ observations.append(observation)
+
+ ordered = _validated_observations(observations)
+ if tuple(observations) != ordered:
+ raise ValueError("observation JSONL records are not in canonical identity order")
+ return ordered
diff --git a/dimos/benchmark/spatiotemporal/ports.py b/dimos/benchmark/spatiotemporal/ports.py
new file mode 100644
index 0000000000..98b45c1a05
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/ports.py
@@ -0,0 +1,155 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Frozen callable boundaries for perception and candidate evaluation."""
+
+from collections.abc import Sequence
+from enum import StrEnum
+from hashlib import sha256
+from pathlib import Path
+from typing import Protocol, Self
+
+from pydantic import Field, model_validator
+
+from dimos.benchmark.spatiotemporal.models import (
+ BoundingBox2D,
+ NonEmptyString,
+ ObjectObservation,
+ OracleBundleManifest,
+ PublicBundleManifest,
+ Question,
+ Sha256Digest,
+ StrictFrozenModel,
+)
+from dimos.benchmark.spatiotemporal.utilities import canonical_json_bytes, canonical_model_json
+from dimos.msgs.sensor_msgs.Image import Image
+
+
+class DetectedObject(StrictFrozenModel):
+ """One detector result containing only perception-seam data."""
+
+ object_id: NonEmptyString
+ label: NonEmptyString
+ box: BoundingBox2D
+ confidence: float = Field(ge=0.0, le=1.0, allow_inf_nan=False)
+
+
+class CandidateReadiness(StrictFrozenModel):
+ """Candidate ingestion state with no teacher or oracle artifacts."""
+
+ ready: bool
+ ingested_frame_count: int = Field(ge=0)
+ detail: NonEmptyString | None = None
+
+ @model_validator(mode="after")
+ def validate_readiness(self) -> "CandidateReadiness":
+ if self.ready and self.ingested_frame_count == 0:
+ raise ValueError("a ready candidate must ingest at least one frame")
+ return self
+
+
+class ReplayInsufficiencyCode(StrEnum):
+ """Stable reasons that saved observations cannot produce an evaluation bundle."""
+
+ EMPTY_OBSERVATIONS = "empty_observations"
+ MIXED_EPISODES = "mixed_episodes"
+ NO_RELATIONS = "no_relations"
+ NO_QUESTIONS = "no_questions"
+
+
+class ReplayInsufficiencyError(ValueError):
+ """Actionable replay failure with a machine-readable stable reason."""
+
+ def __init__(self, code: ReplayInsufficiencyCode, message: str) -> None:
+ self.code = code
+ super().__init__(message)
+
+
+def replay_bundle_logical_sha256(
+ public_manifest: PublicBundleManifest,
+ oracle_manifest: OracleBundleManifest,
+) -> str:
+ """Hash canonical manifest values without extraction-root paths."""
+ return sha256(
+ canonical_json_bytes(
+ {
+ "oracle_manifest": oracle_manifest.model_dump(mode="json"),
+ "public_manifest": public_manifest.model_dump(mode="json"),
+ }
+ )
+ ).hexdigest()
+
+
+class ReplayBundleResult(StrictFrozenModel):
+ """Root-independent manifests and logical hash returned by replay generation."""
+
+ public_manifest: PublicBundleManifest
+ oracle_manifest: OracleBundleManifest
+ logical_sha256: Sha256Digest
+
+ @model_validator(mode="after")
+ def validate_result(self) -> Self:
+ public = self.public_manifest
+ oracle = self.oracle_manifest
+ identities = {
+ (
+ public.schema_version,
+ public.bundle_id,
+ public.episode_id,
+ public.source_video_sha256,
+ ),
+ (
+ oracle.schema_version,
+ oracle.bundle_id,
+ oracle.episode_id,
+ oracle.source_video_sha256,
+ ),
+ }
+ if len(identities) != 1:
+ raise ValueError("public and oracle manifests must share one release identity")
+ expected_public_sha256 = sha256(f"{canonical_model_json(public)}\n".encode()).hexdigest()
+ if oracle.public_manifest_sha256 != expected_public_sha256:
+ raise ValueError("oracle manifest must bind the canonical public manifest")
+ if self.logical_sha256 != replay_bundle_logical_sha256(public, oracle):
+ raise ValueError("logical SHA-256 does not match canonical manifests")
+ return self
+
+
+class ObservationBundleGenerator(Protocol):
+ """Teacher-side seam that turns canonical observations into one bundle."""
+
+ def generate(
+ self,
+ observations: Sequence[ObjectObservation],
+ output_root: Path,
+ source_video_sha256: str,
+ ) -> ReplayBundleResult: ...
+
+
+class ObservationDetector(Protocol):
+ """Narrow image-to-detections seam used by video sampling."""
+
+ def detect(self, image: Image) -> Sequence[DetectedObject]: ...
+
+ def close(self) -> None: ...
+
+
+class CandidateAnswerer(Protocol):
+ """Public-only video ingestion and question answering seam."""
+
+ def ingest_video(self, video_path: Path) -> CandidateReadiness: ...
+
+ def answer(self, question: Question) -> str | bool | None: ...
+
+ def close(self) -> None: ...
diff --git a/dimos/benchmark/spatiotemporal/relations.py b/dimos/benchmark/spatiotemporal/relations.py
new file mode 100644
index 0000000000..65837ed003
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/relations.py
@@ -0,0 +1,112 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Image-plane relation derivation."""
+
+from dataclasses import dataclass
+from math import isfinite
+from typing import Literal
+
+from dimos.benchmark.spatiotemporal.models import ObjectObservation, SpatialPredicate
+
+
+@dataclass(frozen=True)
+class SpatialRelationCandidate:
+ """A temporary accepted relation backed by two observations."""
+
+ subject: ObjectObservation
+ predicate: SpatialPredicate
+ object: ObjectObservation
+ margin: float
+
+
+def _derive_ordered(
+ subject: ObjectObservation,
+ object_: ObjectObservation,
+ margin: float,
+ predicate: SpatialPredicate,
+ axis: Literal["x", "y"],
+ *,
+ reverse: bool = False,
+) -> SpatialRelationCandidate | None:
+ if not isfinite(margin) or not 0.0 <= margin <= 1.0:
+ raise ValueError("margin must be finite and within [0, 1]")
+ subject_sample = (subject.episode_id, subject.frame_id, subject.timestamp_s)
+ object_sample = (object_.episode_id, object_.frame_id, object_.timestamp_s)
+ if subject_sample != object_sample:
+ raise ValueError("relations require observations from the same sample")
+ if subject.object_id == object_.object_id:
+ return None
+ before, after = (object_, subject) if reverse else (subject, object_)
+ before_max = getattr(before.box, f"{axis}_max")
+ after_min = getattr(after.box, f"{axis}_min")
+ separation = after_min - before_max
+ if separation > margin:
+ return SpatialRelationCandidate(
+ subject=subject,
+ predicate=predicate,
+ object=object_,
+ margin=margin,
+ )
+ return None
+
+
+def derive_left_of(
+ subject: ObjectObservation,
+ object_: ObjectObservation,
+ margin: float,
+) -> SpatialRelationCandidate | None:
+ """Derive an accepted left-of candidate when evidence is sufficient."""
+ return _derive_ordered(subject, object_, margin, SpatialPredicate.LEFT_OF, "x")
+
+
+def derive_right_of(
+ subject: ObjectObservation,
+ object_: ObjectObservation,
+ margin: float,
+) -> SpatialRelationCandidate | None:
+ """Derive right-of by reusing left-of with swapped arguments."""
+ return _derive_ordered(
+ subject,
+ object_,
+ margin,
+ SpatialPredicate.RIGHT_OF,
+ "x",
+ reverse=True,
+ )
+
+
+def derive_above(
+ subject: ObjectObservation,
+ object_: ObjectObservation,
+ margin: float,
+) -> SpatialRelationCandidate | None:
+ """Derive an accepted above candidate when evidence is sufficient."""
+ return _derive_ordered(subject, object_, margin, SpatialPredicate.ABOVE, "y")
+
+
+def derive_below(
+ subject: ObjectObservation,
+ object_: ObjectObservation,
+ margin: float,
+) -> SpatialRelationCandidate | None:
+ """Derive below by reusing above with swapped arguments."""
+ return _derive_ordered(
+ subject,
+ object_,
+ margin,
+ SpatialPredicate.BELOW,
+ "y",
+ reverse=True,
+ )
diff --git a/dimos/benchmark/spatiotemporal/replay.py b/dimos/benchmark/spatiotemporal/replay.py
new file mode 100644
index 0000000000..d023eb88b5
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/replay.py
@@ -0,0 +1,195 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Deterministic replay of saved observations into evaluation bundles."""
+
+from collections import defaultdict
+from collections.abc import Sequence
+from itertools import permutations
+from pathlib import Path
+
+from dimos.benchmark.spatiotemporal.bundles import load_bundle, write_bundle
+from dimos.benchmark.spatiotemporal.generation import (
+ generate_spatial_question_cases,
+ generate_temporal_question_cases,
+)
+from dimos.benchmark.spatiotemporal.intervals import build_relation_intervals
+from dimos.benchmark.spatiotemporal.models import (
+ ObjectObservation,
+ RelationFact,
+)
+from dimos.benchmark.spatiotemporal.observation_io import read_observations
+from dimos.benchmark.spatiotemporal.ports import (
+ ObservationBundleGenerator,
+ ReplayBundleResult,
+ ReplayInsufficiencyCode,
+ ReplayInsufficiencyError,
+ replay_bundle_logical_sha256,
+)
+from dimos.benchmark.spatiotemporal.relations import (
+ SpatialRelationCandidate,
+ derive_above,
+ derive_left_of,
+)
+from dimos.benchmark.spatiotemporal.utilities import SCHEMA_VERSION, stable_id
+
+_RELATION_DERIVERS = (derive_left_of, derive_above)
+
+
+def _relation_facts(
+ observations: Sequence[ObjectObservation], margin: float
+) -> tuple[RelationFact, ...]:
+ by_sample: dict[tuple[int, float], list[ObjectObservation]] = defaultdict(list)
+ for observation in observations:
+ by_sample[(observation.frame_id, observation.timestamp_s)].append(observation)
+
+ facts: dict[tuple[int, str], RelationFact] = {}
+ for sample in by_sample.values():
+ for subject, object_ in permutations(sample, 2):
+ for derive in _RELATION_DERIVERS:
+ candidate = derive(subject, object_, margin)
+ if candidate is None:
+ continue
+ fact = _relation_fact(candidate)
+ facts[(fact.frame_id, fact.relation_id)] = fact
+ return tuple(facts[key] for key in sorted(facts))
+
+
+def _relation_fact(candidate: SpatialRelationCandidate) -> RelationFact:
+ subject = candidate.subject
+ object_ = candidate.object
+ relation_id = stable_id(
+ "relation",
+ {
+ "object_ids": (subject.object_id, object_.object_id),
+ "predicate": candidate.predicate.value,
+ "schema_version": SCHEMA_VERSION,
+ },
+ )
+ return RelationFact(
+ relation_id=relation_id,
+ episode_id=subject.episode_id,
+ frame_id=subject.frame_id,
+ timestamp_s=subject.timestamp_s,
+ subject_id=subject.object_id,
+ predicate=candidate.predicate,
+ object_id=object_.object_id,
+ evidence_frame_ids=(subject.frame_id,),
+ )
+
+
+class DeterministicObservationBundleGenerator:
+ """Generate deterministic bundles from canonical object observations."""
+
+ def __init__(
+ self,
+ *,
+ relation_margin: float = 0.0,
+ sample_schedule: Sequence[tuple[int, float]] | None = None,
+ ) -> None:
+ self._relation_margin = relation_margin
+ self._sample_schedule = tuple(sample_schedule) if sample_schedule is not None else None
+
+ def generate(
+ self,
+ observations: Sequence[ObjectObservation],
+ output_root: Path,
+ source_video_sha256: str,
+ ) -> ReplayBundleResult:
+ """Generate one public/oracle evaluation bundle."""
+ if not observations:
+ raise ReplayInsufficiencyError(
+ ReplayInsufficiencyCode.EMPTY_OBSERVATIONS,
+ "saved observations are empty; capture at least one sampled frame",
+ )
+ episodes = {observation.episode_id for observation in observations}
+ if len(episodes) != 1:
+ raise ReplayInsufficiencyError(
+ ReplayInsufficiencyCode.MIXED_EPISODES,
+ "saved observations contain multiple episodes; replay one episode at a time",
+ )
+
+ facts = _relation_facts(observations, self._relation_margin)
+ if not facts:
+ raise ReplayInsufficiencyError(
+ ReplayInsufficiencyCode.NO_RELATIONS,
+ "saved observations contain no spatially separated object pairs",
+ )
+ intervals = build_relation_intervals(facts, sample_schedule=self._sample_schedule)
+ sample_frame_ids = (
+ tuple(frame_id for frame_id, _ in self._sample_schedule)
+ if self._sample_schedule is not None
+ else tuple(sorted({observation.frame_id for observation in observations}))
+ )
+ spatial_cases = generate_spatial_question_cases(facts, sample_frame_ids=sample_frame_ids)
+ temporal_cases = generate_temporal_question_cases(intervals)
+ questions = tuple(
+ sorted(
+ (
+ *(question for question, _ in spatial_cases),
+ *(question for question, _ in temporal_cases),
+ ),
+ key=lambda question: question.question_id,
+ )
+ )
+ if not questions:
+ raise ReplayInsufficiencyError(
+ ReplayInsufficiencyCode.NO_QUESTIONS,
+ "accepted relations produced no evaluation questions; inspect generation inputs",
+ )
+ answers = tuple(
+ sorted(
+ (
+ *(answer for _, answer in spatial_cases),
+ *(answer for _, answer in temporal_cases),
+ ),
+ key=lambda answer: answer.question_id,
+ )
+ )
+ episode_id = next(iter(episodes))
+ write_bundle(
+ output_root,
+ episode_id=episode_id,
+ source_video_sha256=source_video_sha256,
+ questions=questions,
+ observations=observations,
+ relation_facts=facts,
+ relation_intervals=intervals,
+ answers=answers,
+ )
+ bundle = load_bundle(output_root)
+ return ReplayBundleResult(
+ public_manifest=bundle.public_manifest,
+ oracle_manifest=bundle.oracle_manifest,
+ logical_sha256=replay_bundle_logical_sha256(
+ bundle.public_manifest, bundle.oracle_manifest
+ ),
+ )
+
+
+def replay_observations(
+ observations_path: Path,
+ output_root: Path,
+ source_video_sha256: str,
+ *,
+ sample_schedule: Sequence[tuple[int, float]] | None = None,
+ generator: ObservationBundleGenerator | None = None,
+) -> ReplayBundleResult:
+ """Read canonical saved observations and generate an evaluation bundle."""
+ if generator is not None and sample_schedule is not None:
+ raise ValueError("sample_schedule cannot be combined with a custom generator")
+ observations = read_observations(observations_path)
+ if generator is None:
+ generator = DeterministicObservationBundleGenerator(sample_schedule=sample_schedule)
+ return generator.generate(observations, output_root, source_video_sha256)
diff --git a/dimos/benchmark/spatiotemporal/reproduce_reference.sh b/dimos/benchmark/spatiotemporal/reproduce_reference.sh
new file mode 100755
index 0000000000..f6eab4bd91
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/reproduce_reference.sh
@@ -0,0 +1,160 @@
+#!/usr/bin/env bash
+# Reproduce the reference spatiotemporal video-relation evaluation from a checkout.
+set -euo pipefail
+
+usage() {
+ cat <<'EOF'
+Usage: reproduce_reference.sh [--skip-setup] [--skip-tests]
+
+ --skip-setup Do not run LFS materialization, uv sync, model extraction, or CLIP setup.
+ --skip-tests Do not run pytest, Ruff, or mypy before the real demo.
+EOF
+}
+
+skip_setup=false
+skip_tests=false
+while (($#)); do
+ case "$1" in
+ --skip-setup) skip_setup=true ;;
+ --skip-tests) skip_tests=true ;;
+ -h|--help) usage; exit 0 ;;
+ *) echo "Unknown argument: $1" >&2; usage >&2; exit 2 ;;
+ esac
+ shift
+done
+
+for command in git uv; do
+ command -v "$command" >/dev/null || {
+ echo "Missing required command: $command" >&2
+ exit 1
+ }
+done
+
+repo_root="$(git rev-parse --show-toplevel)"
+cd "$repo_root"
+
+if ! $skip_setup; then
+ command -v git-lfs >/dev/null || {
+ echo "Missing required command: git-lfs" >&2
+ exit 1
+ }
+
+ git lfs install --local
+ git lfs pull --include='assets/simple_demo.mp4,data/.lfs/models_yoloe.tar.gz'
+
+ python3 - <<'PY'
+from pathlib import Path
+for name in ("assets/simple_demo.mp4", "data/.lfs/models_yoloe.tar.gz"):
+ size = Path(name).stat().st_size
+ print(f"{name}: {size:,} bytes")
+ assert size > 1_000_000, f"{name} is still an LFS pointer"
+PY
+
+ uv sync --group lint
+ uv run python -c "from dimos.utils.data import get_data; print(get_data('models_yoloe'))"
+
+ if ! uv run python -c 'import clip' >/dev/null 2>&1; then
+ uv pip install 'git+https://github.com/ultralytics/CLIP.git'
+ fi
+fi
+
+if ! $skip_tests; then
+ uv run pytest dimos/benchmark/spatiotemporal -q
+ uv run ruff format --check dimos/benchmark/spatiotemporal
+ uv run ruff check dimos/benchmark/spatiotemporal
+ uv run --group lint mypy dimos/benchmark/spatiotemporal
+fi
+
+uv run python -m dimos.benchmark.spatiotemporal.demo \
+ --source-video assets/simple_demo.mp4 \
+ --output-root .artifacts/spatiotemporal-video-qa \
+ --duration-s 25 \
+ --frame-stride 150
+
+uv run python - <<'PY'
+import hashlib
+import json
+from pathlib import Path
+
+root = Path(".artifacts/spatiotemporal-video-qa")
+summary_path = root / "summary.json"
+summary = json.loads(summary_path.read_text(encoding="utf-8"))
+teacher = summary["teacher"]
+candidate = summary["candidate"]
+review = summary["review"]
+
+assert summary["video"]["duration_s"] == 25
+assert summary["video"]["frames"] == 750
+assert teacher["detector_repeat_equal"] is True
+assert teacher["relation_facts"] > 0
+assert teacher["relation_intervals"] > 0
+assert teacher["questions"] == teacher["answers"]
+assert candidate["candidate_used_oracle"] is False
+assert candidate["baseline_kind"] == "public_only_plumbing_smoke_test"
+assert candidate["visually_grounded"] is False
+assert candidate["questions_answered"] == teacher["questions"]
+assert candidate["status_counts"]["missing"] == 0
+assert candidate["status_counts"]["invalid"] == 0
+assert review["question_count"] == teacher["questions"]
+assert (root / review["index_path"]).is_file()
+
+questions = [
+ json.loads(line)
+ for line in (root / "bundle-a/public/questions.jsonl")
+ .read_text(encoding="utf-8")
+ .splitlines()
+]
+answers = [
+ json.loads(line)
+ for line in (root / "bundle-a/oracle/answers.jsonl")
+ .read_text(encoding="utf-8")
+ .splitlines()
+]
+assert len(questions) == len(answers) == teacher["questions"]
+for question in questions:
+ assert "expected" not in question
+ assert "evidence_frame_ids" not in question
+ assert "evidence_interval_ids" not in question
+
+answers_by_id = {answer["question_id"]: answer for answer in answers}
+for family in ("spatial", "temporal"):
+ expected = [
+ answers_by_id[question["question_id"]]["expected"]
+ for question in questions
+ if question["question_kind"] == family
+ ]
+ assert expected.count(True) == expected.count(False)
+
+print("GENERATED_EXAMPLES")
+for family, expected in (("spatial", True), ("spatial", False), ("temporal", True)):
+ question = next(
+ question
+ for question in questions
+ if question["question_kind"] == family
+ and answers_by_id[question["question_id"]]["expected"] is expected
+ )
+ answer = answers_by_id[question["question_id"]]
+ print(f" [{family} expected={str(expected).lower()}] {question['text']}")
+ print(
+ " evidence_frames="
+ f"{answer['evidence_frame_ids']} evidence_intervals="
+ f"{answer['evidence_interval_ids']}"
+ )
+
+digest = hashlib.sha256(summary_path.read_bytes()).hexdigest()
+print(f"SUMMARY_GATES=PASS sha256={digest}")
+PY
+
+cat <<'EOF'
+
+Reproduction complete.
+
+Summary:
+ .artifacts/spatiotemporal-video-qa/summary.json
+
+Evidence viewer:
+ .artifacts/spatiotemporal-video-qa/evidence-viewer/index.html
+
+On macOS, open it with:
+ open .artifacts/spatiotemporal-video-qa/evidence-viewer/index.html
+EOF
diff --git a/dimos/benchmark/spatiotemporal/runner.py b/dimos/benchmark/spatiotemporal/runner.py
new file mode 100644
index 0000000000..8128a4140d
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/runner.py
@@ -0,0 +1,123 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Candidate evaluation runner helpers."""
+
+from collections.abc import Collection, Mapping, Sequence
+import re
+
+from dimos.benchmark.spatiotemporal.models import (
+ OracleAnswer,
+ Prediction,
+ PredictionStatus,
+ Question,
+ QuestionId,
+ QuestionKind,
+)
+from dimos.benchmark.spatiotemporal.scoring import (
+ EvaluationReport,
+ QuestionDiagnostic,
+ QuestionPredicate,
+ summarize_statuses,
+)
+
+
+def parse_candidate_prediction(
+ question_id: QuestionId,
+ raw_answer: str | bool | None,
+ expected: bool,
+) -> tuple[PredictionStatus, Prediction | None]:
+ """Parse one candidate answer and classify it against private truth."""
+ if raw_answer is None:
+ return PredictionStatus.MISSING, None
+ if isinstance(raw_answer, bool):
+ answer = raw_answer
+ elif raw_answer == "yes":
+ answer = True
+ elif raw_answer == "no":
+ answer = False
+ else:
+ return PredictionStatus.INVALID, None
+
+ prediction = Prediction(question_id=question_id, answer=answer)
+ status = PredictionStatus.CORRECT if answer is expected else PredictionStatus.INCORRECT
+ return status, prediction
+
+
+def build_evaluation_report(
+ questions: Sequence[Question],
+ oracles: Mapping[QuestionId, OracleAnswer],
+ raw_answers: Mapping[QuestionId, str | bool | None],
+ *,
+ source_video_sha256: str,
+ diagnostic_statuses: Collection[PredictionStatus] = tuple(PredictionStatus),
+) -> EvaluationReport:
+ """Join public questions with private truth and aggregate candidate outcomes."""
+ if re.fullmatch(r"[0-9a-f]{64}", source_video_sha256) is None:
+ raise ValueError("source video SHA-256 must be 64 lowercase hexadecimal characters")
+
+ question_ids = tuple(question.question_id for question in questions)
+ if len(set(question_ids)) != len(question_ids):
+ raise ValueError("question IDs must be unique")
+ if set(oracles) != set(question_ids):
+ raise ValueError("oracle answers must exactly match public questions")
+ if set(raw_answers).difference(question_ids):
+ raise ValueError("candidate answers contain unknown question IDs")
+
+ selected_statuses = frozenset(diagnostic_statuses)
+ statuses: list[PredictionStatus] = []
+ diagnostics: list[QuestionDiagnostic] = []
+ statuses_by_family: dict[QuestionKind, list[PredictionStatus]] = {}
+ statuses_by_predicate: dict[QuestionPredicate, list[PredictionStatus]] = {}
+
+ for question in questions:
+ oracle = oracles[question.question_id]
+ if oracle.question_id != question.question_id:
+ raise ValueError("oracle map keys and question IDs must match")
+ status, prediction = parse_candidate_prediction(
+ question.question_id,
+ raw_answers.get(question.question_id),
+ oracle.expected,
+ )
+ statuses.append(status)
+ statuses_by_family.setdefault(question.question_kind, []).append(status)
+ statuses_by_predicate.setdefault(question.predicate, []).append(status)
+ if status in selected_statuses:
+ diagnostics.append(
+ QuestionDiagnostic(
+ question_id=question.question_id,
+ status=status,
+ question_kind=question.question_kind,
+ predicate=question.predicate,
+ expected=oracle.expected,
+ predicted=prediction.answer if prediction is not None else None,
+ evidence_frame_ids=oracle.evidence_frame_ids,
+ evidence_interval_ids=oracle.evidence_interval_ids,
+ )
+ )
+
+ return EvaluationReport(
+ source_video_sha256=source_video_sha256,
+ overall=summarize_statuses(tuple(statuses)),
+ status_counts={status: statuses.count(status) for status in PredictionStatus},
+ by_family={
+ family: summarize_statuses(tuple(family_statuses))
+ for family, family_statuses in statuses_by_family.items()
+ },
+ by_predicate={
+ predicate: summarize_statuses(tuple(predicate_statuses))
+ for predicate, predicate_statuses in statuses_by_predicate.items()
+ },
+ diagnostics=tuple(diagnostics),
+ )
diff --git a/dimos/benchmark/spatiotemporal/scoring.py b/dimos/benchmark/spatiotemporal/scoring.py
new file mode 100644
index 0000000000..886dce6072
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/scoring.py
@@ -0,0 +1,91 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Exact scoring for typed spatiotemporal predictions."""
+
+from collections.abc import Mapping
+from dataclasses import dataclass
+
+from dimos.benchmark.spatiotemporal.models import (
+ IntervalId,
+ OracleAnswer,
+ Prediction,
+ PredictionStatus,
+ Question,
+ QuestionId,
+ QuestionKind,
+ QuestionResult,
+ SpatialPredicate,
+ TemporalPredicate,
+)
+
+QuestionPredicate = SpatialPredicate | TemporalPredicate
+
+
+@dataclass(frozen=True, slots=True)
+class ScoreSummary:
+ """Aggregate exact-match counts for one report slice."""
+
+ total: int
+ correct: int
+ accuracy: float
+
+
+@dataclass(frozen=True, slots=True)
+class QuestionDiagnostic:
+ """Evidence-linked diagnostic information for one public question."""
+
+ question_id: QuestionId
+ status: PredictionStatus
+ question_kind: QuestionKind
+ predicate: QuestionPredicate
+ expected: bool
+ predicted: bool | None
+ evidence_frame_ids: tuple[int, ...]
+ evidence_interval_ids: tuple[IntervalId, ...]
+
+
+@dataclass(frozen=True, slots=True)
+class EvaluationReport:
+ """Aggregate candidate scores linked to source identity and teacher evidence."""
+
+ source_video_sha256: str
+ overall: ScoreSummary
+ status_counts: Mapping[PredictionStatus, int]
+ by_family: Mapping[QuestionKind, ScoreSummary]
+ by_predicate: Mapping[QuestionPredicate, ScoreSummary]
+ diagnostics: tuple[QuestionDiagnostic, ...]
+
+
+def summarize_statuses(statuses: tuple[PredictionStatus, ...]) -> ScoreSummary:
+ """Summarize exact candidate statuses, counting all outcomes in the denominator."""
+ total = len(statuses)
+ correct = statuses.count(PredictionStatus.CORRECT)
+ return ScoreSummary(total=total, correct=correct, accuracy=correct / total if total else 0.0)
+
+
+def score_prediction(
+ question: Question,
+ oracle: OracleAnswer,
+ prediction: Prediction,
+) -> QuestionResult:
+ """Score one typed prediction against separately supplied private truth."""
+ if len({question.question_id, oracle.question_id, prediction.question_id}) != 1:
+ raise ValueError("question IDs must match for scoring")
+ return QuestionResult(
+ question_id=question.question_id,
+ expected=oracle.expected,
+ predicted=prediction.answer,
+ correct=prediction.answer is oracle.expected,
+ )
diff --git a/dimos/benchmark/spatiotemporal/temporal_memory_answerer.py b/dimos/benchmark/spatiotemporal/temporal_memory_answerer.py
new file mode 100644
index 0000000000..1669c0adaf
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/temporal_memory_answerer.py
@@ -0,0 +1,66 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Public-only CandidateAnswerer adapter for TemporalMemory evaluation."""
+
+from collections.abc import Callable
+from pathlib import Path
+
+from dimos.benchmark.spatiotemporal.models import Question
+from dimos.benchmark.spatiotemporal.ports import CandidateReadiness
+
+
+class TemporalMemoryAnswerer:
+ """Adapt TemporalMemory ingestion, query, and cleanup callables for evaluation."""
+
+ def __init__(
+ self,
+ *,
+ ingest: Callable[[Path], int],
+ query: Callable[[str], str | bool | None],
+ cleanup: Callable[[], None],
+ ) -> None:
+ self._ingest = ingest
+ self._query = query
+ self._cleanup = cleanup
+ self._readiness: CandidateReadiness | None = None
+ self._closed = False
+
+ def ingest_video(self, video_path: Path) -> CandidateReadiness:
+ """Ingest only the public source video and report candidate readiness."""
+ self._require_open()
+ readiness = CandidateReadiness(
+ ready=(frame_count := self._ingest(video_path)) > 0,
+ ingested_frame_count=frame_count,
+ )
+ self._readiness = readiness
+ return readiness
+
+ def answer(self, question: Question) -> str | bool | None:
+ """Answer a public question only after successful video ingestion."""
+ self._require_open()
+ if self._readiness is None or not self._readiness.ready:
+ raise RuntimeError("video must be ingested before answering questions")
+ return self._query(question.text)
+
+ def close(self) -> None:
+ """Release TemporalMemory resources exactly once."""
+ if self._closed:
+ return
+ self._closed = True
+ self._cleanup()
+
+ def _require_open(self) -> None:
+ if self._closed:
+ raise RuntimeError("TemporalMemory answerer is closed")
diff --git a/dimos/benchmark/spatiotemporal/test_bundles.py b/dimos/benchmark/spatiotemporal/test_bundles.py
new file mode 100644
index 0000000000..f3a81a2dde
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/test_bundles.py
@@ -0,0 +1,487 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for deterministic public/private evaluation bundles."""
+
+from hashlib import sha256
+from pathlib import Path
+
+import pytest
+
+from dimos.benchmark.spatiotemporal.bundles import load_bundle, write_bundle
+from dimos.benchmark.spatiotemporal.models import (
+ BoundingBox2D,
+ ObjectObservation,
+ OracleAnswer,
+ OracleBundleManifest,
+ PublicBundleManifest,
+ Question,
+ QuestionKind,
+ RelationFact,
+ RelationInterval,
+ SpatialPredicate,
+ TemporalPredicate,
+)
+from dimos.benchmark.spatiotemporal.utilities import (
+ SCHEMA_VERSION,
+ JsonValue,
+ canonical_model_json,
+ stable_id,
+ stable_question_id,
+)
+
+
+def _spatial_records() -> tuple[
+ Question,
+ ObjectObservation,
+ RelationFact,
+ OracleAnswer,
+]:
+ episode_id = "episode_1"
+ subject_id = "obj_red"
+ object_id = "obj_blue"
+ predicate = SpatialPredicate.LEFT_OF
+ relation_id = stable_id(
+ "relation",
+ {
+ "object_ids": (subject_id, object_id),
+ "predicate": predicate.value,
+ "schema_version": SCHEMA_VERSION,
+ },
+ )
+ question_id = stable_question_id(
+ episode_id=episode_id,
+ object_ids=(subject_id, object_id),
+ predicate=predicate.value,
+ question_kind=QuestionKind.SPATIAL.value,
+ )
+ question = Question(
+ question_id=question_id,
+ episode_id=episode_id,
+ text="Is obj_red left of obj_blue?",
+ question_kind=QuestionKind.SPATIAL,
+ predicate=predicate,
+ object_ids=(subject_id, object_id),
+ )
+ observation = ObjectObservation(
+ episode_id=episode_id,
+ frame_id=3,
+ timestamp_s=0.3,
+ object_id=subject_id,
+ label="red object",
+ box=BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.3, y_max=0.4),
+ confidence=0.9,
+ )
+ fact = RelationFact(
+ relation_id=relation_id,
+ episode_id=episode_id,
+ frame_id=3,
+ timestamp_s=0.3,
+ subject_id=subject_id,
+ predicate=predicate,
+ object_id=object_id,
+ evidence_frame_ids=(3,),
+ )
+ answer = OracleAnswer(
+ question_id=question_id,
+ expected=True,
+ evidence_frame_ids=(3,),
+ )
+ return question, observation, fact, answer
+
+
+def _interval_for_fact(fact: RelationFact) -> RelationInterval:
+ contract: dict[str, JsonValue] = {
+ "end_frame_id": fact.frame_id,
+ "end_timestamp_s": fact.timestamp_s,
+ "episode_id": fact.episode_id,
+ "object_id": fact.object_id,
+ "predicate": fact.predicate.value,
+ "relation_id": fact.relation_id,
+ "schema_version": SCHEMA_VERSION,
+ "start_frame_id": fact.frame_id,
+ "start_timestamp_s": fact.timestamp_s,
+ "subject_id": fact.subject_id,
+ }
+ return RelationInterval(
+ interval_id=stable_id("interval", contract),
+ relation_id=fact.relation_id,
+ episode_id=fact.episode_id,
+ subject_id=fact.subject_id,
+ predicate=fact.predicate,
+ object_id=fact.object_id,
+ start_frame_id=fact.frame_id,
+ end_frame_id=fact.frame_id,
+ start_timestamp_s=fact.timestamp_s,
+ end_timestamp_s=fact.timestamp_s,
+ evidence_frame_ids=(fact.frame_id,),
+ )
+
+
+def test_writes_and_loads_public_questions_without_private_teacher_data(tmp_path: Path) -> None:
+ question, observation, fact, answer = _spatial_records()
+
+ write_bundle(
+ tmp_path,
+ episode_id="episode_1",
+ source_video_sha256="1" * 64,
+ questions=(question,),
+ observations=(observation,),
+ relation_facts=(fact,),
+ relation_intervals=(),
+ answers=(answer,),
+ )
+ loaded = load_bundle(tmp_path)
+
+ assert loaded.questions == (question,)
+ assert loaded.observations == (observation,)
+ assert loaded.relation_facts == (fact,)
+ assert loaded.relation_intervals == ()
+ assert loaded.answers == (answer,)
+ public_text = "\n".join(
+ path.read_text(encoding="utf-8") for path in sorted((tmp_path / "public").iterdir())
+ )
+ assert "expected" not in public_text
+ assert "evidence_frame_ids" not in public_text
+ assert "confidence" not in public_text
+ assert (tmp_path / "oracle" / "answers.jsonl").is_file()
+
+
+def test_bundle_output_is_byte_identical_across_roots(tmp_path: Path) -> None:
+ question, observation, fact, answer = _spatial_records()
+ roots = (tmp_path / "first", tmp_path / "second")
+ for root in roots:
+ write_bundle(
+ root,
+ episode_id="episode_1",
+ source_video_sha256="1" * 64,
+ questions=(question,),
+ observations=(observation,),
+ relation_facts=(fact,),
+ relation_intervals=(),
+ answers=(answer,),
+ )
+
+ outputs = [
+ {path.relative_to(root): path.read_bytes() for path in root.rglob("*") if path.is_file()}
+ for root in roots
+ ]
+ assert outputs[0] == outputs[1]
+
+
+def test_rejects_duplicate_public_question_references(tmp_path: Path) -> None:
+ question, observation, fact, answer = _spatial_records()
+
+ with pytest.raises(ValueError, match="duplicate question ID"):
+ write_bundle(
+ tmp_path,
+ episode_id="episode_1",
+ source_video_sha256="1" * 64,
+ questions=(question, question),
+ observations=(observation,),
+ relation_facts=(fact,),
+ relation_intervals=(),
+ answers=(answer,),
+ )
+
+
+def test_rejects_oracle_answers_for_foreign_questions(tmp_path: Path) -> None:
+ question, observation, fact, _ = _spatial_records()
+ foreign_answer = OracleAnswer(
+ question_id=f"question_{'0' * 64}",
+ expected=True,
+ evidence_frame_ids=(3,),
+ )
+
+ with pytest.raises(ValueError, match="foreign question ID"):
+ write_bundle(
+ tmp_path,
+ episode_id="episode_1",
+ source_video_sha256="1" * 64,
+ questions=(question,),
+ observations=(observation,),
+ relation_facts=(fact,),
+ relation_intervals=(),
+ answers=(foreign_answer,),
+ )
+
+
+def test_rejects_missing_oracle_answer_references(tmp_path: Path) -> None:
+ question, observation, fact, _ = _spatial_records()
+
+ with pytest.raises(ValueError, match="missing oracle answer"):
+ write_bundle(
+ tmp_path,
+ episode_id="episode_1",
+ source_video_sha256="1" * 64,
+ questions=(question,),
+ observations=(observation,),
+ relation_facts=(fact,),
+ relation_intervals=(),
+ answers=(),
+ )
+
+
+def test_rejects_duplicate_oracle_answer_references(tmp_path: Path) -> None:
+ question, observation, fact, answer = _spatial_records()
+
+ with pytest.raises(ValueError, match="duplicate oracle answer"):
+ write_bundle(
+ tmp_path,
+ episode_id="episode_1",
+ source_video_sha256="1" * 64,
+ questions=(question,),
+ observations=(observation,),
+ relation_facts=(fact,),
+ relation_intervals=(),
+ answers=(answer, answer),
+ )
+
+
+def test_rejects_temporal_questions_with_foreign_relation_references(tmp_path: Path) -> None:
+ _, observation, fact, _ = _spatial_records()
+ references = (fact.relation_id, f"relation_{'0' * 64}")
+ question_id = stable_question_id(
+ episode_id="episode_1",
+ predicate=TemporalPredicate.BEFORE.value,
+ question_kind=QuestionKind.TEMPORAL.value,
+ reference_ids=references,
+ )
+ question = Question(
+ question_id=question_id,
+ episode_id="episode_1",
+ text="Did the first relation happen before the second?",
+ question_kind=QuestionKind.TEMPORAL,
+ predicate=TemporalPredicate.BEFORE,
+ object_ids=(),
+ reference_ids=references,
+ )
+ answer = OracleAnswer(question_id=question_id, expected=True, evidence_frame_ids=(3,))
+
+ with pytest.raises(ValueError, match="foreign relation reference"):
+ write_bundle(
+ tmp_path,
+ episode_id="episode_1",
+ source_video_sha256="1" * 64,
+ questions=(question,),
+ observations=(observation,),
+ relation_facts=(fact,),
+ relation_intervals=(_interval_for_fact(fact),),
+ answers=(answer,),
+ )
+
+
+def test_rejects_answers_with_foreign_interval_evidence(tmp_path: Path) -> None:
+ question, observation, fact, _ = _spatial_records()
+ answer = OracleAnswer(
+ question_id=question.question_id,
+ expected=True,
+ evidence_interval_ids=(f"interval_{'0' * 64}",),
+ )
+
+ with pytest.raises(ValueError, match="foreign interval reference"):
+ write_bundle(
+ tmp_path,
+ episode_id="episode_1",
+ source_video_sha256="1" * 64,
+ questions=(question,),
+ observations=(observation,),
+ relation_facts=(fact,),
+ relation_intervals=(_interval_for_fact(fact),),
+ answers=(answer,),
+ )
+
+
+def test_rejects_records_from_a_foreign_episode(tmp_path: Path) -> None:
+ question, observation, fact, answer = _spatial_records()
+ foreign_question = question.model_copy(update={"episode_id": "episode_2"})
+
+ with pytest.raises(ValueError, match="foreign episode"):
+ write_bundle(
+ tmp_path,
+ episode_id="episode_1",
+ source_video_sha256="1" * 64,
+ questions=(foreign_question,),
+ observations=(observation,),
+ relation_facts=(fact,),
+ relation_intervals=(),
+ answers=(answer,),
+ )
+
+
+def test_loader_rejects_artifacts_that_do_not_match_the_manifest(tmp_path: Path) -> None:
+ question, observation, fact, answer = _spatial_records()
+ write_bundle(
+ tmp_path,
+ episode_id="episode_1",
+ source_video_sha256="1" * 64,
+ questions=(question,),
+ observations=(observation,),
+ relation_facts=(fact,),
+ relation_intervals=(),
+ answers=(answer,),
+ )
+ questions_path = tmp_path / "public" / "questions.jsonl"
+ questions_path.write_bytes(questions_path.read_bytes() + b"\n")
+
+ with pytest.raises(ValueError, match="artifact digest"):
+ load_bundle(tmp_path)
+
+
+def test_loader_rejects_traversal_artifact_paths(tmp_path: Path) -> None:
+ question, observation, fact, answer = _spatial_records()
+ write_bundle(
+ tmp_path,
+ episode_id="episode_1",
+ source_video_sha256="1" * 64,
+ questions=(question,),
+ observations=(observation,),
+ relation_facts=(fact,),
+ relation_intervals=(),
+ answers=(answer,),
+ )
+ public_path = tmp_path / "public" / "manifest.json"
+ public = PublicBundleManifest.model_validate_json(public_path.read_bytes())
+ public = public.model_copy(
+ update={"questions": public.questions.model_copy(update={"path": "../questions.jsonl"})}
+ )
+ public_bytes = f"{canonical_model_json(public)}\n".encode()
+ public_path.write_bytes(public_bytes)
+ oracle_path = tmp_path / "oracle" / "manifest.json"
+ oracle = OracleBundleManifest.model_validate_json(oracle_path.read_bytes()).model_copy(
+ update={"public_manifest_sha256": sha256(public_bytes).hexdigest()}
+ )
+ oracle_path.write_text(f"{canonical_model_json(oracle)}\n", encoding="utf-8")
+
+ with pytest.raises(ValueError, match="relative canonical path"):
+ load_bundle(tmp_path)
+
+
+def test_loader_rejects_symlinked_artifacts(tmp_path: Path) -> None:
+ question, observation, fact, answer = _spatial_records()
+ bundle_root = tmp_path / "bundle"
+ write_bundle(
+ bundle_root,
+ episode_id="episode_1",
+ source_video_sha256="1" * 64,
+ questions=(question,),
+ observations=(observation,),
+ relation_facts=(fact,),
+ relation_intervals=(),
+ answers=(answer,),
+ )
+ questions_path = bundle_root / "public" / "questions.jsonl"
+ external_questions_path = tmp_path / "questions.jsonl"
+ questions_path.replace(external_questions_path)
+ questions_path.symlink_to(external_questions_path)
+
+ with pytest.raises(ValueError, match="symlink"):
+ load_bundle(bundle_root)
+
+
+def test_loader_rejects_symlinked_manifests(tmp_path: Path) -> None:
+ question, observation, fact, answer = _spatial_records()
+ bundle_root = tmp_path / "bundle"
+ write_bundle(
+ bundle_root,
+ episode_id="episode_1",
+ source_video_sha256="1" * 64,
+ questions=(question,),
+ observations=(observation,),
+ relation_facts=(fact,),
+ relation_intervals=(),
+ answers=(answer,),
+ )
+ manifest_path = bundle_root / "oracle" / "manifest.json"
+ external_manifest_path = tmp_path / "manifest.json"
+ manifest_path.replace(external_manifest_path)
+ manifest_path.symlink_to(external_manifest_path)
+
+ with pytest.raises(ValueError, match="symlink"):
+ load_bundle(bundle_root)
+
+
+def test_writer_rejects_symlinked_output_directories(tmp_path: Path) -> None:
+ question, observation, fact, answer = _spatial_records()
+ bundle_root = tmp_path / "bundle"
+ bundle_root.mkdir()
+ external_root = tmp_path / "external"
+ external_root.mkdir()
+ (bundle_root / "public").symlink_to(external_root, target_is_directory=True)
+
+ with pytest.raises(ValueError, match="symlink"):
+ write_bundle(
+ bundle_root,
+ episode_id="episode_1",
+ source_video_sha256="1" * 64,
+ questions=(question,),
+ observations=(observation,),
+ relation_facts=(fact,),
+ relation_intervals=(),
+ answers=(answer,),
+ )
+
+ assert not tuple(external_root.iterdir())
+
+
+def test_rejects_duplicate_interval_identities(tmp_path: Path) -> None:
+ question, observation, fact, answer = _spatial_records()
+ interval = _interval_for_fact(fact)
+
+ with pytest.raises(ValueError, match="duplicate interval ID"):
+ write_bundle(
+ tmp_path,
+ episode_id="episode_1",
+ source_video_sha256="1" * 64,
+ questions=(question,),
+ observations=(observation,),
+ relation_facts=(fact,),
+ relation_intervals=(interval, interval),
+ answers=(answer,),
+ )
+
+
+def test_loader_rejects_malformed_episode_metadata_with_valid_digests(tmp_path: Path) -> None:
+ question, observation, fact, answer = _spatial_records()
+ write_bundle(
+ tmp_path,
+ episode_id="episode_1",
+ source_video_sha256="1" * 64,
+ questions=(question,),
+ observations=(observation,),
+ relation_facts=(fact,),
+ relation_intervals=(),
+ answers=(answer,),
+ )
+ episode_bytes = b"{}\n"
+ (tmp_path / "public" / "episode.json").write_bytes(episode_bytes)
+ public_path = tmp_path / "public" / "manifest.json"
+ public = PublicBundleManifest.model_validate_json(public_path.read_bytes())
+ public = public.model_copy(
+ update={
+ "episode": public.episode.model_copy(
+ update={"sha256": sha256(episode_bytes).hexdigest()}
+ )
+ }
+ )
+ public_bytes = f"{canonical_model_json(public)}\n".encode()
+ public_path.write_bytes(public_bytes)
+ oracle_path = tmp_path / "oracle" / "manifest.json"
+ oracle = OracleBundleManifest.model_validate_json(oracle_path.read_bytes()).model_copy(
+ update={"public_manifest_sha256": sha256(public_bytes).hexdigest()}
+ )
+ oracle_path.write_text(f"{canonical_model_json(oracle)}\n", encoding="utf-8")
+
+ with pytest.raises(ValueError, match="episode metadata"):
+ load_bundle(tmp_path)
diff --git a/dimos/benchmark/spatiotemporal/test_demo.py b/dimos/benchmark/spatiotemporal/test_demo.py
new file mode 100644
index 0000000000..768b909af7
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/test_demo.py
@@ -0,0 +1,306 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for the one-command spatiotemporal video-QA demonstration."""
+
+import inspect
+import json
+import os
+from pathlib import Path
+from types import SimpleNamespace
+
+import cv2
+import numpy as np
+import pytest
+
+from dimos.benchmark.spatiotemporal import demo
+from dimos.benchmark.spatiotemporal.demo import (
+ TimestampScriptedPublicVlModel,
+ _run_temporal_memory_candidate,
+ _trim_video,
+ run_demo,
+)
+from dimos.msgs.sensor_msgs.Image import Image
+
+
+def test_scripted_candidate_answer_depends_only_on_public_question() -> None:
+ model = TimestampScriptedPublicVlModel()
+ image = Image.from_numpy(np.zeros((2, 2, 3), dtype=np.uint8))
+ question = "Did relation_a happen before relation_b?"
+ first = model.query(
+ image,
+ f'**Question:** {question}\n**Context:** {{"timestamp": 1}}',
+ )
+ second = model.query(
+ image,
+ f'**Question:** {question}\n**Context:** {{"timestamp": 999}}',
+ )
+
+ assert first == second
+ assert first in {"yes", "no"}
+
+
+@pytest.mark.parametrize("duration_s", [14, 31])
+def test_demo_rejects_video_outside_constrained_duration(duration_s: int) -> None:
+ with pytest.raises(ValueError, match="between 15 and 30 seconds"):
+ run_demo(
+ Path("not-opened.mp4"),
+ Path("not-created"),
+ duration_s=duration_s,
+ )
+
+
+def test_candidate_boundary_accepts_no_private_bundle() -> None:
+ assert tuple(inspect.signature(_run_temporal_memory_candidate).parameters) == (
+ "video",
+ "questions",
+ "output_root",
+ "duration_s",
+ )
+
+
+def test_temporal_memory_subprocess_overrides_ambient_log_path(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ outside = tmp_path / "ambient-logs"
+ candidate_root = tmp_path / "candidate"
+ candidate_root.mkdir()
+ video = candidate_root / "video.mp4"
+ video.write_bytes(b"video")
+ observed: dict[str, object] = {}
+ monkeypatch.setenv("DIMOS_RUN_LOG_DIR", str(outside))
+
+ def fake_run(
+ command: list[str],
+ *,
+ check: bool,
+ cwd: Path,
+ env: dict[str, str],
+ ) -> SimpleNamespace:
+ observed.update(command=command, check=check, cwd=cwd, env=env)
+ questions_path = Path(command[command.index("--questions") + 1])
+ result_path = Path(command[command.index("--result") + 1])
+ assert questions_path.read_text(encoding="utf-8") == ""
+ result_path.write_text(
+ json.dumps({"answers": {}, "runtime": {"ready": True}}),
+ encoding="utf-8",
+ )
+ return SimpleNamespace(returncode=0)
+
+ monkeypatch.setattr(demo.subprocess, "run", fake_run)
+
+ answers, runtime = _run_temporal_memory_candidate(video, (), candidate_root, 25)
+
+ child_env = observed["env"]
+ assert isinstance(child_env, dict)
+ assert child_env["DIMOS_RUN_LOG_DIR"] == str(candidate_root / "run-logs")
+ assert os.environ["DIMOS_RUN_LOG_DIR"] == str(outside)
+ assert answers == {}
+ assert runtime == {"ready": True}
+ assert not outside.exists()
+
+
+def test_demo_uses_ephemeral_candidate_root_outside_teacher_output(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ source = tmp_path / "source.mp4"
+ source.write_bytes(b"video")
+ output_root = tmp_path / "teacher-output"
+ bundle = SimpleNamespace(questions=())
+ observed: dict[str, Path] = {}
+
+ def trim(_source: Path, target: Path, _duration_s: float) -> tuple[float, int]:
+ target.write_bytes(b"trimmed video")
+ return 1.0, 25
+
+ monkeypatch.setattr(demo, "_trim_video", trim)
+ monkeypatch.setattr(demo, "_teacher_run", lambda *_: (bundle, {}))
+
+ def candidate(
+ video: Path,
+ questions: tuple[()],
+ candidate_root: Path,
+ duration_s: int,
+ ) -> tuple[dict, dict]:
+ del questions, duration_s
+ observed["root"] = candidate_root
+ observed["video"] = video
+ assert candidate_root == candidate_root.resolve()
+ assert not candidate_root.is_relative_to(output_root.resolve())
+ assert video.parent == candidate_root
+ assert not video.is_relative_to(output_root.resolve())
+ assert video.read_bytes() == b"trimmed video"
+ assert not (candidate_root / "bundle-a").exists()
+ return {}, {}
+
+ monkeypatch.setattr(demo, "_run_temporal_memory_candidate", candidate)
+ monkeypatch.setattr(demo, "_score_candidate", lambda *_: {})
+ monkeypatch.setattr(
+ demo,
+ "write_evidence_viewer",
+ lambda *_: SimpleNamespace(
+ index_path="index.html", evidence_frame_count=0, question_count=0
+ ),
+ )
+
+ run_demo(source, output_root)
+
+ assert not observed["root"].exists()
+ assert not observed["video"].exists()
+
+
+def test_demo_writes_evaluator_evidence_viewer(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ source = tmp_path / "source.mp4"
+ source.write_bytes(b"video")
+ output_root = tmp_path / "output"
+ bundle = SimpleNamespace(questions=())
+ observed: dict[str, object] = {}
+
+ def trim(_source: Path, target: Path, _duration_s: float) -> tuple[float, int]:
+ target.write_bytes(b"trimmed video")
+ return 1.0, 25
+
+ def viewer(video: Path, private_bundle: object, root: Path) -> SimpleNamespace:
+ observed.update(video=video, bundle=private_bundle, root=root)
+ return SimpleNamespace(index_path="index.html", evidence_frame_count=3, question_count=7)
+
+ monkeypatch.setattr(demo, "_trim_video", trim)
+ monkeypatch.setattr(demo, "_teacher_run", lambda *_: (bundle, {}))
+ monkeypatch.setattr(demo, "_run_temporal_memory_candidate", lambda *_: ({}, {}))
+ monkeypatch.setattr(demo, "_score_candidate", lambda *_: {})
+ monkeypatch.setattr(demo, "write_evidence_viewer", viewer)
+
+ summary = run_demo(source, output_root)
+
+ assert observed == {
+ "video": output_root / "office_robot_25s.mp4",
+ "bundle": bundle,
+ "root": output_root / "evidence-viewer",
+ }
+ assert summary["review"] == {
+ "evidence_frame_count": 3,
+ "index_path": "evidence-viewer/index.html",
+ "question_count": 7,
+ }
+
+
+def test_invalid_stride_has_no_output_side_effect(tmp_path: Path) -> None:
+ output_root = tmp_path / "output"
+ with pytest.raises(ValueError, match="stride must be positive"):
+ run_demo(Path("not-opened.mp4"), output_root, frame_stride=0)
+ assert not output_root.exists()
+
+
+def test_demo_rejects_symlinked_output_root(tmp_path: Path) -> None:
+ source = tmp_path / "source.mp4"
+ source.write_bytes(b"not decoded")
+ actual_output = tmp_path / "actual"
+ actual_output.mkdir()
+ linked_output = tmp_path / "linked"
+ linked_output.symlink_to(actual_output, target_is_directory=True)
+
+ with pytest.raises(ValueError, match="contains a symlink"):
+ run_demo(source, linked_output)
+
+
+def test_demo_rejects_source_target_alias(tmp_path: Path) -> None:
+ output_root = tmp_path / "output"
+ output_root.mkdir()
+ source = output_root / "office_robot_25s.mp4"
+ source.write_bytes(b"not decoded")
+
+ with pytest.raises(ValueError, match="paths must differ"):
+ run_demo(source, output_root)
+
+
+def test_demo_rejects_symlinked_summary_artifact(tmp_path: Path) -> None:
+ source = tmp_path / "source.mp4"
+ source.write_bytes(b"not decoded")
+ output_root = tmp_path / "output"
+ output_root.mkdir()
+ outside = tmp_path / "outside.json"
+ (output_root / "summary.json").symlink_to(outside)
+
+ with pytest.raises(ValueError, match="contains a symlink"):
+ run_demo(source, output_root)
+ assert not outside.exists()
+
+
+def test_trim_rejects_hard_link_source_target_alias(tmp_path: Path) -> None:
+ source = tmp_path / "source.mp4"
+ source.write_bytes(b"must remain unchanged")
+ target = tmp_path / "target.mp4"
+ target.hardlink_to(source)
+
+ with pytest.raises(ValueError, match="paths must differ"):
+ _trim_video(source, target, 15.0)
+ assert source.read_bytes() == b"must remain unchanged"
+
+
+def test_trim_rejects_silent_video_writer_failure(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ source = tmp_path / "source.mp4"
+ source.write_bytes(b"source placeholder")
+ target = tmp_path / "target.mp4"
+
+ class SourceCapture:
+ def __init__(self) -> None:
+ self.reads = 0
+
+ def isOpened(self) -> bool:
+ return True
+
+ def get(self, prop: int) -> float:
+ return {
+ cv2.CAP_PROP_FPS: 1.0,
+ cv2.CAP_PROP_FRAME_WIDTH: 2.0,
+ cv2.CAP_PROP_FRAME_HEIGHT: 2.0,
+ }.get(prop, 0.0)
+
+ def read(self) -> tuple[bool, np.ndarray | None]:
+ self.reads += 1
+ return True, np.zeros((2, 2, 3), dtype=np.uint8)
+
+ def release(self) -> None:
+ return None
+
+ class VerificationCapture:
+ def isOpened(self) -> bool:
+ return True
+
+ def read(self) -> tuple[bool, None]:
+ return False, None
+
+ def release(self) -> None:
+ return None
+
+ class SilentWriter:
+ def isOpened(self) -> bool:
+ return True
+
+ def write(self, frame: np.ndarray) -> None:
+ return None
+
+ def release(self) -> None:
+ return None
+
+ captures = iter((SourceCapture(), VerificationCapture()))
+ monkeypatch.setattr(cv2, "VideoCapture", lambda _: next(captures))
+ monkeypatch.setattr(cv2, "VideoWriter", lambda *args: SilentWriter())
+
+ with pytest.raises(RuntimeError, match="0 decodable frames"):
+ _trim_video(source, target, 15.0)
diff --git a/dimos/benchmark/spatiotemporal/test_evidence_viewer.py b/dimos/benchmark/spatiotemporal/test_evidence_viewer.py
new file mode 100644
index 0000000000..4abc7599d5
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/test_evidence_viewer.py
@@ -0,0 +1,273 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for the evaluator-only, human-inspectable evidence viewer."""
+
+from pathlib import Path
+from types import SimpleNamespace
+from typing import cast
+
+import cv2
+import numpy as np
+import pytest
+
+from dimos.benchmark.spatiotemporal.evidence_viewer import (
+ EvidenceBundle,
+ _annotate_frame,
+ _object_color,
+ write_evidence_viewer,
+)
+from dimos.benchmark.spatiotemporal.models import (
+ BoundingBox2D,
+ ObjectObservation,
+ OracleAnswer,
+ Question,
+ QuestionKind,
+ SpatialPredicate,
+)
+from dimos.benchmark.spatiotemporal.utilities import stable_question_id
+
+
+class _FakeCapture:
+ def __init__(self, frame_count: int = 1) -> None:
+ self._frame_count = frame_count
+ self._frame_id = 0
+ self.released = False
+
+ def isOpened(self) -> bool:
+ return True
+
+ def read(self) -> tuple[bool, np.ndarray | None]:
+ if self._frame_id >= self._frame_count:
+ return False, None
+ frame = np.full((20, 30, 3), self._frame_id, dtype=np.uint8)
+ self._frame_id += 1
+ return True, frame
+
+ def release(self) -> None:
+ self.released = True
+
+
+def _bundle() -> EvidenceBundle:
+ question_id = stable_question_id(
+ episode_id="episode_1",
+ object_ids=("oven", "robot"),
+ predicate=SpatialPredicate.LEFT_OF.value,
+ question_kind=QuestionKind.SPATIAL.value,
+ )
+ return cast(
+ "EvidenceBundle",
+ SimpleNamespace(
+ questions=(
+ Question(
+ question_id=question_id,
+ episode_id="episode_1",
+ text="Did ever appear left of robot?",
+ question_kind=QuestionKind.SPATIAL,
+ predicate=SpatialPredicate.LEFT_OF,
+ object_ids=("oven", "robot"),
+ ),
+ ),
+ answers=(
+ OracleAnswer(
+ question_id=question_id,
+ expected=True,
+ evidence_frame_ids=(0,),
+ ),
+ ),
+ observations=(
+ ObjectObservation(
+ episode_id="episode_1",
+ frame_id=0,
+ timestamp_s=0.0,
+ object_id="oven",
+ label="",
+ box=BoundingBox2D(x_min=0.1, y_min=0.1, x_max=0.4, y_max=0.6),
+ confidence=0.9,
+ ),
+ ObjectObservation(
+ episode_id="episode_1",
+ frame_id=1,
+ timestamp_s=0.1,
+ object_id="robot",
+ label="robot",
+ box=BoundingBox2D(x_min=0.5, y_min=0.1, x_max=0.8, y_max=0.6),
+ confidence=0.8,
+ ),
+ ),
+ relation_intervals=(),
+ ),
+ )
+
+
+def test_writes_deterministic_escaped_pseudo_label_evidence(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ source = tmp_path / "video.mp4"
+ source.write_bytes(b"decoded by fake capture")
+ captures: list[_FakeCapture] = []
+
+ def capture_factory(_: str) -> _FakeCapture:
+ capture = _FakeCapture()
+ captures.append(capture)
+ return capture
+
+ monkeypatch.setattr(cv2, "VideoCapture", capture_factory)
+ roots = (tmp_path / "first", tmp_path / "second")
+
+ results = tuple(write_evidence_viewer(source, _bundle(), root) for root in roots)
+
+ assert results[0] == results[1]
+ assert results[0].evidence_frame_count == 1
+ assert results[0].question_count == 1
+ outputs = [
+ {path.relative_to(root): path.read_bytes() for path in root.rglob("*") if path.is_file()}
+ for root in roots
+ ]
+ assert outputs[0] == outputs[1]
+ html = (roots[0] / "index.html").read_text(encoding="utf-8")
+ assert "teacher pseudo-labels, not ground truth" in html
+ assert "One generated proof chain" in html
+ assert "Robot-motion verification" in html
+ assert "Motion evidence incomplete; review required" in html
+ assert 'aria-pressed="true">All' in html
+ assert 'aria-pressed="false">Spatial' in html
+ assert "Relation timeline" in html
+ assert "Future-project leverage" in html
+ assert "Questions and private oracle" in html
+ assert "Did <oven> ever appear left of robot?" in html
+ assert "<oven>" in html
+ assert "" not in html
+ assert (roots[0] / "frames" / "frame_000000.jpg").is_file()
+ assert "frame_000001.jpg" not in html
+ assert not (roots[0] / "frames" / "frame_000001.jpg").exists()
+ assert all(capture.released for capture in captures)
+
+
+def test_annotation_maps_normalized_box_to_frame_pixels() -> None:
+ observation = _bundle().observations[0]
+ annotated = _annotate_frame(np.zeros((20, 30, 3), dtype=np.uint8), (observation,))
+
+ assert tuple(int(component) for component in annotated[12, 12]) == _object_color(
+ observation.object_id
+ )
+
+
+def test_resolves_temporal_interval_to_sparse_nonzero_frame(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ source = tmp_path / "video.mp4"
+ source.write_bytes(b"video")
+ base = _bundle()
+ interval_id = f"interval_{'1' * 64}"
+ bundle = cast(
+ "EvidenceBundle",
+ SimpleNamespace(
+ questions=base.questions,
+ answers=(
+ OracleAnswer(
+ question_id=base.questions[0].question_id,
+ expected=True,
+ evidence_interval_ids=(interval_id,),
+ ),
+ ),
+ observations=base.observations,
+ relation_intervals=(
+ SimpleNamespace(
+ interval_id=interval_id,
+ relation_id=f"relation_{'2' * 64}",
+ episode_id="episode_1",
+ subject_id="oven",
+ predicate=SpatialPredicate.LEFT_OF,
+ object_id="robot",
+ start_frame_id=2,
+ end_frame_id=2,
+ start_timestamp_s=0.2,
+ end_timestamp_s=0.2,
+ evidence_frame_ids=(2,),
+ ),
+ ),
+ ),
+ )
+ monkeypatch.setattr(cv2, "VideoCapture", lambda _: _FakeCapture(frame_count=3))
+
+ result = write_evidence_viewer(source, bundle, tmp_path / "viewer")
+
+ assert result.evidence_frame_count == 1
+ assert (tmp_path / "viewer" / "frames" / "frame_000002.jpg").is_file()
+ html = (tmp_path / "viewer" / "index.html").read_text(encoding="utf-8")
+ assert "frame_000002.jpg" in html
+
+
+def test_rejects_symlinked_viewer_output(tmp_path: Path) -> None:
+ source = tmp_path / "video.mp4"
+ source.write_bytes(b"video")
+ actual = tmp_path / "actual"
+ actual.mkdir()
+ linked = tmp_path / "linked"
+ linked.symlink_to(actual, target_is_directory=True)
+
+ with pytest.raises(ValueError, match="symlink"):
+ write_evidence_viewer(source, _bundle(), linked)
+
+ assert not tuple(actual.iterdir())
+
+
+def test_rejects_symlinked_output_ancestor(tmp_path: Path) -> None:
+ source = tmp_path / "video.mp4"
+ source.write_bytes(b"video")
+ external = tmp_path / "external"
+ external.mkdir()
+ linked_parent = tmp_path / "linked-parent"
+ linked_parent.symlink_to(external, target_is_directory=True)
+
+ with pytest.raises(ValueError, match="symlink"):
+ write_evidence_viewer(source, _bundle(), linked_parent / "viewer")
+
+ assert not tuple(external.iterdir())
+
+
+def test_rejects_source_inside_output_without_deleting_it(tmp_path: Path) -> None:
+ output = tmp_path / "viewer"
+ output.mkdir()
+ source = output / "video.mp4"
+ source.write_bytes(b"must survive")
+
+ with pytest.raises(ValueError, match="inside viewer output"):
+ write_evidence_viewer(source, _bundle(), output)
+
+ assert source.read_bytes() == b"must survive"
+
+
+def test_failed_render_preserves_existing_viewer(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ source = tmp_path / "video.mp4"
+ source.write_bytes(b"video")
+ output = tmp_path / "viewer"
+ output.mkdir()
+ marker = output / "existing.txt"
+ marker.write_text("keep me", encoding="utf-8")
+
+ class BrokenCapture(_FakeCapture):
+ def read(self) -> tuple[bool, None]:
+ return False, None
+
+ monkeypatch.setattr(cv2, "VideoCapture", lambda _: BrokenCapture())
+
+ with pytest.raises(RuntimeError, match="decode evidence frame"):
+ write_evidence_viewer(source, _bundle(), output)
+
+ assert marker.read_text(encoding="utf-8") == "keep me"
+ assert tuple(output.iterdir()) == (marker,)
diff --git a/dimos/benchmark/spatiotemporal/test_generation.py b/dimos/benchmark/spatiotemporal/test_generation.py
new file mode 100644
index 0000000000..eda84f4f05
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/test_generation.py
@@ -0,0 +1,358 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Behavioral tests for dataset question generation."""
+
+from collections import Counter
+
+import pytest
+
+from dimos.benchmark.spatiotemporal.generation import (
+ generate_spatial_questions,
+ generate_temporal_question_cases,
+)
+from dimos.benchmark.spatiotemporal.models import (
+ RelationFact,
+ RelationInterval,
+ SpatialPredicate,
+ TemporalPredicate,
+)
+from dimos.benchmark.spatiotemporal.utilities import (
+ SCHEMA_VERSION,
+ JsonValue,
+ stable_id,
+)
+
+
+def _interval(
+ subject_id: str,
+ predicate: SpatialPredicate,
+ object_id: str,
+ start_frame_id: int,
+ end_frame_id: int,
+ start_timestamp_s: float | None = None,
+ end_timestamp_s: float | None = None,
+ episode_id: str = "episode_1",
+) -> RelationInterval:
+ if start_timestamp_s is None:
+ start_timestamp_s = start_frame_id / 10
+ if end_timestamp_s is None:
+ end_timestamp_s = end_frame_id / 10
+ relation_id = stable_id(
+ "relation",
+ {
+ "object_ids": (subject_id, object_id),
+ "predicate": predicate.value,
+ "schema_version": SCHEMA_VERSION,
+ },
+ )
+ contract: dict[str, JsonValue] = {
+ "end_frame_id": end_frame_id,
+ "end_timestamp_s": end_timestamp_s,
+ "episode_id": episode_id,
+ "object_id": object_id,
+ "predicate": predicate.value,
+ "relation_id": relation_id,
+ "schema_version": SCHEMA_VERSION,
+ "start_frame_id": start_frame_id,
+ "start_timestamp_s": start_timestamp_s,
+ "subject_id": subject_id,
+ }
+ return RelationInterval(
+ interval_id=stable_id("interval", contract),
+ relation_id=relation_id,
+ episode_id=episode_id,
+ subject_id=subject_id,
+ predicate=predicate,
+ object_id=object_id,
+ start_frame_id=start_frame_id,
+ end_frame_id=end_frame_id,
+ start_timestamp_s=start_timestamp_s,
+ end_timestamp_s=end_timestamp_s,
+ evidence_frame_ids=tuple(range(start_frame_id, end_frame_id + 1)),
+ )
+
+
+def test_generates_balanced_spatial_question_cases() -> None:
+ from dimos.benchmark.spatiotemporal.generation import generate_spatial_question_cases
+
+ relation_id = stable_id(
+ "relation",
+ {
+ "object_ids": ("obj_red", "obj_blue"),
+ "predicate": SpatialPredicate.LEFT_OF.value,
+ "schema_version": SCHEMA_VERSION,
+ },
+ )
+ fact = RelationFact(
+ relation_id=relation_id,
+ episode_id="episode_1",
+ frame_id=12,
+ timestamp_s=0.5,
+ subject_id="obj_red",
+ predicate=SpatialPredicate.LEFT_OF,
+ object_id="obj_blue",
+ evidence_frame_ids=(12,),
+ )
+
+ cases = generate_spatial_question_cases((fact,))
+
+ assert len(cases) == 2
+ assert Counter(answer.expected for _, answer in cases) == {True: 1, False: 1}
+ assert {(question.predicate, answer.expected) for question, answer in cases} == {
+ (SpatialPredicate.LEFT_OF, True),
+ (SpatialPredicate.RIGHT_OF, False),
+ }
+ assert all(answer.evidence_frame_ids == (12,) for _, answer in cases)
+
+
+def test_spatial_cases_are_order_independent_when_relation_flips() -> None:
+ def fact(predicate: SpatialPredicate, frame_id: int) -> RelationFact:
+ return RelationFact(
+ relation_id=stable_id(
+ "relation",
+ {
+ "object_ids": ("obj_a", "obj_b"),
+ "predicate": predicate.value,
+ "schema_version": SCHEMA_VERSION,
+ },
+ ),
+ episode_id="episode_1",
+ frame_id=frame_id,
+ timestamp_s=frame_id / 10,
+ subject_id="obj_a",
+ predicate=predicate,
+ object_id="obj_b",
+ evidence_frame_ids=(frame_id,),
+ )
+
+ from dimos.benchmark.spatiotemporal.generation import generate_spatial_question_cases
+
+ facts = (fact(SpatialPredicate.LEFT_OF, 1), fact(SpatialPredicate.RIGHT_OF, 2))
+ forward = generate_spatial_question_cases(facts)
+ backward = generate_spatial_question_cases(tuple(reversed(facts)))
+
+ assert tuple(
+ (question.model_dump_json(), answer.model_dump_json()) for question, answer in forward
+ ) == tuple(
+ (question.model_dump_json(), answer.model_dump_json()) for question, answer in backward
+ )
+ assert Counter(answer.expected for _, answer in forward) == {True: 2}
+
+
+def test_generates_one_public_spatial_question_per_accepted_relation() -> None:
+ relation_id = stable_id(
+ "relation",
+ {
+ "object_ids": ("obj_red", "obj_blue"),
+ "predicate": SpatialPredicate.LEFT_OF.value,
+ "schema_version": SCHEMA_VERSION,
+ },
+ )
+ facts = tuple(
+ RelationFact(
+ relation_id=relation_id,
+ episode_id="episode_1",
+ frame_id=frame_id,
+ timestamp_s=timestamp_s,
+ subject_id="obj_red",
+ predicate=SpatialPredicate.LEFT_OF,
+ object_id="obj_blue",
+ evidence_frame_ids=(frame_id,),
+ )
+ for frame_id, timestamp_s in ((12, 0.5), (13, 0.6))
+ )
+
+ questions = generate_spatial_questions(facts)
+
+ assert len(questions) == 1
+ question = questions[0]
+ assert (
+ question.question_id
+ == "question_6c4562bfb3f21f38d6086b9331af250ebd473ce9162af6bf6cb8a28d94978f83"
+ )
+ assert question.text == "Did obj_red ever appear left of obj_blue?"
+ assert question.object_ids == ("obj_red", "obj_blue")
+ public_record = question.model_dump(mode="json")
+ assert (
+ not {
+ "relation_id",
+ "frame_id",
+ "timestamp_s",
+ "evidence_frame_ids",
+ "expected",
+ }
+ & public_record.keys()
+ )
+
+
+def test_spatial_question_ids_are_scoped_to_episode() -> None:
+ relation_id = stable_id(
+ "relation",
+ {
+ "object_ids": ("obj_red", "obj_blue"),
+ "predicate": SpatialPredicate.LEFT_OF.value,
+ "schema_version": SCHEMA_VERSION,
+ },
+ )
+ fact = RelationFact(
+ relation_id=relation_id,
+ episode_id="episode_1",
+ frame_id=12,
+ timestamp_s=0.5,
+ subject_id="obj_red",
+ predicate=SpatialPredicate.LEFT_OF,
+ object_id="obj_blue",
+ evidence_frame_ids=(12,),
+ )
+
+ first = generate_spatial_questions((fact,))[0]
+ second = generate_spatial_questions((fact.model_copy(update={"episode_id": "episode_2"}),))[0]
+
+ assert first.question_id != second.question_id
+
+
+def test_rejects_facts_from_multiple_episodes() -> None:
+ relation_id = stable_id(
+ "relation",
+ {
+ "object_ids": ("obj_red", "obj_blue"),
+ "predicate": SpatialPredicate.LEFT_OF.value,
+ "schema_version": SCHEMA_VERSION,
+ },
+ )
+ facts = tuple(
+ RelationFact(
+ relation_id=relation_id,
+ episode_id=episode_id,
+ frame_id=frame_id,
+ timestamp_s=timestamp_s,
+ subject_id="obj_red",
+ predicate=SpatialPredicate.LEFT_OF,
+ object_id="obj_blue",
+ evidence_frame_ids=(frame_id,),
+ )
+ for episode_id, frame_id, timestamp_s in (
+ ("episode_1", 12, 0.5),
+ ("episode_2", 13, 0.6),
+ )
+ )
+
+ with pytest.raises(ValueError, match="one episode"):
+ generate_spatial_questions(facts)
+
+
+def test_generates_balanced_temporal_questions_in_byte_stable_order() -> None:
+ first = _interval("obj_red", SpatialPredicate.LEFT_OF, "obj_blue", 1, 2)
+ second = _interval("obj_green", SpatialPredicate.ABOVE, "obj_yellow", 4, 5)
+
+ cases = generate_temporal_question_cases((second, first))
+
+ assert tuple(
+ (question.model_dump_json(), answer.model_dump_json()) for question, answer in cases
+ ) == tuple(
+ (question.model_dump_json(), answer.model_dump_json())
+ for question, answer in generate_temporal_question_cases((first, second))
+ )
+ assert {(question.predicate, question.reference_ids) for question, _ in cases} == {
+ (TemporalPredicate.BEFORE, (first.relation_id, second.relation_id)),
+ (TemporalPredicate.AFTER, (second.relation_id, first.relation_id)),
+ (TemporalPredicate.AFTER, (first.relation_id, second.relation_id)),
+ (TemporalPredicate.BEFORE, (second.relation_id, first.relation_id)),
+ }
+ assert Counter((question.predicate, answer.expected) for question, answer in cases) == {
+ (TemporalPredicate.BEFORE, True): 1,
+ (TemporalPredicate.BEFORE, False): 1,
+ (TemporalPredicate.AFTER, True): 1,
+ (TemporalPredicate.AFTER, False): 1,
+ }
+
+
+def test_temporal_questions_use_human_readable_relation_descriptions() -> None:
+ first = _interval("obj_red", SpatialPredicate.LEFT_OF, "obj_blue", 1, 2)
+ second = _interval("obj_green", SpatialPredicate.ABOVE, "obj_yellow", 4, 5)
+
+ questions = tuple(question for question, _ in generate_temporal_question_cases((first, second)))
+
+ assert all("relation_" not in question.text for question in questions)
+ assert all("obj_red left of obj_blue" in question.text for question in questions)
+ assert all("obj_green above obj_yellow" in question.text for question in questions)
+
+
+def test_temporal_question_ids_are_scoped_to_episode() -> None:
+ def question_ids(episode_id: str) -> set[str]:
+ first = _interval(
+ "obj_red",
+ SpatialPredicate.LEFT_OF,
+ "obj_blue",
+ 1,
+ 2,
+ episode_id=episode_id,
+ )
+ second = _interval(
+ "obj_green",
+ SpatialPredicate.ABOVE,
+ "obj_yellow",
+ 4,
+ 5,
+ episode_id=episode_id,
+ )
+ return {
+ question.question_id
+ for question, _ in generate_temporal_question_cases((first, second))
+ }
+
+ assert question_ids("episode_1").isdisjoint(question_ids("episode_2"))
+
+
+def test_omits_temporal_questions_when_any_interval_pair_is_non_strict() -> None:
+ relation_a_early = _interval("obj_red", SpatialPredicate.LEFT_OF, "obj_blue", 1, 2)
+ relation_a_overlap = _interval("obj_red", SpatialPredicate.LEFT_OF, "obj_blue", 4, 7)
+ relation_b = _interval("obj_green", SpatialPredicate.ABOVE, "obj_yellow", 5, 6)
+
+ assert (
+ generate_temporal_question_cases((relation_a_early, relation_a_overlap, relation_b)) == ()
+ )
+
+
+def test_omits_temporal_questions_with_bidirectional_interval_proofs() -> None:
+ relation_a_early = _interval("obj_red", SpatialPredicate.LEFT_OF, "obj_blue", 1, 2)
+ relation_b_early = _interval("obj_green", SpatialPredicate.ABOVE, "obj_yellow", 4, 5)
+ relation_b_late = _interval("obj_green", SpatialPredicate.ABOVE, "obj_yellow", 7, 8)
+ relation_a_late = _interval("obj_red", SpatialPredicate.LEFT_OF, "obj_blue", 10, 11)
+
+ assert (
+ generate_temporal_question_cases(
+ (relation_a_late, relation_b_early, relation_a_early, relation_b_late)
+ )
+ == ()
+ )
+
+
+def test_requires_strict_frame_and_timestamp_interval_order() -> None:
+ first = _interval("obj_red", SpatialPredicate.LEFT_OF, "obj_blue", 1, 2)
+ touching = _interval("obj_green", SpatialPredicate.ABOVE, "obj_yellow", 2, 4)
+ overlapping = _interval("obj_green", SpatialPredicate.ABOVE, "obj_yellow", 1, 4)
+ disagreeing = _interval(
+ "obj_green",
+ SpatialPredicate.ABOVE,
+ "obj_yellow",
+ 4,
+ 5,
+ start_timestamp_s=-0.2,
+ end_timestamp_s=-0.1,
+ )
+
+ for unproven_second in (touching, overlapping, disagreeing):
+ assert generate_temporal_question_cases((first, unproven_second)) == ()
diff --git a/dimos/benchmark/spatiotemporal/test_intervals.py b/dimos/benchmark/spatiotemporal/test_intervals.py
new file mode 100644
index 0000000000..94ac58760a
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/test_intervals.py
@@ -0,0 +1,273 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for deterministic relation interval construction."""
+
+import pytest
+
+from dimos.benchmark.spatiotemporal.models import (
+ RelationFact,
+ RelationInterval,
+ SpatialPredicate,
+ TemporalPredicate,
+)
+from dimos.benchmark.spatiotemporal.utilities import SCHEMA_VERSION, stable_id
+
+
+def _relation_id(subject_id: str, predicate: SpatialPredicate, object_id: str) -> str:
+ return stable_id(
+ "relation",
+ {
+ "object_ids": (subject_id, object_id),
+ "predicate": predicate.value,
+ "schema_version": SCHEMA_VERSION,
+ },
+ )
+
+
+def _fact(
+ frame_id: int,
+ timestamp_s: float,
+ *,
+ episode_id: str = "episode_1",
+ subject_id: str = "mug_1",
+ predicate: SpatialPredicate = SpatialPredicate.LEFT_OF,
+ object_id: str = "laptop_1",
+) -> RelationFact:
+ return RelationFact(
+ relation_id=_relation_id(subject_id, predicate, object_id),
+ episode_id=episode_id,
+ frame_id=frame_id,
+ timestamp_s=timestamp_s,
+ subject_id=subject_id,
+ predicate=predicate,
+ object_id=object_id,
+ evidence_frame_ids=(frame_id,),
+ )
+
+
+def test_coalesces_relations_at_consecutive_sparse_samples() -> None:
+ from dimos.benchmark.spatiotemporal.intervals import build_relation_intervals
+
+ intervals = build_relation_intervals(
+ (_fact(0, 0.0), _fact(150, 5.0)),
+ sample_schedule=((0, 0.0), (150, 5.0), (300, 10.0)),
+ )
+
+ assert len(intervals) == 1
+ assert intervals[0].start_frame_id == 0
+ assert intervals[0].end_frame_id == 150
+ assert intervals[0].evidence_frame_ids == (0, 150)
+
+
+def test_relation_absent_at_intervening_sample_breaks_interval() -> None:
+ from dimos.benchmark.spatiotemporal.intervals import build_relation_intervals
+
+ intervals = build_relation_intervals(
+ (_fact(0, 0.0), _fact(300, 10.0)),
+ sample_schedule=((0, 0.0), (150, 5.0), (300, 10.0)),
+ )
+
+ assert [(interval.start_frame_id, interval.end_frame_id) for interval in intervals] == [
+ (0, 0),
+ (300, 300),
+ ]
+
+
+def test_coalesces_only_consecutive_samples_with_stable_relation_identity() -> None:
+ from dimos.benchmark.spatiotemporal.intervals import build_relation_intervals
+
+ intervals = build_relation_intervals(
+ (
+ _fact(1, 0.1),
+ _fact(2, 0.2),
+ _fact(
+ 2,
+ 0.2,
+ subject_id="lamp_1",
+ predicate=SpatialPredicate.ABOVE,
+ object_id="table_1",
+ ),
+ _fact(4, 0.4),
+ _fact(1, 0.1, episode_id="episode_2"),
+ )
+ )
+
+ assert [
+ (
+ interval.episode_id,
+ interval.subject_id,
+ interval.predicate,
+ interval.object_id,
+ interval.start_frame_id,
+ interval.end_frame_id,
+ interval.evidence_frame_ids,
+ )
+ for interval in intervals
+ ] == [
+ (
+ "episode_1",
+ "mug_1",
+ SpatialPredicate.LEFT_OF,
+ "laptop_1",
+ 1,
+ 2,
+ (1, 2),
+ ),
+ (
+ "episode_1",
+ "mug_1",
+ SpatialPredicate.LEFT_OF,
+ "laptop_1",
+ 4,
+ 4,
+ (4,),
+ ),
+ (
+ "episode_1",
+ "lamp_1",
+ SpatialPredicate.ABOVE,
+ "table_1",
+ 2,
+ 2,
+ (2,),
+ ),
+ (
+ "episode_2",
+ "mug_1",
+ SpatialPredicate.LEFT_OF,
+ "laptop_1",
+ 1,
+ 1,
+ (1,),
+ ),
+ ]
+
+
+@pytest.mark.parametrize(
+ "facts",
+ [
+ (
+ _fact(1, 0.1),
+ _fact(
+ 1,
+ 0.2,
+ subject_id="lamp_1",
+ predicate=SpatialPredicate.ABOVE,
+ object_id="table_1",
+ ),
+ ),
+ (_fact(1, 0.1), _fact(2, 0.1)),
+ (_fact(1, 0.2), _fact(2, 0.1)),
+ ],
+)
+def test_rejects_conflicting_episode_sample_schedules(
+ facts: tuple[RelationFact, ...],
+) -> None:
+ from dimos.benchmark.spatiotemporal.intervals import build_relation_intervals
+
+ with pytest.raises(ValueError, match="conflicting sample schedule"):
+ build_relation_intervals(facts)
+
+
+def test_derives_only_unanimous_strict_temporal_ordering() -> None:
+ from dimos.benchmark.spatiotemporal.intervals import (
+ build_relation_intervals,
+ derive_temporal_predicate,
+ )
+
+ earlier = _relation_id("mug_1", SpatialPredicate.LEFT_OF, "laptop_1")
+ later = _relation_id("lamp_1", SpatialPredicate.ABOVE, "table_1")
+
+ def paired_intervals(
+ first_frames: tuple[int, ...], second_frames: tuple[int, ...]
+ ) -> tuple[RelationInterval, ...]:
+ return build_relation_intervals(
+ tuple(_fact(frame, frame / 10) for frame in first_frames)
+ + tuple(
+ _fact(
+ frame,
+ frame / 10,
+ subject_id="lamp_1",
+ predicate=SpatialPredicate.ABOVE,
+ object_id="table_1",
+ )
+ for frame in second_frames
+ )
+ )
+
+ ordered_facts = (
+ _fact(1, 0.1),
+ _fact(2, 0.2),
+ _fact(
+ 4,
+ 0.4,
+ subject_id="lamp_1",
+ predicate=SpatialPredicate.ABOVE,
+ object_id="table_1",
+ ),
+ _fact(
+ 5,
+ 0.5,
+ subject_id="lamp_1",
+ predicate=SpatialPredicate.ABOVE,
+ object_id="table_1",
+ ),
+ )
+ ordered = build_relation_intervals(ordered_facts)
+
+ assert derive_temporal_predicate(earlier, later, ordered) is TemporalPredicate.BEFORE
+ assert derive_temporal_predicate(later, earlier, ordered) is TemporalPredicate.AFTER
+ assert (
+ derive_temporal_predicate(earlier, later, tuple(reversed(ordered)))
+ is TemporalPredicate.BEFORE
+ )
+
+ touching = paired_intervals((1, 2), (2, 3))
+ overlapping = paired_intervals((1, 2, 3), (2, 3, 4))
+ containing = paired_intervals((1, 2, 3, 4, 5), (2, 3, 4))
+ contradictory = build_relation_intervals(
+ (
+ *ordered_facts,
+ _fact(4, 0.4, episode_id="episode_2"),
+ _fact(
+ 1,
+ 0.1,
+ episode_id="episode_2",
+ subject_id="lamp_1",
+ predicate=SpatialPredicate.ABOVE,
+ object_id="table_1",
+ ),
+ )
+ )
+ coordinate_contradiction = build_relation_intervals((_fact(1, 0.4),)) + (
+ build_relation_intervals(
+ (
+ _fact(
+ 4,
+ 0.1,
+ subject_id="lamp_1",
+ predicate=SpatialPredicate.ABOVE,
+ object_id="table_1",
+ ),
+ )
+ )
+ )
+
+ assert derive_temporal_predicate(earlier, later, touching) is None
+ assert derive_temporal_predicate(earlier, later, overlapping) is None
+ assert derive_temporal_predicate(earlier, later, containing) is None
+ assert derive_temporal_predicate(earlier, "relation_" + "0" * 64, ordered) is None
+ assert derive_temporal_predicate(earlier, later, contradictory) is None
+ assert derive_temporal_predicate(earlier, later, coordinate_contradiction) is None
diff --git a/dimos/benchmark/spatiotemporal/test_models.py b/dimos/benchmark/spatiotemporal/test_models.py
new file mode 100644
index 0000000000..063ae5c731
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/test_models.py
@@ -0,0 +1,331 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for strict spatiotemporal QA contracts."""
+
+from pydantic import ValidationError
+import pytest
+
+from dimos.benchmark.spatiotemporal.models import (
+ BoundingBox2D,
+ BundleArtifact,
+ ObjectObservation,
+ OracleAnswer,
+ OracleBundleManifest,
+ PredictionStatus,
+ PublicBundleManifest,
+ Question,
+ QuestionKind,
+ RelationFact,
+ RelationInterval,
+ SpatialPredicate,
+ TemporalPredicate,
+)
+from dimos.benchmark.spatiotemporal.utilities import (
+ SCHEMA_VERSION,
+ stable_id,
+ stable_question_id,
+)
+
+
+def test_bounding_box_accepts_only_strict_normalized_valid_bounds() -> None:
+ box = BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.4, y_max=0.8)
+
+ assert box.model_dump() == {
+ "x_min": 0.1,
+ "y_min": 0.2,
+ "x_max": 0.4,
+ "y_max": 0.8,
+ }
+
+ invalid_payloads = (
+ {"x_min": 0.4, "y_min": 0.2, "x_max": 0.4, "y_max": 0.8},
+ {"x_min": 0.5, "y_min": 0.2, "x_max": 0.4, "y_max": 0.8},
+ {"x_min": 0.1, "y_min": 0.8, "x_max": 0.4, "y_max": 0.8},
+ {"x_min": -0.1, "y_min": 0.2, "x_max": 0.4, "y_max": 0.8},
+ {"x_min": 0.1, "y_min": 0.2, "x_max": 1.1, "y_max": 0.8},
+ {"x_min": float("nan"), "y_min": 0.2, "x_max": 0.4, "y_max": 0.8},
+ {"x_min": "0.1", "y_min": 0.2, "x_max": 0.4, "y_max": 0.8},
+ )
+ for payload in invalid_payloads:
+ with pytest.raises(ValidationError):
+ BoundingBox2D.model_validate(payload)
+
+ with pytest.raises(ValidationError):
+ BoundingBox2D.model_validate(
+ {
+ "x_min": 0.1,
+ "y_min": 0.2,
+ "x_max": 0.4,
+ "y_max": 0.8,
+ "unexpected": True,
+ }
+ )
+ with pytest.raises(ValidationError):
+ box.x_min = 0.0
+
+
+def test_object_observation_rejects_invalid_identity_time_and_confidence() -> None:
+ observation = ObjectObservation(
+ episode_id="episode_1",
+ frame_id=12,
+ timestamp_s=0.5,
+ object_id="mug_1",
+ label="mug",
+ box=BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.3, y_max=0.4),
+ confidence=0.9,
+ )
+
+ assert observation.object_id == "mug_1"
+
+ invalid_overrides = (
+ {"episode_id": ""},
+ {"frame_id": -1},
+ {"timestamp_s": float("inf")},
+ {"object_id": ""},
+ {"object_id": "mu\u0301g"},
+ {"label": ""},
+ {"confidence": -0.1},
+ {"confidence": 1.1},
+ )
+ payload = observation.model_dump()
+ for override in invalid_overrides:
+ with pytest.raises(ValidationError):
+ ObjectObservation.model_validate(payload | override)
+
+
+def test_object_observation_allows_finite_negative_timestamp_translation() -> None:
+ observation = ObjectObservation(
+ episode_id="episode_1",
+ frame_id=12,
+ timestamp_s=-0.5,
+ object_id="mug_1",
+ label="mug",
+ box=BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.3, y_max=0.4),
+ confidence=0.9,
+ )
+
+ assert observation.timestamp_s == -0.5
+
+
+def test_question_rejects_malformed_id_and_non_nfc_object_identity() -> None:
+ payload = {
+ "question_id": stable_question_id(
+ episode_id="episode_1",
+ object_ids=("obj_red", "obj_blue"),
+ predicate=SpatialPredicate.LEFT_OF.value,
+ question_kind=QuestionKind.SPATIAL.value,
+ ),
+ "episode_id": "episode_1",
+ "text": "Is the mug left of the laptop at the end?",
+ "question_kind": QuestionKind.SPATIAL,
+ "predicate": SpatialPredicate.LEFT_OF,
+ "object_ids": ("obj_red", "obj_blue"),
+ }
+
+ assert Question.model_validate(payload).object_ids == ("obj_red", "obj_blue")
+ with pytest.raises(ValidationError):
+ Question.model_validate(payload | {"question_id": "question_1"})
+ with pytest.raises(ValidationError):
+ Question.model_validate(payload | {"question_id": f"question_{'0' * 64}"})
+ with pytest.raises(ValidationError):
+ Question.model_validate(payload | {"object_ids": ("mu\u0301g_1", "laptop_1")})
+ with pytest.raises(ValidationError):
+ Question.model_validate(payload | {"object_ids": ("\ud800", "laptop_1")})
+ with pytest.raises(ValidationError):
+ Question.model_validate(payload | {"object_ids": ("obj_red", "obj_red")})
+
+
+def test_relation_fact_has_stable_private_sample_identity() -> None:
+ contract = {
+ "object_ids": ("obj_red", "obj_blue"),
+ "predicate": SpatialPredicate.LEFT_OF.value,
+ "schema_version": SCHEMA_VERSION,
+ }
+ fact = RelationFact(
+ relation_id=stable_id("relation", contract),
+ episode_id="episode_1",
+ frame_id=12,
+ timestamp_s=0.5,
+ subject_id="obj_red",
+ predicate=SpatialPredicate.LEFT_OF,
+ object_id="obj_blue",
+ evidence_frame_ids=(12,),
+ )
+
+ assert (
+ fact.relation_id
+ == "relation_d4350b93518e2cc7628cc560706c72ea137ce4ba887f4bced0d6a9513d0319b3"
+ )
+ with pytest.raises(ValidationError, match="relation ID"):
+ RelationFact.model_validate(fact.model_dump() | {"relation_id": f"relation_{'0' * 64}"})
+ with pytest.raises(ValidationError, match="must differ"):
+ RelationFact.model_validate(fact.model_dump() | {"object_id": "obj_red"})
+ with pytest.raises(ValidationError, match="must cite exactly"):
+ RelationFact.model_validate(fact.model_dump() | {"evidence_frame_ids": (11, 12)})
+ with pytest.raises(ValidationError, match="ordered and unique"):
+ RelationFact.model_validate(fact.model_dump() | {"evidence_frame_ids": (12, 12)})
+
+
+def test_relation_interval_has_strict_consistent_bounds() -> None:
+ relation_id = stable_id(
+ "relation",
+ {
+ "object_ids": ("obj_red", "obj_blue"),
+ "predicate": SpatialPredicate.LEFT_OF.value,
+ "schema_version": SCHEMA_VERSION,
+ },
+ )
+ contract = {
+ "end_frame_id": 14,
+ "end_timestamp_s": 0.7,
+ "episode_id": "episode_1",
+ "object_id": "obj_blue",
+ "predicate": SpatialPredicate.LEFT_OF.value,
+ "relation_id": relation_id,
+ "schema_version": SCHEMA_VERSION,
+ "start_frame_id": 12,
+ "start_timestamp_s": 0.5,
+ "subject_id": "obj_red",
+ }
+ interval = RelationInterval(
+ interval_id=stable_id("interval", contract),
+ relation_id=relation_id,
+ episode_id="episode_1",
+ subject_id="obj_red",
+ predicate=SpatialPredicate.LEFT_OF,
+ object_id="obj_blue",
+ start_frame_id=12,
+ end_frame_id=14,
+ start_timestamp_s=0.5,
+ end_timestamp_s=0.7,
+ evidence_frame_ids=(12, 13, 14),
+ )
+
+ assert interval.start_timestamp_s < interval.end_timestamp_s
+ for end_frame_id, end_timestamp_s in ((12, 0.7), (14, 0.5)):
+ contradictory_contract = contract | {
+ "end_frame_id": end_frame_id,
+ "end_timestamp_s": end_timestamp_s,
+ }
+ contradictory_payload = interval.model_dump() | {
+ "end_frame_id": end_frame_id,
+ "end_timestamp_s": end_timestamp_s,
+ "interval_id": stable_id("interval", contradictory_contract),
+ "evidence_frame_ids": (12,) if end_frame_id == 12 else (12, 13, 14),
+ }
+ with pytest.raises(ValidationError, match="frame and timestamp bounds"):
+ RelationInterval.model_validate(contradictory_payload)
+
+ for override in (
+ {"end_frame_id": 11},
+ {"end_timestamp_s": 0.4},
+ {"evidence_frame_ids": (12, 14, 13)},
+ {"interval_id": f"interval_{'0' * 64}"},
+ ):
+ with pytest.raises(ValidationError):
+ RelationInterval.model_validate(interval.model_dump() | override)
+
+
+def test_question_variants_are_disjoint_and_have_stable_ids() -> None:
+ references = (f"relation_{'1' * 64}", f"relation_{'2' * 64}")
+ question_id = stable_question_id(
+ episode_id="episode_1",
+ predicate=TemporalPredicate.BEFORE.value,
+ question_kind=QuestionKind.TEMPORAL.value,
+ reference_ids=references,
+ )
+ temporal = Question(
+ question_id=question_id,
+ episode_id="episode_1",
+ text="Did relation one happen before relation two?",
+ question_kind=QuestionKind.TEMPORAL,
+ predicate=TemporalPredicate.BEFORE,
+ object_ids=(),
+ reference_ids=references,
+ )
+
+ assert temporal.predicate is TemporalPredicate.BEFORE
+ invalid_variants = (
+ {"question_kind": QuestionKind.SPATIAL},
+ {"predicate": SpatialPredicate.LEFT_OF},
+ {"object_ids": ("obj_red", "obj_blue")},
+ {"reference_ids": (references[0], references[0])},
+ {"reference_ids": ("relation_not-a-digest", references[1])},
+ )
+ for override in invalid_variants:
+ with pytest.raises(ValidationError):
+ Question.model_validate(temporal.model_dump() | override)
+
+
+def test_oracle_evidence_and_prediction_status_contracts_are_private_and_closed() -> None:
+ question_id = f"question_{'1' * 64}"
+ oracle = OracleAnswer(
+ question_id=question_id,
+ expected=True,
+ evidence_frame_ids=(12,),
+ evidence_interval_ids=(f"interval_{'2' * 64}",),
+ )
+
+ assert oracle.evidence_interval_ids == (f"interval_{'2' * 64}",)
+ assert {status.value for status in PredictionStatus} == {
+ "correct",
+ "incorrect",
+ "missing",
+ "invalid",
+ }
+
+
+def test_public_and_oracle_manifests_share_only_release_identity() -> None:
+ source_sha256 = "2" * 64
+ bundle_id = stable_id(
+ "bundle",
+ {
+ "episode_id": "episode_1",
+ "schema_version": SCHEMA_VERSION,
+ "source_video_sha256": source_sha256,
+ },
+ )
+ public = PublicBundleManifest(
+ schema_version=SCHEMA_VERSION,
+ bundle_id=bundle_id,
+ episode_id="episode_1",
+ source_video_sha256=source_sha256,
+ episode=BundleArtifact(path="public/episode.json", sha256="3" * 64, record_count=1),
+ questions=BundleArtifact(path="public/questions.jsonl", sha256="4" * 64, record_count=12),
+ )
+ oracle = OracleBundleManifest(
+ schema_version=SCHEMA_VERSION,
+ bundle_id=public.bundle_id,
+ episode_id=public.episode_id,
+ source_video_sha256=source_sha256,
+ public_manifest_sha256="8" * 64,
+ observations=BundleArtifact(
+ path="oracle/observations.jsonl", sha256="5" * 64, record_count=30
+ ),
+ relation_intervals=BundleArtifact(
+ path="oracle/relation_intervals.jsonl", sha256="6" * 64, record_count=8
+ ),
+ relation_facts=BundleArtifact(
+ path="oracle/relation_facts.jsonl", sha256="9" * 64, record_count=20
+ ),
+ answers=BundleArtifact(path="oracle/answers.jsonl", sha256="7" * 64, record_count=12),
+ )
+
+ public_keys = set(public.model_dump())
+ assert "observations" not in public_keys
+ assert "answers" not in public_keys
+ assert oracle.bundle_id == public.bundle_id
+ with pytest.raises(ValidationError, match="relative canonical path"):
+ BundleArtifact(path="../oracle/answers.jsonl", sha256="7" * 64, record_count=12)
diff --git a/dimos/benchmark/spatiotemporal/test_observation_io.py b/dimos/benchmark/spatiotemporal/test_observation_io.py
new file mode 100644
index 0000000000..a8976afeeb
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/test_observation_io.py
@@ -0,0 +1,135 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for canonical teacher-observation persistence."""
+
+from pathlib import Path
+
+import pytest
+
+from dimos.benchmark.spatiotemporal.models import BoundingBox2D, ObjectObservation
+from dimos.benchmark.spatiotemporal.observation_io import read_observations, write_observations
+
+
+def _observation(object_id: str, label: str) -> ObjectObservation:
+ return ObjectObservation(
+ episode_id="episode_1",
+ frame_id=12,
+ timestamp_s=0.5,
+ object_id=object_id,
+ label=label,
+ box=BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.3, y_max=0.5),
+ confidence=0.9,
+ )
+
+
+def test_observation_jsonl_round_trip_is_canonical_and_strict(tmp_path: Path) -> None:
+ path = tmp_path / "observations.jsonl"
+ mug = _observation("mug_1", "mug")
+ laptop = _observation("laptop_1", "laptop")
+
+ write_observations(path, (mug, laptop))
+
+ expected = (
+ b'{"box":{"x_max":0.3,"x_min":0.1,"y_max":0.5,"y_min":0.2},'
+ b'"confidence":0.9,"episode_id":"episode_1","frame_id":12,'
+ b'"label":"laptop","object_id":"laptop_1","timestamp_s":0.5}\n'
+ b'{"box":{"x_max":0.3,"x_min":0.1,"y_max":0.5,"y_min":0.2},'
+ b'"confidence":0.9,"episode_id":"episode_1","frame_id":12,'
+ b'"label":"mug","object_id":"mug_1","timestamp_s":0.5}\n'
+ )
+ assert path.read_bytes() == expected
+ assert read_observations(path) == (laptop, mug)
+ write_observations(path, read_observations(path))
+ assert path.read_bytes() == expected
+
+ invalid_documents = (
+ (expected.splitlines(keepends=True)[0] * 2, "duplicate observation identity"),
+ (
+ expected.splitlines(keepends=True)[0]
+ + expected.splitlines(keepends=True)[0].replace(
+ b'"label":"laptop"', b'"label":"computer"'
+ ),
+ "conflicting observation identity",
+ ),
+ (expected.replace(b'"confidence":0.9', b'"confidence":NaN', 1), "invalid JSON"),
+ (
+ expected.replace(b'"timestamp_s":0.5', b'"timestamp_s":1e999', 1),
+ "invalid observation",
+ ),
+ (expected.replace(b'"x_max":0.3', b'"x_max":1e999', 1), "invalid observation"),
+ (expected.replace(b'"x_min":0.1', b'"x_min":0.4', 1), "invalid observation"),
+ (expected.replace(b'"timestamp_s":0.5', b'"timestamp_s":-0.5', 1), "timestamp"),
+ (expected.replace(b'"label":"laptop"', '"label":"cafe\u0301"'.encode(), 1), "NFC"),
+ )
+ for document, message in invalid_documents:
+ path.write_bytes(document)
+ with pytest.raises(ValueError, match=message):
+ read_observations(path)
+
+ with pytest.raises(ValueError, match="duplicate observation identity"):
+ write_observations(path, (laptop, laptop))
+ conflicting_laptop = laptop.model_copy(update={"label": "computer"})
+ with pytest.raises(ValueError, match="conflicting observation identity"):
+ write_observations(path, (laptop, conflicting_laptop))
+
+
+def test_read_observations_rejects_duplicate_json_field_names(tmp_path: Path) -> None:
+ path = tmp_path / "observations.jsonl"
+ path.write_bytes(
+ b'{"box":{"x_max":0.3,"x_min":0.1,"y_max":0.5,"y_min":0.2},'
+ b'"confidence":0.9,"episode_id":"episode_1","frame_id":12,'
+ b'"label":"laptop","object_id":"mug_1","object_id":"laptop_1",'
+ b'"timestamp_s":0.5}\n'
+ )
+
+ with pytest.raises(ValueError, match="duplicate JSON field: object_id"):
+ read_observations(path)
+
+
+@pytest.mark.parametrize("mutation", ("whitespace", "crlf", "no-newline", "out-of-order"))
+def test_read_observations_rejects_noncanonical_jsonl(tmp_path: Path, mutation: str) -> None:
+ path = tmp_path / "observations.jsonl"
+ write_observations(
+ path,
+ (_observation("mug_1", "mug"), _observation("laptop_1", "laptop")),
+ )
+ canonical = path.read_bytes()
+ lines = canonical.splitlines(keepends=True)
+ mutations = {
+ "whitespace": canonical.replace(b'"confidence":0.9', b'"confidence": 0.9', 1),
+ "crlf": canonical.replace(b"\n", b"\r\n"),
+ "no-newline": canonical.rstrip(b"\n"),
+ "out-of-order": b"".join(reversed(lines)),
+ }
+ path.write_bytes(mutations[mutation])
+
+ with pytest.raises(ValueError, match="canonical"):
+ read_observations(path)
+
+
+@pytest.mark.parametrize("invalid_field", ("timestamp", "box", "label"))
+def test_write_observations_revalidates_model_instances(tmp_path: Path, invalid_field: str) -> None:
+ path = tmp_path / "observations.jsonl"
+ observation = _observation("laptop_1", "laptop")
+ invalid_observations = {
+ "timestamp": observation.model_copy(update={"timestamp_s": float("inf")}),
+ "box": observation.model_copy(
+ update={"box": observation.box.model_copy(update={"x_min": 0.4})}
+ ),
+ "label": observation.model_copy(update={"label": "cafe\u0301"}),
+ }
+
+ with pytest.raises(ValueError):
+ write_observations(path, (invalid_observations[invalid_field],))
diff --git a/dimos/benchmark/spatiotemporal/test_ports.py b/dimos/benchmark/spatiotemporal/test_ports.py
new file mode 100644
index 0000000000..16d6e9c971
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/test_ports.py
@@ -0,0 +1,147 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for frozen perception and candidate callable ports."""
+
+from collections.abc import Sequence
+from hashlib import sha256
+from pathlib import Path
+
+from pydantic import ValidationError
+import pytest
+
+from dimos.benchmark.spatiotemporal.models import (
+ BoundingBox2D,
+ BundleArtifact,
+ ObjectObservation,
+ OracleBundleManifest,
+ PublicBundleManifest,
+ Question,
+)
+from dimos.benchmark.spatiotemporal.ports import (
+ CandidateAnswerer,
+ CandidateReadiness,
+ DetectedObject,
+ ObservationBundleGenerator,
+ ReplayBundleResult,
+ replay_bundle_logical_sha256,
+)
+from dimos.benchmark.spatiotemporal.utilities import (
+ SCHEMA_VERSION,
+ canonical_model_json,
+ stable_id,
+)
+
+
+def test_detected_object_contains_only_perception_seam_data() -> None:
+ detected = DetectedObject(
+ object_id="track_7",
+ label="mug",
+ box=BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.3, y_max=0.4),
+ confidence=0.9,
+ )
+
+ assert set(detected.model_dump()) == {"object_id", "label", "box", "confidence"}
+
+
+def test_candidate_readiness_contains_no_teacher_or_oracle_data() -> None:
+ readiness = CandidateReadiness(ready=True, ingested_frame_count=24, detail=None)
+
+ dumped = readiness.model_dump()
+ assert dumped == {"ready": True, "ingested_frame_count": 24, "detail": None}
+ forbidden = {"observations", "boxes", "intervals", "answers", "evidence", "oracle"}
+ assert forbidden.isdisjoint(dumped)
+ with pytest.raises(ValidationError, match="at least one frame"):
+ CandidateReadiness(ready=True, ingested_frame_count=0, detail=None)
+
+
+def test_candidate_answerer_signature_is_expressible_without_private_records() -> None:
+ class FakeCandidate:
+ def ingest_video(self, video_path: Path) -> CandidateReadiness:
+ assert video_path == Path("episode.mp4")
+ return CandidateReadiness(ready=True, ingested_frame_count=1, detail=None)
+
+ def answer(self, question: Question) -> str | bool | None:
+ return None
+
+ def close(self) -> None:
+ return None
+
+ candidate = FakeCandidate()
+ typed_candidate: CandidateAnswerer = candidate
+ assert typed_candidate.ingest_video(Path("episode.mp4")).ready is True
+
+
+def test_observation_bundle_generator_has_root_independent_typed_result() -> None:
+ source_sha256 = "1" * 64
+ bundle_id = stable_id(
+ "bundle",
+ {
+ "episode_id": "episode_1",
+ "schema_version": SCHEMA_VERSION,
+ "source_video_sha256": source_sha256,
+ },
+ )
+ public = PublicBundleManifest(
+ schema_version=SCHEMA_VERSION,
+ bundle_id=bundle_id,
+ episode_id="episode_1",
+ source_video_sha256=source_sha256,
+ episode=BundleArtifact(path="public/episode.json", sha256="2" * 64, record_count=1),
+ questions=BundleArtifact(path="public/questions.jsonl", sha256="3" * 64, record_count=12),
+ )
+ public_sha256 = sha256(f"{canonical_model_json(public)}\n".encode()).hexdigest()
+ oracle = OracleBundleManifest(
+ schema_version=SCHEMA_VERSION,
+ bundle_id=bundle_id,
+ episode_id="episode_1",
+ source_video_sha256=source_sha256,
+ public_manifest_sha256=public_sha256,
+ observations=BundleArtifact(
+ path="oracle/observations.jsonl", sha256="5" * 64, record_count=30
+ ),
+ relation_facts=BundleArtifact(
+ path="oracle/relation_facts.jsonl", sha256="6" * 64, record_count=20
+ ),
+ relation_intervals=BundleArtifact(
+ path="oracle/relation_intervals.jsonl", sha256="7" * 64, record_count=8
+ ),
+ answers=BundleArtifact(path="oracle/answers.jsonl", sha256="8" * 64, record_count=12),
+ )
+ logical_sha256 = replay_bundle_logical_sha256(public, oracle)
+ result = ReplayBundleResult(
+ public_manifest=public,
+ oracle_manifest=oracle,
+ logical_sha256=logical_sha256,
+ )
+
+ class FakeGenerator:
+ def generate(
+ self,
+ observations: Sequence[ObjectObservation],
+ output_root: Path,
+ source_video_sha256: str,
+ ) -> ReplayBundleResult:
+ assert output_root == Path("release")
+ assert source_video_sha256 == source_sha256
+ return result
+
+ generator: ObservationBundleGenerator = FakeGenerator()
+ assert generator.generate((), Path("release"), source_sha256) == result
+ with pytest.raises(ValidationError, match="logical SHA-256"):
+ ReplayBundleResult(
+ public_manifest=public,
+ oracle_manifest=oracle,
+ logical_sha256="0" * 64,
+ )
diff --git a/dimos/benchmark/spatiotemporal/test_relations.py b/dimos/benchmark/spatiotemporal/test_relations.py
new file mode 100644
index 0000000000..f2be4a5b21
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/test_relations.py
@@ -0,0 +1,272 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for image-plane relation derivation."""
+
+from math import inf, nextafter
+
+import pytest
+
+from dimos.benchmark.spatiotemporal import relations
+from dimos.benchmark.spatiotemporal.models import BoundingBox2D, ObjectObservation, SpatialPredicate
+from dimos.benchmark.spatiotemporal.relations import derive_left_of
+
+
+def _observation(object_id: str, label: str, box: BoundingBox2D) -> ObjectObservation:
+ return ObjectObservation(
+ episode_id="episode_1",
+ frame_id=12,
+ timestamp_s=0.5,
+ object_id=object_id,
+ label=label,
+ box=box,
+ confidence=0.9,
+ )
+
+
+def _transform_x(
+ observation: ObjectObservation,
+ *,
+ offset: float = 0.0,
+ mirror: bool = False,
+) -> ObjectObservation:
+ box = observation.box
+ x_min, x_max = (1.0 - box.x_max, 1.0 - box.x_min) if mirror else (box.x_min, box.x_max)
+ return observation.model_copy(
+ update={"box": box.model_copy(update={"x_min": x_min + offset, "x_max": x_max + offset})}
+ )
+
+
+def test_accepts_left_of_relation_only_above_strict_margin() -> None:
+ mug = _observation(
+ "mug_1",
+ "mug",
+ BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.3, y_max=0.5),
+ )
+ laptop = _observation(
+ "laptop_1",
+ "laptop",
+ BoundingBox2D(x_min=0.5, y_min=0.2, x_max=0.8, y_max=0.6),
+ )
+
+ relation = derive_left_of(mug, laptop, margin=0.1)
+
+ assert relation is not None
+ assert relation.subject is mug
+ assert relation.predicate is SpatialPredicate.LEFT_OF
+ assert relation.object is laptop
+ assert relation.margin == 0.1
+ assert derive_left_of(mug, laptop, margin=0.2 - 1e-12) is not None
+ assert derive_left_of(mug, laptop, margin=0.2) is None
+ assert derive_left_of(mug, laptop, margin=0.2 + 1e-12) is None
+
+
+def test_equality_at_margin_remains_ambiguous_after_translation() -> None:
+ touching_margin = (
+ _observation(
+ "mug_1",
+ "mug",
+ BoundingBox2D(x_min=0.0, y_min=0.2, x_max=0.125, y_max=0.5),
+ ),
+ _observation(
+ "laptop_1",
+ "laptop",
+ BoundingBox2D(x_min=0.25, y_min=0.2, x_max=0.5, y_max=0.6),
+ ),
+ )
+ translated = (
+ _transform_x(touching_margin[0], offset=0.125),
+ _transform_x(touching_margin[1], offset=0.125),
+ )
+
+ assert derive_left_of(*touching_margin, margin=0.125) is None
+ assert derive_left_of(*translated, margin=0.125) is None
+
+
+def test_accepted_relation_remains_accepted_after_translation() -> None:
+ separated = (
+ _observation(
+ "mug_1",
+ "mug",
+ BoundingBox2D(x_min=0.0, y_min=0.2, x_max=0.125, y_max=0.5),
+ ),
+ _observation(
+ "laptop_1",
+ "laptop",
+ BoundingBox2D(x_min=0.375, y_min=0.2, x_max=0.5, y_max=0.6),
+ ),
+ )
+ translated = (
+ _transform_x(separated[0], offset=0.125),
+ _transform_x(separated[1], offset=0.125),
+ )
+
+ assert derive_left_of(*separated, margin=0.125) is not None
+ assert derive_left_of(*translated, margin=0.125) is not None
+
+
+def test_next_representable_separation_above_margin_is_accepted() -> None:
+ subject = _observation(
+ "subject_1",
+ "subject",
+ BoundingBox2D(x_min=0.0, y_min=0.2, x_max=0.0625, y_max=0.5),
+ )
+ object_ = _observation(
+ "object_1",
+ "object",
+ BoundingBox2D(
+ x_min=nextafter(0.1875, inf),
+ y_min=0.2,
+ x_max=0.5,
+ y_max=0.6,
+ ),
+ )
+
+ assert derive_left_of(subject, object_, margin=0.125) is not None
+
+
+@pytest.mark.parametrize(
+ ("subject_box", "object_box"),
+ [
+ (
+ BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.6, y_max=0.5),
+ BoundingBox2D(x_min=0.4, y_min=0.2, x_max=0.8, y_max=0.6),
+ ),
+ (
+ BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.9, y_max=0.7),
+ BoundingBox2D(x_min=0.3, y_min=0.3, x_max=0.6, y_max=0.5),
+ ),
+ ],
+)
+def test_overlap_and_containment_are_ambiguous(
+ subject_box: BoundingBox2D,
+ object_box: BoundingBox2D,
+) -> None:
+ subject = _observation("subject_1", "subject", subject_box)
+ object_ = _observation("object_1", "object", object_box)
+
+ assert relations.derive_left_of(subject, object_, margin=0.0) is None
+ assert relations.derive_right_of(subject, object_, margin=0.0) is None
+
+
+def test_horizontal_mirror_swaps_left_and_right() -> None:
+ mug = _observation(
+ "mug_1",
+ "mug",
+ BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.3, y_max=0.5),
+ )
+ laptop = _observation(
+ "laptop_1",
+ "laptop",
+ BoundingBox2D(x_min=0.6, y_min=0.2, x_max=0.8, y_max=0.6),
+ )
+
+ original = derive_left_of(mug, laptop, margin=0.1)
+ mirrored = relations.derive_right_of(
+ _transform_x(mug, mirror=True),
+ _transform_x(laptop, mirror=True),
+ margin=0.1,
+ )
+
+ assert original is not None
+ assert mirrored is not None
+ assert mirrored.subject.object_id == original.subject.object_id
+ assert mirrored.object.object_id == original.object.object_id
+
+
+def test_right_of_is_left_of_with_arguments_swapped() -> None:
+ mug = _observation(
+ "mug_1",
+ "mug",
+ BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.3, y_max=0.5),
+ )
+ laptop = _observation(
+ "laptop_1",
+ "laptop",
+ BoundingBox2D(x_min=0.5, y_min=0.2, x_max=0.8, y_max=0.6),
+ )
+
+ relation = relations.derive_right_of(laptop, mug, margin=0.1)
+
+ assert relation is not None
+ assert relation.subject is laptop
+ assert relation.predicate is SpatialPredicate.RIGHT_OF
+ assert relation.object is mug
+ assert relations.derive_right_of(mug, laptop, margin=0.1) is None
+
+
+def test_vertical_predicates_are_strict_inverses() -> None:
+ lamp = _observation(
+ "lamp_1",
+ "lamp",
+ BoundingBox2D(x_min=0.2, y_min=0.1, x_max=0.5, y_max=0.3),
+ )
+ table = _observation(
+ "table_1",
+ "table",
+ BoundingBox2D(x_min=0.1, y_min=0.6, x_max=0.9, y_max=0.8),
+ )
+
+ above = relations.derive_above(lamp, table, margin=0.2)
+ below = relations.derive_below(table, lamp, margin=0.2)
+
+ assert above is not None
+ assert above.predicate is SpatialPredicate.ABOVE
+ assert below is not None
+ assert below.predicate is SpatialPredicate.BELOW
+ assert relations.derive_above(table, lamp, margin=0.2) is None
+ assert relations.derive_below(lamp, table, margin=0.2) is None
+ assert relations.derive_above(lamp, table, margin=0.3) is None
+ assert relations.derive_below(table, lamp, margin=0.3) is None
+
+
+def test_rejects_invalid_margin_and_cross_sample_comparisons() -> None:
+ mug = _observation(
+ "mug_1",
+ "mug",
+ BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.3, y_max=0.5),
+ )
+ laptop = _observation(
+ "laptop_1",
+ "laptop",
+ BoundingBox2D(x_min=0.5, y_min=0.2, x_max=0.8, y_max=0.6),
+ )
+
+ for invalid_margin in (-0.1, 1.1, float("nan"), float("inf")):
+ with pytest.raises(ValueError, match="margin"):
+ derive_left_of(mug, laptop, margin=invalid_margin)
+
+ for update in (
+ {"episode_id": "episode_2"},
+ {"frame_id": 13},
+ {"timestamp_s": 0.6},
+ ):
+ other_sample = laptop.model_copy(update=update)
+ with pytest.raises(ValueError, match="same sample"):
+ derive_left_of(mug, other_sample, margin=0.1)
+
+
+def test_same_object_identity_never_produces_a_relation() -> None:
+ left = _observation(
+ "mug_1",
+ "mug",
+ BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.3, y_max=0.5),
+ )
+ right = _observation(
+ "mug_1",
+ "mug",
+ BoundingBox2D(x_min=0.5, y_min=0.2, x_max=0.8, y_max=0.6),
+ )
+
+ assert derive_left_of(left, right, margin=0.1) is None
diff --git a/dimos/benchmark/spatiotemporal/test_replay.py b/dimos/benchmark/spatiotemporal/test_replay.py
new file mode 100644
index 0000000000..becb0fa240
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/test_replay.py
@@ -0,0 +1,262 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Behavioral tests for deterministic observation replay."""
+
+from collections import Counter
+from pathlib import Path
+
+import pytest
+
+from dimos.benchmark.spatiotemporal import replay
+from dimos.benchmark.spatiotemporal.bundles import load_bundle
+from dimos.benchmark.spatiotemporal.models import (
+ BoundingBox2D,
+ ObjectObservation,
+ QuestionKind,
+)
+from dimos.benchmark.spatiotemporal.observation_io import read_observations, write_observations
+from dimos.benchmark.spatiotemporal.ports import (
+ ReplayInsufficiencyCode,
+ ReplayInsufficiencyError,
+)
+from dimos.benchmark.spatiotemporal.replay import (
+ DeterministicObservationBundleGenerator,
+ replay_observations,
+)
+
+
+def _observation(
+ *,
+ frame_id: int,
+ object_id: str,
+ box: BoundingBox2D,
+ episode_id: str = "episode_1",
+) -> ObjectObservation:
+ return ObjectObservation(
+ episode_id=episode_id,
+ frame_id=frame_id,
+ timestamp_s=frame_id / 10,
+ object_id=object_id,
+ label=object_id,
+ box=box,
+ confidence=0.9,
+ )
+
+
+def test_replays_saved_observations_to_root_independent_bundles(tmp_path: Path) -> None:
+ observations = (
+ _observation(
+ frame_id=1,
+ object_id="red",
+ box=BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.2, y_max=0.4),
+ ),
+ _observation(
+ frame_id=1,
+ object_id="blue",
+ box=BoundingBox2D(x_min=0.7, y_min=0.2, x_max=0.8, y_max=0.4),
+ ),
+ _observation(
+ frame_id=3,
+ object_id="green",
+ box=BoundingBox2D(x_min=0.2, y_min=0.1, x_max=0.4, y_max=0.2),
+ ),
+ _observation(
+ frame_id=3,
+ object_id="yellow",
+ box=BoundingBox2D(x_min=0.2, y_min=0.7, x_max=0.4, y_max=0.8),
+ ),
+ )
+ observations_path = tmp_path / "observations.jsonl"
+ write_observations(observations_path, observations)
+
+ results = tuple(
+ replay_observations(observations_path, root, "1" * 64)
+ for root in (tmp_path / "first", tmp_path / "second")
+ )
+
+ assert results[0].logical_sha256 == results[1].logical_sha256
+ for root, result in zip((tmp_path / "first", tmp_path / "second"), results, strict=True):
+ bundle = load_bundle(root)
+ assert bundle.public_manifest == result.public_manifest
+ assert bundle.oracle_manifest == result.oracle_manifest
+ assert bundle.observations == read_observations(observations_path)
+ assert bundle.relation_facts
+ assert bundle.relation_intervals
+ assert bundle.questions
+ assert len(bundle.answers) == len(bundle.questions)
+
+
+def test_replay_coalesces_relations_across_consecutive_sparse_samples(tmp_path: Path) -> None:
+ left = BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.2, y_max=0.4)
+ right = BoundingBox2D(x_min=0.7, y_min=0.2, x_max=0.8, y_max=0.4)
+ observations = tuple(
+ _observation(frame_id=frame_id, object_id=object_id, box=box)
+ for frame_id in (0, 150)
+ for object_id, box in (("left", left), ("right", right))
+ )
+
+ DeterministicObservationBundleGenerator(sample_schedule=((0, 0.0), (150, 15.0))).generate(
+ observations, tmp_path / "bundle", "1" * 64
+ )
+ intervals = load_bundle(tmp_path / "bundle").relation_intervals
+
+ assert len(intervals) == 1
+ assert all(interval.start_frame_id == 0 for interval in intervals)
+ assert all(interval.end_frame_id == 150 for interval in intervals)
+ assert all(interval.evidence_frame_ids == (0, 150) for interval in intervals)
+
+
+def test_replay_writes_balanced_spatial_oracles(tmp_path: Path) -> None:
+ observations = (
+ _observation(
+ frame_id=1,
+ object_id="left",
+ box=BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.2, y_max=0.4),
+ ),
+ _observation(
+ frame_id=1,
+ object_id="right",
+ box=BoundingBox2D(x_min=0.7, y_min=0.2, x_max=0.8, y_max=0.4),
+ ),
+ )
+
+ DeterministicObservationBundleGenerator().generate(observations, tmp_path / "bundle", "1" * 64)
+ bundle = load_bundle(tmp_path / "bundle")
+ spatial_ids = {
+ question.question_id
+ for question in bundle.questions
+ if question.question_kind is QuestionKind.SPATIAL
+ }
+
+ assert len(bundle.relation_facts) == 1
+ assert bundle.relation_facts[0].predicate.value == "left-of"
+ assert len(spatial_ids) == 2
+ assert Counter(
+ answer.expected for answer in bundle.answers if answer.question_id in spatial_ids
+ ) == {True: 1, False: 1}
+
+
+def test_replay_does_not_bridge_an_empty_intervening_sample(tmp_path: Path) -> None:
+ left = BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.2, y_max=0.4)
+ right = BoundingBox2D(x_min=0.7, y_min=0.2, x_max=0.8, y_max=0.4)
+ observations = tuple(
+ _observation(frame_id=frame_id, object_id=object_id, box=box)
+ for frame_id in (0, 300)
+ for object_id, box in (("left", left), ("right", right))
+ )
+
+ DeterministicObservationBundleGenerator(
+ sample_schedule=((0, 0.0), (150, 15.0), (300, 30.0))
+ ).generate(observations, tmp_path / "bundle", "1" * 64)
+ intervals = load_bundle(tmp_path / "bundle").relation_intervals
+
+ assert len(intervals) == 2
+ assert all(interval.start_frame_id == interval.end_frame_id for interval in intervals)
+
+
+def test_replay_observations_accepts_complete_sample_schedule(tmp_path: Path) -> None:
+ left = BoundingBox2D(x_min=0.1, y_min=0.2, x_max=0.2, y_max=0.4)
+ right = BoundingBox2D(x_min=0.7, y_min=0.2, x_max=0.8, y_max=0.4)
+ observations = tuple(
+ _observation(frame_id=frame_id, object_id=object_id, box=box)
+ for frame_id in (0, 150)
+ for object_id, box in (("left", left), ("right", right))
+ )
+ observations_path = tmp_path / "observations.jsonl"
+ write_observations(observations_path, observations)
+
+ replay_observations(
+ observations_path,
+ tmp_path / "bundle",
+ "1" * 64,
+ sample_schedule=((0, 0.0), (150, 15.0)),
+ )
+
+ assert len(load_bundle(tmp_path / "bundle").relation_intervals) == 1
+
+
+@pytest.mark.parametrize(
+ ("observations", "expected_code", "expected_guidance"),
+ [
+ ((), ReplayInsufficiencyCode.EMPTY_OBSERVATIONS, "capture at least one sampled frame"),
+ (
+ (
+ _observation(
+ frame_id=1,
+ object_id="red",
+ box=BoundingBox2D(x_min=0.1, y_min=0.1, x_max=0.2, y_max=0.2),
+ ),
+ _observation(
+ frame_id=2,
+ object_id="blue",
+ box=BoundingBox2D(x_min=0.7, y_min=0.7, x_max=0.8, y_max=0.8),
+ episode_id="episode_2",
+ ),
+ ),
+ ReplayInsufficiencyCode.MIXED_EPISODES,
+ "replay one episode at a time",
+ ),
+ (
+ (
+ _observation(
+ frame_id=1,
+ object_id="red",
+ box=BoundingBox2D(x_min=0.1, y_min=0.1, x_max=0.2, y_max=0.2),
+ ),
+ ),
+ ReplayInsufficiencyCode.NO_RELATIONS,
+ "spatially separated object pairs",
+ ),
+ ],
+)
+def test_reports_actionable_stable_insufficiency_reasons(
+ tmp_path: Path,
+ observations: tuple[ObjectObservation, ...],
+ expected_code: ReplayInsufficiencyCode,
+ expected_guidance: str,
+) -> None:
+ generator = DeterministicObservationBundleGenerator()
+
+ with pytest.raises(ReplayInsufficiencyError) as raised:
+ generator.generate(observations, tmp_path / "bundle", "1" * 64)
+
+ assert raised.value.code is expected_code
+ assert expected_guidance in str(raised.value)
+
+
+def test_reports_when_accepted_relations_produce_no_questions(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ observations = (
+ _observation(
+ frame_id=1,
+ object_id="red",
+ box=BoundingBox2D(x_min=0.1, y_min=0.1, x_max=0.2, y_max=0.2),
+ ),
+ _observation(
+ frame_id=1,
+ object_id="blue",
+ box=BoundingBox2D(x_min=0.7, y_min=0.1, x_max=0.8, y_max=0.2),
+ ),
+ )
+ monkeypatch.setattr(replay, "generate_spatial_question_cases", lambda facts, **kwargs: ())
+
+ with pytest.raises(ReplayInsufficiencyError) as raised:
+ DeterministicObservationBundleGenerator().generate(
+ observations, tmp_path / "bundle", "1" * 64
+ )
+
+ assert raised.value.code is ReplayInsufficiencyCode.NO_QUESTIONS
+ assert "evaluation questions" in str(raised.value)
diff --git a/dimos/benchmark/spatiotemporal/test_runner.py b/dimos/benchmark/spatiotemporal/test_runner.py
new file mode 100644
index 0000000000..15585cb3a8
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/test_runner.py
@@ -0,0 +1,129 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for candidate answer parsing and evidence-linked reporting."""
+
+import pytest
+
+from dimos.benchmark.spatiotemporal.models import (
+ OracleAnswer,
+ PredictionStatus,
+ Question,
+ QuestionKind,
+ SpatialPredicate,
+)
+from dimos.benchmark.spatiotemporal.runner import (
+ build_evaluation_report,
+ parse_candidate_prediction,
+)
+from dimos.benchmark.spatiotemporal.utilities import stable_question_id
+
+QUESTION_ID = stable_question_id(
+ episode_id="episode_1",
+ object_ids=("obj_red", "obj_blue"),
+ predicate=SpatialPredicate.LEFT_OF.value,
+ question_kind=QuestionKind.SPATIAL.value,
+)
+
+
+@pytest.mark.parametrize(
+ ("raw_answer", "expected", "expected_status", "expected_prediction"),
+ (
+ (True, True, PredictionStatus.CORRECT, True),
+ (False, True, PredictionStatus.INCORRECT, False),
+ ("yes", True, PredictionStatus.CORRECT, True),
+ ("no", False, PredictionStatus.CORRECT, False),
+ (None, True, PredictionStatus.MISSING, None),
+ ("true", True, PredictionStatus.INVALID, None),
+ ("yes, definitely", True, PredictionStatus.INVALID, None),
+ ),
+)
+def test_parses_only_boolean_and_explicit_yes_no_answers(
+ raw_answer: str | bool | None,
+ expected: bool,
+ expected_status: PredictionStatus,
+ expected_prediction: bool | None,
+) -> None:
+ status, prediction = parse_candidate_prediction(QUESTION_ID, raw_answer, expected)
+
+ assert status is expected_status
+ if expected_prediction is None:
+ assert prediction is None
+ else:
+ assert prediction is not None
+ assert prediction.question_id == QUESTION_ID
+ assert prediction.answer is expected_prediction
+
+
+def test_builds_evidence_linked_aggregate_report_with_filtered_diagnostics() -> None:
+ questions = tuple(
+ Question(
+ question_id=stable_question_id(
+ episode_id="episode_1",
+ object_ids=("obj_red", "obj_blue"),
+ predicate=predicate.value,
+ question_kind=QuestionKind.SPATIAL.value,
+ ),
+ episode_id="episode_1",
+ text=f"Is the red object {predicate.value} the blue object?",
+ question_kind=QuestionKind.SPATIAL,
+ predicate=predicate,
+ object_ids=("obj_red", "obj_blue"),
+ )
+ for predicate in (SpatialPredicate.LEFT_OF, SpatialPredicate.ABOVE)
+ )
+ oracles = {
+ questions[0].question_id: OracleAnswer(
+ question_id=questions[0].question_id,
+ expected=True,
+ evidence_frame_ids=(4,),
+ ),
+ questions[1].question_id: OracleAnswer(
+ question_id=questions[1].question_id,
+ expected=False,
+ evidence_frame_ids=(9,),
+ ),
+ }
+
+ report = build_evaluation_report(
+ questions,
+ oracles,
+ {questions[0].question_id: "yes"},
+ source_video_sha256="a" * 64,
+ diagnostic_statuses={PredictionStatus.MISSING},
+ )
+
+ assert report.source_video_sha256 == "a" * 64
+ assert report.overall.total == 2
+ assert report.overall.correct == 1
+ assert report.overall.accuracy == 0.5
+ assert report.status_counts == {
+ PredictionStatus.CORRECT: 1,
+ PredictionStatus.INCORRECT: 0,
+ PredictionStatus.MISSING: 1,
+ PredictionStatus.INVALID: 0,
+ }
+ assert report.by_family[QuestionKind.SPATIAL].total == 2
+ assert report.by_predicate[SpatialPredicate.LEFT_OF].correct == 1
+ assert report.by_predicate[SpatialPredicate.ABOVE].correct == 0
+ assert len(report.diagnostics) == 1
+ diagnostic = report.diagnostics[0]
+ assert diagnostic.question_id == questions[1].question_id
+ assert diagnostic.status is PredictionStatus.MISSING
+ assert diagnostic.question_kind is QuestionKind.SPATIAL
+ assert diagnostic.predicate is SpatialPredicate.ABOVE
+ assert diagnostic.expected is False
+ assert diagnostic.predicted is None
+ assert diagnostic.evidence_frame_ids == (9,)
+ assert diagnostic.evidence_interval_ids == ()
diff --git a/dimos/benchmark/spatiotemporal/test_scoring.py b/dimos/benchmark/spatiotemporal/test_scoring.py
new file mode 100644
index 0000000000..c47e886065
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/test_scoring.py
@@ -0,0 +1,109 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for exact typed prediction scoring."""
+
+import pytest
+
+from dimos.benchmark.spatiotemporal.models import (
+ OracleAnswer,
+ Prediction,
+ Question,
+ QuestionKind,
+ SpatialPredicate,
+)
+from dimos.benchmark.spatiotemporal.scoring import score_prediction
+from dimos.benchmark.spatiotemporal.utilities import stable_question_id
+
+QUESTION_ID = stable_question_id(
+ episode_id="episode_1",
+ object_ids=("obj_red", "obj_blue"),
+ predicate=SpatialPredicate.LEFT_OF.value,
+ question_kind=QuestionKind.SPATIAL.value,
+)
+
+
+def test_scores_exact_boolean_prediction_with_separate_oracle() -> None:
+ question = Question(
+ question_id=QUESTION_ID,
+ episode_id="episode_1",
+ text="Is the mug left of the laptop at the end?",
+ question_kind=QuestionKind.SPATIAL,
+ predicate=SpatialPredicate.LEFT_OF,
+ object_ids=("obj_red", "obj_blue"),
+ )
+ oracle = OracleAnswer(
+ question_id=question.question_id,
+ expected=True,
+ evidence_frame_ids=(12,),
+ )
+ prediction = Prediction(question_id=question.question_id, answer=True)
+
+ result = score_prediction(question, oracle, prediction)
+
+ assert result.question_id == question.question_id
+ assert result.expected is True
+ assert result.predicted is True
+ assert result.correct is True
+
+
+def test_scores_always_yes_incorrect_against_false_private_truth() -> None:
+ question = Question(
+ question_id=QUESTION_ID,
+ episode_id="episode_1",
+ text="Is the mug left of the laptop at the end?",
+ question_kind=QuestionKind.SPATIAL,
+ predicate=SpatialPredicate.LEFT_OF,
+ object_ids=("obj_red", "obj_blue"),
+ )
+ oracle = OracleAnswer(
+ question_id=question.question_id,
+ expected=False,
+ evidence_frame_ids=(12,),
+ )
+
+ result = score_prediction(
+ question,
+ oracle,
+ Prediction(question_id=question.question_id, answer=True),
+ )
+
+ assert result.correct is False
+
+
+def test_rejects_oracle_or_prediction_for_another_question() -> None:
+ question_id = QUESTION_ID
+ other_id = f"question_{'0' * 64}"
+ question = Question(
+ question_id=question_id,
+ episode_id="episode_1",
+ text="Is the mug left of the laptop at the end?",
+ question_kind=QuestionKind.SPATIAL,
+ predicate=SpatialPredicate.LEFT_OF,
+ object_ids=("obj_red", "obj_blue"),
+ )
+
+ mismatched_records = (
+ (
+ OracleAnswer(question_id=other_id, expected=True, evidence_frame_ids=(12,)),
+ Prediction(question_id=question_id, answer=True),
+ ),
+ (
+ OracleAnswer(question_id=question_id, expected=True, evidence_frame_ids=(12,)),
+ Prediction(question_id=other_id, answer=True),
+ ),
+ )
+ for oracle, prediction in mismatched_records:
+ with pytest.raises(ValueError, match="question IDs"):
+ score_prediction(question, oracle, prediction)
diff --git a/dimos/benchmark/spatiotemporal/test_temporal_memory_answerer.py b/dimos/benchmark/spatiotemporal/test_temporal_memory_answerer.py
new file mode 100644
index 0000000000..889373b026
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/test_temporal_memory_answerer.py
@@ -0,0 +1,73 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for the public-only TemporalMemory candidate adapter."""
+
+from pathlib import Path
+
+import pytest
+
+from dimos.benchmark.spatiotemporal.models import Question, QuestionKind, SpatialPredicate
+from dimos.benchmark.spatiotemporal.temporal_memory_answerer import TemporalMemoryAnswerer
+from dimos.benchmark.spatiotemporal.utilities import stable_question_id
+
+
+def test_temporal_memory_answerer_enforces_public_candidate_lifecycle(tmp_path: Path) -> None:
+ calls: list[tuple[str, object]] = []
+
+ def ingest(video_path: Path) -> int:
+ calls.append(("ingest", video_path))
+ return 12
+
+ def query(question_text: str) -> str:
+ calls.append(("query", question_text))
+ return "yes"
+
+ def cleanup() -> None:
+ calls.append(("cleanup", None))
+
+ answerer = TemporalMemoryAnswerer(ingest=ingest, query=query, cleanup=cleanup)
+ question = Question(
+ question_id=stable_question_id(
+ episode_id="episode_1",
+ object_ids=("obj_red", "obj_blue"),
+ predicate=SpatialPredicate.LEFT_OF.value,
+ question_kind=QuestionKind.SPATIAL.value,
+ ),
+ episode_id="episode_1",
+ text="Is the red object left of the blue object?",
+ question_kind=QuestionKind.SPATIAL,
+ predicate=SpatialPredicate.LEFT_OF,
+ object_ids=("obj_red", "obj_blue"),
+ )
+ video_path = tmp_path / "public-video.mp4"
+
+ with pytest.raises(RuntimeError, match="ingest"):
+ answerer.answer(question)
+
+ readiness = answerer.ingest_video(video_path)
+ assert readiness.ready is True
+ assert readiness.ingested_frame_count == 12
+ assert answerer.answer(question) == "yes"
+
+ answerer.close()
+ answerer.close()
+
+ assert calls == [
+ ("ingest", video_path),
+ ("query", question.text),
+ ("cleanup", None),
+ ]
+ with pytest.raises(RuntimeError, match="closed"):
+ answerer.ingest_video(video_path)
diff --git a/dimos/benchmark/spatiotemporal/test_utilities.py b/dimos/benchmark/spatiotemporal/test_utilities.py
new file mode 100644
index 0000000000..049b302833
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/test_utilities.py
@@ -0,0 +1,44 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for deterministic serialization helpers."""
+
+import json
+
+from dimos.benchmark.spatiotemporal.models import Question, QuestionKind, SpatialPredicate
+from dimos.benchmark.spatiotemporal.utilities import canonical_model_json, stable_question_id
+
+
+def test_serializes_strict_records_to_identical_canonical_json() -> None:
+ question = Question(
+ question_id=stable_question_id(
+ episode_id="episode_1",
+ object_ids=("obj_red", "obj_blue"),
+ predicate=SpatialPredicate.LEFT_OF.value,
+ question_kind=QuestionKind.SPATIAL.value,
+ ),
+ episode_id="episode_1",
+ text="Is the mug left of the laptop at the end?",
+ question_kind=QuestionKind.SPATIAL,
+ predicate=SpatialPredicate.LEFT_OF,
+ object_ids=("obj_red", "obj_blue"),
+ )
+
+ first = canonical_model_json(question)
+ second = canonical_model_json(question)
+
+ assert first == second
+ assert json.loads(first) == question.model_dump(mode="json")
+ assert ": " not in first
+ assert ", " not in first
diff --git a/dimos/benchmark/spatiotemporal/test_video_adapter.py b/dimos/benchmark/spatiotemporal/test_video_adapter.py
new file mode 100644
index 0000000000..58f7e116f1
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/test_video_adapter.py
@@ -0,0 +1,201 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from pathlib import Path
+
+import cv2
+import numpy as np
+import pytest
+
+from dimos.benchmark.spatiotemporal.models import BoundingBox2D
+from dimos.benchmark.spatiotemporal.ports import DetectedObject
+from dimos.benchmark.spatiotemporal.video_adapter import OpenCVVideoSampler
+from dimos.msgs.sensor_msgs.Image import Image, ImageFormat
+
+
+class _FakeCapture:
+ def __init__(
+ self,
+ frames: list[np.ndarray],
+ *,
+ declared_frame_count: int | None = None,
+ fps: float = 2.0,
+ opened: bool = True,
+ ) -> None:
+ self._frames = iter(frames)
+ self._declared_frame_count = (
+ len(frames) if declared_frame_count is None else declared_frame_count
+ )
+ self._fps = fps
+ self._opened = opened
+ self.released = False
+
+ def isOpened(self) -> bool:
+ return self._opened
+
+ def get(self, property_id: int) -> float:
+ if property_id == cv2.CAP_PROP_FPS:
+ return self._fps
+ if property_id == cv2.CAP_PROP_FRAME_COUNT:
+ return float(self._declared_frame_count)
+ raise AssertionError(f"unexpected capture property: {property_id}")
+
+ def read(self) -> tuple[bool, np.ndarray | None]:
+ frame = next(self._frames, None)
+ return frame is not None, frame
+
+ def release(self) -> None:
+ self.released = True
+
+
+class _FakeDetector:
+ def __init__(self) -> None:
+ self.images: list[Image] = []
+ self.closed = False
+
+ def detect(self, image: Image) -> tuple[DetectedObject, ...]:
+ self.images.append(image)
+ return (
+ DetectedObject(
+ object_id="mug-1",
+ label="mug",
+ box=BoundingBox2D(x_min=0.25, y_min=0.25, x_max=0.75, y_max=0.75),
+ confidence=0.9,
+ ),
+ )
+
+ def close(self) -> None:
+ self.closed = True
+
+
+class _EmptyDetector(_FakeDetector):
+ def detect(self, image: Image) -> tuple[DetectedObject, ...]:
+ self.images.append(image)
+ return ()
+
+
+def test_sampler_preserves_schedule_for_zero_detection_frames(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ capture = _FakeCapture([np.zeros((2, 3, 3), dtype=np.uint8) for _ in range(3)])
+ detector = _EmptyDetector()
+ monkeypatch.setattr(cv2, "VideoCapture", lambda _: capture)
+
+ sampler = OpenCVVideoSampler(detector=detector)
+
+ assert sampler.sample(Path("episode.mp4"), episode_id="episode-1") == ()
+ assert sampler.sample_schedule == ((0, 0.0), (1, 0.5), (2, 1.0))
+
+
+def test_sampler_deterministically_normalizes_sampled_frames_and_cleans_up(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ frames = [
+ np.full((2, 3, 3), (index, index + 1, index + 2), dtype=np.uint8) for index in range(4)
+ ]
+ capture = _FakeCapture(frames)
+ detector = _FakeDetector()
+ monkeypatch.setattr(cv2, "VideoCapture", lambda _: capture)
+
+ sampler = OpenCVVideoSampler(detector=detector, frame_stride=2)
+ observations = sampler.sample(Path("episode.mp4"), episode_id="episode-1")
+ sampler.close()
+
+ assert [(item.frame_id, item.timestamp_s) for item in observations] == [
+ (0, 0.0),
+ (2, 1.0),
+ ]
+ assert all(item.episode_id == "episode-1" for item in observations)
+ assert all(item.object_id == "mug-1" for item in observations)
+ assert [image.format for image in detector.images] == [ImageFormat.RGB, ImageFormat.RGB]
+ assert detector.images[0].frame_id == "0"
+ assert detector.images[0].ts == 0.0
+ np.testing.assert_array_equal(detector.images[0].data[0, 0], np.array([2, 1, 0]))
+ assert capture.released
+ assert detector.closed
+
+
+def test_sampler_rejects_non_positive_frame_stride() -> None:
+ with pytest.raises(ValueError, match="frame_stride must be positive"):
+ OpenCVVideoSampler(detector=_FakeDetector(), frame_stride=0)
+
+
+def test_sampler_reports_early_decode_failure_and_releases_capture(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ capture = _FakeCapture(
+ [np.zeros((2, 3, 3), dtype=np.uint8)],
+ declared_frame_count=2,
+ )
+ monkeypatch.setattr(cv2, "VideoCapture", lambda _: capture)
+
+ sampler = OpenCVVideoSampler(detector=_FakeDetector())
+ with pytest.raises(RuntimeError, match="failed to decode frame 1"):
+ sampler.sample(Path("truncated.mp4"), episode_id="episode-1")
+
+ assert capture.released
+
+
+def test_sampler_reports_unopened_video_and_releases_capture(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ capture = _FakeCapture([], opened=False)
+ monkeypatch.setattr(cv2, "VideoCapture", lambda _: capture)
+
+ sampler = OpenCVVideoSampler(detector=_FakeDetector())
+ with pytest.raises(RuntimeError, match="failed to open video: missing.mp4"):
+ sampler.sample(Path("missing.mp4"), episode_id="episode-1")
+
+ assert capture.released
+
+
+def test_sampler_reports_zero_frame_video_and_releases_capture(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ capture = _FakeCapture([])
+ monkeypatch.setattr(cv2, "VideoCapture", lambda _: capture)
+
+ sampler = OpenCVVideoSampler(detector=_FakeDetector())
+ with pytest.raises(RuntimeError, match="video contains no decodable frames"):
+ sampler.sample(Path("empty.mp4"), episode_id="episode-1")
+
+ assert capture.released
+
+
+@pytest.mark.parametrize("fps", [0.0, -1.0, float("nan")])
+def test_sampler_rejects_invalid_fps_and_releases_capture(
+ monkeypatch: pytest.MonkeyPatch,
+ fps: float,
+) -> None:
+ capture = _FakeCapture([np.zeros((2, 3, 3), dtype=np.uint8)], fps=fps)
+ monkeypatch.setattr(cv2, "VideoCapture", lambda _: capture)
+
+ sampler = OpenCVVideoSampler(detector=_FakeDetector())
+ with pytest.raises(ValueError, match="video FPS must be finite and positive"):
+ sampler.sample(Path("invalid-fps.mp4"), episode_id="episode-1")
+
+ assert capture.released
+
+
+def test_sampler_rejects_non_bgr_frame_dimensions_and_releases_capture(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ capture = _FakeCapture([np.zeros((2, 3), dtype=np.uint8)])
+ monkeypatch.setattr(cv2, "VideoCapture", lambda _: capture)
+
+ sampler = OpenCVVideoSampler(detector=_FakeDetector())
+ with pytest.raises(ValueError, match="decoded frame 0 must have BGR dimensions"):
+ sampler.sample(Path("grayscale.mp4"), episode_id="episode-1")
+
+ assert capture.released
diff --git a/dimos/benchmark/spatiotemporal/test_yoloe_adapter.py b/dimos/benchmark/spatiotemporal/test_yoloe_adapter.py
new file mode 100644
index 0000000000..c3d79644ff
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/test_yoloe_adapter.py
@@ -0,0 +1,219 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from dataclasses import dataclass
+
+import numpy as np
+import pytest
+
+from dimos.benchmark.spatiotemporal.yoloe_adapter import YoloeObservationDetector
+from dimos.msgs.sensor_msgs.Image import Image, ImageFormat
+
+
+@dataclass
+class _FakeDetection:
+ bbox: tuple[float, float, float, float]
+ track_id: int
+ confidence: float
+ name: str
+
+
+class _FakeYoloeDetector:
+ def __init__(self, frames: list[list[_FakeDetection]]) -> None:
+ self._frames = iter(frames)
+ self.images: list[Image] = []
+ self.cleanup_calls: list[str] = []
+ self.fail_stop = False
+ self.fail_prompts = False
+ self.prompts: list[str] | None = None
+
+ def set_prompts(self, text: list[str]) -> None:
+ self.prompts = text
+ if self.fail_prompts:
+ raise RuntimeError("prompt setup failed")
+
+ def process_image(self, image: Image) -> object:
+ self.images.append(image)
+ return type("Detections", (), {"detections": next(self._frames)})()
+
+ def stop(self) -> None:
+ self.cleanup_calls.append("stop")
+ if self.fail_stop:
+ raise RuntimeError("stop failed")
+
+ def close(self) -> None:
+ self.cleanup_calls.append("close")
+
+
+def _image() -> Image:
+ return Image.from_numpy(
+ np.zeros((100, 200, 3), dtype=np.uint8),
+ format=ImageFormat.RGB,
+ )
+
+
+def test_adapter_preserves_native_ids_and_uses_unique_prompt_fallback() -> None:
+ detector = _FakeYoloeDetector(
+ [
+ [
+ _FakeDetection((20.0, 10.0, 60.0, 50.0), 17, 0.9, "bottle"),
+ _FakeDetection((100.0, 20.0, 180.0, 80.0), -1, 0.8, "mug"),
+ ]
+ ]
+ )
+ adapter = YoloeObservationDetector(
+ prompts=("bottle", "mug"),
+ detector_factory=lambda: detector,
+ )
+
+ detections = adapter.detect(_image())
+
+ assert [(item.object_id, item.label) for item in detections] == [
+ ("17", "bottle"),
+ ("mug", "mug"),
+ ]
+ assert detections[0].box.model_dump() == {
+ "x_min": 0.1,
+ "y_min": 0.1,
+ "x_max": 0.3,
+ "y_max": 0.5,
+ }
+ assert detector.images
+
+
+def test_adapter_rejects_duplicate_fallback_labels() -> None:
+ detector = _FakeYoloeDetector(
+ [
+ [
+ _FakeDetection((20.0, 10.0, 60.0, 50.0), -1, 0.9, "mug"),
+ _FakeDetection((100.0, 20.0, 180.0, 80.0), -1, 0.8, "mug"),
+ ]
+ ]
+ )
+ adapter = YoloeObservationDetector(
+ prompts=("mug",),
+ detector_factory=lambda: detector,
+ )
+
+ with pytest.raises(ValueError, match="duplicate fallback label: mug"):
+ adapter.detect(_image())
+
+
+def test_adapter_rejects_fallback_when_tracked_detection_has_same_label() -> None:
+ detector = _FakeYoloeDetector(
+ [
+ [
+ _FakeDetection((20.0, 10.0, 60.0, 50.0), 17, 0.9, "mug"),
+ _FakeDetection((100.0, 20.0, 180.0, 80.0), -1, 0.8, "mug"),
+ ]
+ ]
+ )
+ adapter = YoloeObservationDetector(
+ prompts=("mug",),
+ detector_factory=lambda: detector,
+ )
+
+ with pytest.raises(ValueError, match="duplicate fallback label: mug"):
+ adapter.detect(_image())
+
+
+def test_adapter_rejects_fallback_for_non_prompt_label() -> None:
+ detector = _FakeYoloeDetector([[_FakeDetection((20.0, 10.0, 60.0, 50.0), -1, 0.9, "plate")]])
+ adapter = YoloeObservationDetector(
+ prompts=("mug",),
+ detector_factory=lambda: detector,
+ )
+
+ with pytest.raises(ValueError, match="fallback label is not a configured prompt: plate"):
+ adapter.detect(_image())
+
+
+def test_adapter_records_continuity_and_drop_statistics() -> None:
+ detector = _FakeYoloeDetector(
+ [
+ [
+ _FakeDetection((20.0, 10.0, 60.0, 50.0), 17, 0.9, "bottle"),
+ _FakeDetection((100.0, 20.0, 180.0, 80.0), -1, 0.8, "mug"),
+ ],
+ [
+ _FakeDetection((25.0, 10.0, 65.0, 50.0), 17, 0.9, "bottle"),
+ _FakeDetection((105.0, 20.0, 185.0, 80.0), 18, 0.8, "mug"),
+ ],
+ [_FakeDetection((110.0, 20.0, 190.0, 80.0), 18, 0.8, "mug")],
+ ]
+ )
+ adapter = YoloeObservationDetector(
+ prompts=("bottle", "mug"),
+ detector_factory=lambda: detector,
+ )
+
+ adapter.detect(_image())
+ adapter.detect(_image())
+ adapter.detect(_image())
+
+ assert adapter.statistics.frames_processed == 3
+ assert adapter.statistics.detections_emitted == 5
+ assert adapter.statistics.fallback_detections == 1
+ assert adapter.statistics.continued_tracks == 2
+ assert adapter.statistics.dropped_tracks == 2
+
+
+def test_adapter_closes_detector_even_when_stop_fails() -> None:
+ detector = _FakeYoloeDetector([[]])
+ detector.fail_stop = True
+ adapter = YoloeObservationDetector(
+ prompts=("mug",),
+ detector_factory=lambda: detector,
+ )
+ adapter.detect(_image())
+
+ with pytest.raises(RuntimeError, match="stop failed"):
+ adapter.close()
+
+ assert detector.cleanup_calls == ["stop", "close"]
+
+
+def test_close_before_detection_is_terminal_without_constructing_detector() -> None:
+ constructions = 0
+
+ def factory() -> _FakeYoloeDetector:
+ nonlocal constructions
+ constructions += 1
+ return _FakeYoloeDetector([[]])
+
+ adapter = YoloeObservationDetector(prompts=("mug",), detector_factory=factory)
+
+ adapter.close()
+
+ with pytest.raises(RuntimeError, match="detector is closed"):
+ adapter.detect(_image())
+ assert constructions == 0
+
+
+def test_failed_prompt_setup_cleans_up_and_retries_construction() -> None:
+ failed = _FakeYoloeDetector([[]])
+ failed.fail_prompts = True
+ recovered = _FakeYoloeDetector([[]])
+ candidates = iter((failed, recovered))
+ adapter = YoloeObservationDetector(
+ prompts=("mug",),
+ detector_factory=lambda: next(candidates),
+ )
+
+ with pytest.raises(RuntimeError, match="prompt setup failed"):
+ adapter.detect(_image())
+
+ assert failed.cleanup_calls == ["stop", "close"]
+ assert adapter.detect(_image()) == ()
+ assert recovered.prompts == ["mug"]
diff --git a/dimos/benchmark/spatiotemporal/utilities.py b/dimos/benchmark/spatiotemporal/utilities.py
new file mode 100644
index 0000000000..537276905a
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/utilities.py
@@ -0,0 +1,70 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Deterministic serialization and stable-ID helpers."""
+
+from collections.abc import Mapping, Sequence
+from hashlib import sha256
+import json
+from typing import Final, Literal, TypeAlias
+
+from pydantic import BaseModel
+
+JsonValue: TypeAlias = (
+ str | bool | None | int | float | Mapping[str, "JsonValue"] | Sequence["JsonValue"]
+)
+SchemaVersion: TypeAlias = Literal["spatiotemporal-video-qa/v2"]
+SCHEMA_VERSION: Final[SchemaVersion] = "spatiotemporal-video-qa/v2"
+
+
+def canonical_json_bytes(value: JsonValue) -> bytes:
+ """Serialize the benchmark's string-only ID preimages deterministically."""
+ return json.dumps(
+ value,
+ ensure_ascii=False,
+ separators=(",", ":"),
+ sort_keys=True,
+ ).encode("utf-8")
+
+
+def stable_id(prefix: str, preimage: JsonValue) -> str:
+ """Hash one canonical preimage into an opaque stable identifier."""
+ return f"{prefix}_{sha256(canonical_json_bytes(preimage)).hexdigest()}"
+
+
+def stable_question_id(
+ *,
+ episode_id: str,
+ question_kind: str,
+ predicate: str,
+ object_ids: Sequence[str] = (),
+ reference_ids: Sequence[str] = (),
+) -> str:
+ """Create an episode-scoped ID from answer-free public semantics."""
+ return stable_id(
+ "question",
+ {
+ "episode_id": episode_id,
+ "object_ids": tuple(object_ids),
+ "predicate": predicate,
+ "question_kind": question_kind,
+ "reference_ids": tuple(reference_ids),
+ "schema_version": SCHEMA_VERSION,
+ },
+ )
+
+
+def canonical_model_json(model: BaseModel) -> str:
+ """Serialize one strict record to deterministic canonical JSON."""
+ return canonical_json_bytes(model.model_dump(mode="json")).decode("utf-8")
diff --git a/dimos/benchmark/spatiotemporal/video_adapter.py b/dimos/benchmark/spatiotemporal/video_adapter.py
new file mode 100644
index 0000000000..218e2a2393
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/video_adapter.py
@@ -0,0 +1,97 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Deterministic OpenCV video sampling through the frozen detector seam."""
+
+import math
+from pathlib import Path
+
+import cv2
+
+from dimos.benchmark.spatiotemporal.models import ObjectObservation
+from dimos.benchmark.spatiotemporal.ports import ObservationDetector
+from dimos.msgs.sensor_msgs.Image import Image, ImageFormat
+
+
+class OpenCVVideoSampler:
+ """Sample decoded video frames and normalize detector results."""
+
+ def __init__(self, detector: ObservationDetector, frame_stride: int = 1) -> None:
+ if frame_stride < 1:
+ raise ValueError("frame_stride must be positive")
+ self._detector = detector
+ self._frame_stride = frame_stride
+ self._sample_schedule: tuple[tuple[int, float], ...] = ()
+ self._closed = False
+
+ def sample(self, video_path: Path, episode_id: str) -> tuple[ObjectObservation, ...]:
+ """Return detector observations for every selected source frame."""
+ capture = cv2.VideoCapture(str(video_path))
+ observations: list[ObjectObservation] = []
+ sample_schedule: list[tuple[int, float]] = []
+ try:
+ if not capture.isOpened():
+ raise RuntimeError(f"failed to open video: {video_path}")
+ fps = capture.get(cv2.CAP_PROP_FPS)
+ if not math.isfinite(fps) or fps <= 0.0:
+ raise ValueError("video FPS must be finite and positive")
+ declared_frame_count = int(capture.get(cv2.CAP_PROP_FRAME_COUNT))
+ frame_id = 0
+ while True:
+ decoded, frame = capture.read()
+ if not decoded:
+ if frame_id == 0:
+ raise RuntimeError("video contains no decodable frames")
+ if frame_id < declared_frame_count:
+ raise RuntimeError(f"failed to decode frame {frame_id}")
+ break
+ if frame is None or frame.ndim != 3 or frame.shape[2] != 3:
+ raise ValueError(f"decoded frame {frame_id} must have BGR dimensions")
+ if frame_id % self._frame_stride == 0:
+ timestamp_s = frame_id / fps
+ sample_schedule.append((frame_id, timestamp_s))
+ image = Image.from_numpy(
+ cv2.cvtColor(frame, cv2.COLOR_BGR2RGB),
+ format=ImageFormat.RGB,
+ frame_id=str(frame_id),
+ ts=timestamp_s,
+ )
+ observations.extend(
+ ObjectObservation(
+ episode_id=episode_id,
+ frame_id=frame_id,
+ timestamp_s=timestamp_s,
+ object_id=detection.object_id,
+ label=detection.label,
+ box=detection.box,
+ confidence=detection.confidence,
+ )
+ for detection in self._detector.detect(image)
+ )
+ frame_id += 1
+ finally:
+ capture.release()
+ self._sample_schedule = tuple(sample_schedule)
+ return tuple(observations)
+
+ @property
+ def sample_schedule(self) -> tuple[tuple[int, float], ...]:
+ """Return every selected frame, including frames with no detections."""
+ return self._sample_schedule
+
+ def close(self) -> None:
+ """Release the detector owned by this sampler."""
+ if not self._closed:
+ self._detector.close()
+ self._closed = True
diff --git a/dimos/benchmark/spatiotemporal/yoloe_adapter.py b/dimos/benchmark/spatiotemporal/yoloe_adapter.py
new file mode 100644
index 0000000000..bb018ef487
--- /dev/null
+++ b/dimos/benchmark/spatiotemporal/yoloe_adapter.py
@@ -0,0 +1,148 @@
+# Copyright 2025-2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Lazy YOLO-E adapter for the frozen observation-detector seam."""
+
+from collections import Counter
+from collections.abc import Callable, Sequence
+from dataclasses import dataclass
+from typing import Any
+
+from dimos.benchmark.spatiotemporal.models import BoundingBox2D
+from dimos.benchmark.spatiotemporal.ports import DetectedObject
+from dimos.msgs.sensor_msgs.Image import Image
+
+
+@dataclass(frozen=True)
+class YoloeAdapterStatistics:
+ """Accumulated identity continuity statistics."""
+
+ frames_processed: int
+ detections_emitted: int
+ fallback_detections: int
+ continued_tracks: int
+ dropped_tracks: int
+
+
+class YoloeObservationDetector:
+ """Normalize persisted YOLO-E detections while preserving object identity."""
+
+ def __init__(
+ self,
+ prompts: Sequence[str],
+ detector_factory: Callable[[], Any] | None = None,
+ ) -> None:
+ self._prompts = tuple(prompts)
+ self._detector_factory = detector_factory
+ self._detector: Any | None = None
+ self._closed = False
+ self._previous_object_ids: set[str] = set()
+ self._frames_processed = 0
+ self._detections_emitted = 0
+ self._fallback_detections = 0
+ self._continued_tracks = 0
+ self._dropped_tracks = 0
+
+ def detect(self, image: Image) -> tuple[DetectedObject, ...]:
+ """Run persisted tracking and normalize its detections."""
+ if self._closed:
+ raise RuntimeError("detector is closed")
+ detector = self._get_detector()
+ result = detector.process_image(image)
+ label_counts = Counter(detection.name for detection in result.detections)
+ fallback_labels: set[str] = set()
+ for detection in result.detections:
+ if detection.track_id == -1:
+ if detection.name not in self._prompts:
+ raise ValueError(f"fallback label is not a configured prompt: {detection.name}")
+ if label_counts[detection.name] > 1:
+ raise ValueError(f"duplicate fallback label: {detection.name}")
+ fallback_labels.add(detection.name)
+ normalized = tuple(self._normalize(detection, image) for detection in result.detections)
+ object_ids = {detection.object_id for detection in normalized}
+ self._frames_processed += 1
+ self._detections_emitted += len(normalized)
+ self._fallback_detections += len(fallback_labels)
+ self._continued_tracks += len(self._previous_object_ids & object_ids)
+ self._dropped_tracks += len(self._previous_object_ids - object_ids)
+ self._previous_object_ids = object_ids
+ return normalized
+
+ @property
+ def statistics(self) -> YoloeAdapterStatistics:
+ """Return an immutable snapshot of accumulated identity statistics."""
+ return YoloeAdapterStatistics(
+ frames_processed=self._frames_processed,
+ detections_emitted=self._detections_emitted,
+ fallback_detections=self._fallback_detections,
+ continued_tracks=self._continued_tracks,
+ dropped_tracks=self._dropped_tracks,
+ )
+
+ def close(self) -> None:
+ """Release an initialized detector without forcing lazy construction."""
+ if self._closed:
+ return
+ self._closed = True
+ if self._detector is None:
+ return
+ self._cleanup_detector(self._detector)
+
+ @staticmethod
+ def _cleanup_detector(detector: Any) -> None:
+ try:
+ stop = getattr(detector, "stop", None)
+ if stop is not None:
+ stop()
+ finally:
+ close = getattr(detector, "close", None)
+ if close is not None:
+ close()
+
+ def _get_detector(self) -> Any:
+ if self._detector is None:
+ if self._detector_factory is not None:
+ detector = self._detector_factory()
+ else:
+ from dimos.perception.detection.detectors.yoloe import (
+ Yoloe2DDetector,
+ YoloePromptMode,
+ )
+
+ detector = Yoloe2DDetector(prompt_mode=YoloePromptMode.PROMPT)
+ try:
+ detector.set_prompts(text=list(self._prompts))
+ except Exception:
+ try:
+ self._cleanup_detector(detector)
+ finally:
+ raise
+ self._detector = detector
+ return self._detector
+
+ @staticmethod
+ def _normalize(detection: Any, image: Image) -> DetectedObject:
+ x_min, y_min, x_max, y_max = detection.bbox
+ object_id = str(detection.track_id) if detection.track_id != -1 else detection.name
+ return DetectedObject(
+ object_id=object_id,
+ label=detection.name,
+ box=BoundingBox2D(
+ x_min=x_min / image.width,
+ y_min=y_min / image.height,
+ x_max=x_max / image.width,
+ y_max=y_max / image.height,
+ ),
+ confidence=detection.confidence,
+ )
diff --git a/docs/assets/spatiotemporal/.gitattributes b/docs/assets/spatiotemporal/.gitattributes
new file mode 100644
index 0000000000..1eb26f3eb1
--- /dev/null
+++ b/docs/assets/spatiotemporal/.gitattributes
@@ -0,0 +1,2 @@
+# Keep these small PR walkthrough images directly renderable without private LFS credentials.
+*.jpg -filter -diff -merge -text
diff --git a/docs/assets/spatiotemporal/evidence-viewer-overview.jpg b/docs/assets/spatiotemporal/evidence-viewer-overview.jpg
new file mode 100644
index 0000000000..d1bc031e17
Binary files /dev/null and b/docs/assets/spatiotemporal/evidence-viewer-overview.jpg differ
diff --git a/docs/assets/spatiotemporal/relationship-progression.jpg b/docs/assets/spatiotemporal/relationship-progression.jpg
new file mode 100644
index 0000000000..1ec53d9128
Binary files /dev/null and b/docs/assets/spatiotemporal/relationship-progression.jpg differ