feat(benchmark): render dynamic robot relationship videos

[fengweit]

Depends on dimensionalOS#2989

This is a stacked follow-up to dimensionalOS#2989. Its base branch is feat/replayable-video-relation-evals at a4fb14069c67c0dd7b83ef17c927356bf9183e4b, so reviewers can inspect only the dynamic-video increment.

What this adds

A small, directly visible evaluation surface:

input video
  -> refresh YOLO-E detections once per second
  -> choose the highest-confidence configured robot
  -> derive robot-to-object left/right + above/below from 2D box centers
  -> optionally retain valid evidence through brief misses with an explicit age
  -> write and decode-verify a full-frame-rate MP4

The command is intentionally separate from benchmark question generation and scoring. It makes the evaluator's pseudo-label behavior inspectable as the video plays.

Evaluation idea

The reference run samples 46 one-second refresh opportunities across a 46-second source. Real YOLO-E inference produced robot-to-object relationships at 26 refreshes. The three moments below show the evaluation state changing over time.

0s — robot right/below refrigerator	15s — multiple detected object relations	30s — robot left/below refrigerator + chair

These are YOLO-E pseudo-labels. Box-center geometry is deterministic, but semantic labels can be missed or wrong; the output must not be treated as human-verified 3D ground truth.

Bounded-retention POC

The baseline has valid relationships at 26/46 refreshes (56.5%). A two-second hold fills 10 isolated misses and displays relationships at 36/46 refreshes (78.3%). Retained boxes never masquerade as fresh inference: the panel says STALE, shows the exact age, and returns to Robot not detected after the bounded window. No motion is interpolated.

Run it

After materializing the LFS video/model assets:

git lfs pull --include='assets/simple_demo.mp4,data/.lfs/models_yoloe.tar.gz'
uv sync --extra perception
uv run python -c "from dimos.utils.data import get_data; print(get_data('models_yoloe'))"
uv pip install 'git+https://github.com/ultralytics/CLIP.git@0fa238b2ba553fe76dc158348d8e34625e3e2470'
mkdir -p .artifacts/dynamic-robot-relations
uv run python -m dimos.benchmark.spatiotemporal.relationship_video \
  assets/simple_demo.mp4 \
  .artifacts/dynamic-robot-relations/robot-relationships.mp4 \
  --robot-label 'quadruped robot' \
  --update-period 1 \
  --hold-period 2
open .artifacts/dynamic-robot-relations/robot-relationships.mp4

Reference artifact verification:

• 1,380/1,380 frames decoded
• 1080x1920 at 29.997 FPS
• 46.0046 seconds
• full-rate overlay with one-second detector refresh and bounded two-second retention
• output written through a reserved temporary file and atomically replaced

Safety and failure behavior

• rejects source/output aliases and symlink outputs
• validates positive finite refresh periods and source FPS
• preserves an existing destination when rendering fails
• attempts capture, writer, and detector cleanup even when another cleanup raises
• verifies the encoded output decodes to the expected frame count
• keeps generated MP4s, model weights, inspection frames, and local links out of Git

Verification

139 passed
Ruff format: 35 files already formatted
Ruff check: passed
mypy: no issues in 18 source files
static added-line security scan: no findings

Independent read-only code review returned PASS with no blockers. Follow-up polish from that review is included: box labels now print the same object IDs used by relation text, detector cleanup covers capture-construction failure, new outputs use a shareable mode while replacements preserve an existing mode, and setup explicitly materializes the model directory. The visual audit passes artifact integrity and box-center geometry; semantic labels remain an intentional WARN because they are YOLO-E pseudo-labels.

Scope and limitations

• 2D image-plane box-center relations only; not world coordinates or distance
• detector refreshes once per second; no motion interpolation
• latest overlay is held between refreshes to preserve source FPS
• retained boxes are stale snapshots, not interpolated object motion
• missing robot/object detections remain explicit
• tracker IDs and semantic labels are teacher pseudo-label provenance, not physical-identity ground truth
• generated MP4 is intentionally untracked; committed screenshots provide reviewer evidence

[fengweit]

Superseded by the upstream draft PR dimensionalOS#3001.