What do you want, and why?
the only behavior-level check on spatial memory today is e2e tests with hand-measured coordinates (test_dimsim_spatial_memory.py pins the bed at (-3.567, -1.332)). every new question is a manual measurement, and after a prompt or model change there's no repeatable way to tell whether object finding got better or worse.
i built a way to derive the questions and reference positions from a recording the robot already made, and score the goal the shipping navigate_with_text selects — execution stubbed:
- teacher (offline, once per recording): sample RGB on recorded timestamps, skip near-black frames, open-vocab detect (the in-repo YOLOe), project the nearest lidar sweep through
pose ∘ BASE_TO_OPTICAL + fisheye intrinsics, reject ground points, take the depth-IQR inliers' map-frame median as the candidate position. a label qualifies only with ≥2 pairwise-independent views (the robot actually moved between them) and exactly one qualifying cluster; qualified labels sitting on the same spot get linked so one physical object can't become several questions.
- names are the weak half of open-vocab on a dim replay, so a committed human-review overlay (verified / renamed / dropped, with reasons and the crops) gates what becomes a question.
go2_bigoffice yields 26 geometrically qualified labels on 19 locations; review keeps the single-digit subset that survives as askable questions.
- harness: fresh coordinator per question, shipping
SpatialMemory attached read-only, goal + query strings captured from the worker processes, six-state outcome so harness failures can't masquerade as agent failures, one figure per sweep (error distribution per model × system-prompt configuration).
status: implemented and green on a branch — one new directory (dimos/agents/evals/), zero existing-file changes, roughly half of it tests; the unit layer runs on the default lane, recording-backed layers are self_hosted. related: #3069 (this exercises the in-house agent path), #1913 (the point-placement half of it), #3249 covers the complementary authored-scene closed-loop side in dimsim, #3154 applies the same replay-derived-reference idea to the MLS planner. the success contract mirrors dimsim's {passed, reason, score}.
three questions before i open the PR:
- one PR, or split (offline reference pipeline vs harness+tests)?
- is
dimos/agents/evals/ the right home? followed the colocation precedent of nav_3d/evaluator and control/benchmarking
- sample results in the PR: a keyless pipeline-validation run (recorded model transcript) now with live sweep numbers as a follow-up comment, or hold for live numbers?
happy to adjust scope before opening. branch is ready to link if useful.
What do you want, and why?
the only behavior-level check on spatial memory today is e2e tests with hand-measured coordinates (
test_dimsim_spatial_memory.pypins the bed at(-3.567, -1.332)). every new question is a manual measurement, and after a prompt or model change there's no repeatable way to tell whether object finding got better or worse.i built a way to derive the questions and reference positions from a recording the robot already made, and score the goal the shipping
navigate_with_textselects — execution stubbed:pose ∘ BASE_TO_OPTICAL+ fisheye intrinsics, reject ground points, take the depth-IQR inliers' map-frame median as the candidate position. a label qualifies only with ≥2 pairwise-independent views (the robot actually moved between them) and exactly one qualifying cluster; qualified labels sitting on the same spot get linked so one physical object can't become several questions.go2_bigofficeyields 26 geometrically qualified labels on 19 locations; review keeps the single-digit subset that survives as askable questions.SpatialMemoryattached read-only, goal + query strings captured from the worker processes, six-state outcome so harness failures can't masquerade as agent failures, one figure per sweep (error distribution per model × system-prompt configuration).status: implemented and green on a branch — one new directory (
dimos/agents/evals/), zero existing-file changes, roughly half of it tests; the unit layer runs on the default lane, recording-backed layers areself_hosted. related: #3069 (this exercises the in-house agent path), #1913 (the point-placement half of it), #3249 covers the complementary authored-scene closed-loop side in dimsim, #3154 applies the same replay-derived-reference idea to the MLS planner. the success contract mirrors dimsim's{passed, reason, score}.three questions before i open the PR:
dimos/agents/evals/the right home? followed the colocation precedent ofnav_3d/evaluatorandcontrol/benchmarkinghappy to adjust scope before opening. branch is ready to link if useful.