recording based false-negative eval for the MLS planner - #3154
recording based false-negative eval for the MLS planner#3154GuraseesWasNotAvailable wants to merge 3 commits into
Conversation
Adds a self_hosted eval (issue dimensionalOS#2996) that replays a recorded lidar+odom dataset through the production pipeline (RayTraceMap -> MLSPlanner.update_region) and uses the robot's own traversed trajectory as ground truth to measure planner false negatives across a floor x floor matrix. Includes a union-find disconnect diagnostic that confirms each false negative is a genuine 'separate connected surface components' failure, and a coarse box-vs-voxel safety guardrail (reported, not gated). test_recording_eval.py wraps it as a self_hosted regression baseline. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses maintainer feedback (the ray-tracing voxel mapper is the highest-leverage module; 'planning is only as good as the map'): - Auto-detect floors from the trajectory height histogram (detect_floors) so the eval runs on any recording without hardcoded strata. - Make the ray-tracing MapperConfig and PlannerConfig first-class inputs, and add sweep() to A/B several mapper configs against planning accuracy -- the loop for measuring whether a mapper change reduces false negatives. Supports capped replays (max_frames) for fast iteration. - Add a false-negative-vs-separation breakdown to the scorecard; tune the safety-guardrail radius to the planner's wall_clearance. Baseline on athens_stairs: 112/132 false negatives (84.8%) across 3 detected floors, every one a confirmed graph disconnect. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Greptile SummaryAdds a recording-based MLS planner evaluation:
Confidence Score: 5/5The PR appears safe to merge. No blocking failures remain within the scope of the previous review threads. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart LR
DB[Recording database] --> ALIGN[Align lidar and odometry]
ALIGN --> MAP[RayTraceMap]
MAP --> MLS[MLSPlanner map and graph]
ALIGN --> TRAJ[Foot trajectory]
TRAJ --> FLOORS[Detect floors and sample pairs]
FLOORS --> PLAN[Plan each start/goal pair]
MLS --> PLAN
PLAN --> SCORE[False-negative and safety scorecard]
Reviews (2): Last reviewed commit: "fix(nav): address review — empty-replay ..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| The robot lingers on floors (tall histogram bins) and passes through stairs | ||
| quickly (sparse bins), so the well-populated modes are the floors. Returns a | ||
| sorted list of z levels (pose height).""" | ||
| z = feet[:, 2] + robot_height |
There was a problem hiding this comment.
When the required streams contain no lidar/odometry pairs within align_tol, build_map returns an empty one-dimensional feet array and this indexing raises IndexError, causing both the normal CLI and sweep to terminate instead of reporting that no aligned observations were available.
|
|
||
| def test_false_negatives_do_not_regress(scorecard): | ||
| """The headline metric: feasible pairs the planner fails to route.""" | ||
| assert scorecard.false_neg <= BASELINE_FALSE_NEG, ( |
There was a problem hiding this comment.
When the override or canonical recording yields only two detected floors, the evaluation tests at most 56 pairs but still compares against the 112-failure baseline measured over 132 pairs, so even a 100% false-negative rate passes this regression gate.
| assert n_disc <= n_fn, f"cell {i}->{j}: {n_disc} disconnects > {n_fn} false negatives" | ||
|
|
||
|
|
There was a problem hiding this comment.
n_disc <= n_fn is guaranteed because disconnects are counted only after the same iteration increments n_fn; similarly, unsafe >= 0 only checks an increment-only counter initialized to zero. These tests remain green when disconnect classification or safety checking is never meaningfully exercised, creating misleading regression coverage.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
…n gate Greptile review fixes: - build_map/detect_floors now raise a clear ValueError on an empty (no aligned lidar/odom) replay instead of an opaque IndexError. - Pin the regression denominator: assert total == EXPECTED_TOTAL (132) before the absolute false-negative baseline, so fewer detected floors can't pass with a worse rate. - Replace two vacuous assertions: require >=90% of false negatives to be confirmed disconnects, and assert paths were actually produced (unsafe <= produced). Add a unit test for the empty-trajectory guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TomCC7
added
the
first-time-contributor
2 participants
Contribution path
Problem
The MLS planner can return "no path" when a path actually exists. There was no automated way to (a) score the planner on a real recording or (b) tell whether a change to the mapper improves planning.
Solution
A recording-based false-negative eval. The robot traverses the whole recording in one continuous run so any points on its odometry trajectory are provably reachable. The eval replays a recording through the online pipeline, then asks
plan(A, B)for trajectory-sampled start/goal pairs andplan() -> Noneon a walkable pair is a false negative.Baseline on
mid360_athens_stairs: 112/132 false negatives across 3 auto-detected floors. false negatives rise with separation.Ships as
recording_eval.py+ aself_hostedregression test (test_recording_eval.py).How to Test
uv run python -m dimos.navigation.nav_3d.mls_planner.recording_eval data/mid360_athens_stairs.db
uv run python -m dimos.navigation.nav_3d.mls_planner.recording_eval data/mid360_athens_stairs.db --sweep
uv run pytest dimos/navigation/nav_3d/mls_planner/test_recording_eval.py -m self_hosted
AI assistance
Claude Code (Opus 4.8) was used for the following:
Checklist