Feat/per site closed loop eval v2#297
Draft
kosuke55 wants to merge 3 commits into
Draft
Conversation
…4-main Re-implements the per-site closed-loop evaluation feature (auto-discovered multi-site sweep + empty-world objects/no-objects ablation, non-saturating metrics, per-site wandb dashboard, local HTML report) directly on top of tier4-main instead of the prior feat/per-site-closed-loop-eval branch, which had diverged too far to rebase cleanly. Intentionally excludes that branch's grouped-by-area evaluation framework (ClosedLoopEvaluation subclasses, scenario_classification, metrics/grouped_by_area) since it isn't used here; FullRouteClosedLoopEvaluation is called directly instead of through the grouped/full dispatcher. - closed_loop_validate() (train.py): closed_loop_npz_root and closed_loop_sites_npz_root fire independently (not if/else) so an ad-hoc override-scene test and a full per-site sweep can run in the same call; each source picks its own object modes via --closed_loop_npz_object_modes / --closed_loop_sites_object_modes. - reproducer_rollout.py: drop_objects ablation, route_completion/ n_collision_events/n_curb_hits graded metrics, early-abort on GT deviation — merged onto tier4-main's own independent evolution of this file (road-border distance scoring, two-stage unstick, realized-lag tracking). - New: site_discovery, trajectory_colormap, wandb_closed_loop(_workspace), closed_loop_html_report, closed_loop_evaluation, closed_loop_ddp, run_all_sites_closed_loop.py (standalone multi-site + multi-GPU runner). - valid_predictor_closed_loop.py: --out_dir, --drop_objects, and abort_* flags added on top of tier4-main's existing multi-GPU route-sharding CLI. - train_run.py: forwards --closed_loop_sites_npz_root alongside the existing --closed_loop_npz_root. Verified end-to-end with a local smoke run: both sources firing together, both object modes, per-site + cross-site wandb aggregate, combined episode table.
The "diverged" early-abort branch (abort_deviation_m/abort_max_snaps) still referenced the pre-rename s.n_snaps field in its debug-log dict literal and in the abort_max_snaps condition, left over from a 3-way merge that auto-applied this block cleanly (no conflict marker) without updating it to tier4-main's snap_count rename. abort_max_snaps=0 (the default) short-circuits around the condition, but the dict literal inside the "diverged" branch always evaluates s.n_snaps unconditionally, so any real early-abort trip (e.g. a trained model actually diverging) crashes the whole training job. Caught during a real server training run once the model actually diverged on one segment.
…p sections auto_generate_panels=False meant the curated dashboard replaced ALL of the default workspace's panels, not just the noisy one-tiny-panel-per-(site, metric) closed-loop ones -- train/valid loss, lr, epdms, and replan consistency panels disappeared too. Default to auto_generate_panels=True (--no_auto_panels to opt out) so the curated Training/Overview/Scores/Trajectory/Video/Episodes sections sit alongside everything the default workspace already showed, not instead of it.
kosuke55
force-pushed
the
feat/per-site-closed-loop-eval-v2
branch
from
July 24, 2026 13:30
64e9c15 to
a80b07b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://tier4.atlassian.net/browse/T4DEV-57220