Skip to content

feat: wire per-site closed-loop eval into the training loop (PR 6/6)#303

Open
kosuke55 wants to merge 2 commits into
pr5-standalone-multi-site-clifrom
pr6-training-loop-wiring
Open

feat: wire per-site closed-loop eval into the training loop (PR 6/6)#303
kosuke55 wants to merge 2 commits into
pr5-standalone-multi-site-clifrom
pr6-training-loop-wiring

Conversation

@kosuke55

@kosuke55 kosuke55 commented Jul 24, 2026

Copy link
Copy Markdown

Summary

Final part of the stacked closed-loop eval series (stacked on #302). Wires PR1-5 into the training loop:

closed_loop_validate() (train.py) now runs one FullRouteClosedLoopEvaluation (PR2) per (site, object-mode) pair:

  • --closed_loop_npz_root (single arbitrary path) and --closed_loop_sites_npz_root (auto-discovered multi-site root, PR3's site_discovery) fire independently — not if/else — so an ad-hoc override-scene test and a full per-site sweep can run in the same training call, each contributing its own rows to the combined episode table / cross-site aggregate (PR4's wandb_closed_loop).
  • Each source picks its own object modes via --closed_loop_npz_object_modes (default objects-only — usually a single curated scene, where the traffic-reaction ablation doesn't apply) / --closed_loop_sites_object_modes (default objects+noobj — the full per-site objects-vs-no-objects comparison).
  • --closed_loop_sites_npz_root may instead be a curated .json path-list manifest (PR3's discover_sites_from_json) instead of a sites-root directory, so training-time site closed-loop validation can use the same curated manifest as --closed_loop_npz_root's existing JSON-list support, instead of recursively evaluating every route under each site's split dir.

train_config.py/train_predictor.py/train_grpo_predictor.py gain the corresponding CLI/config fields (site roots, object modes, seg_len, the abort_*/unstick_* RolloutParams knobs, wandb video-pick/colormap-metrics/report-base-url). train_run.py forwards --closed_loop_sites_npz_root alongside the existing --closed_loop_npz_root.

Closes out the stacked series: rollout engine (#298) → evaluation framework (#299) → site discovery/colormap (#300) → W&B dashboard/HTML report (#301) → standalone CLI (#302) → this.

Test plan

  • ast.parse + module import check + TrainConfig instantiation check
  • Verified end-to-end on a real 8-GPU server training run (real dataset, real model): both sources firing together, both object modes, per-site + cross-site wandb aggregate, combined episode table, curated W&B dashboard
  • Fixed a real crash surfaced during that run (stale s.n_snaps field reference in PR1's early-abort debug logging, now in PR1/feat: closed-loop rollout engine — objects/no-objects ablation + graded metrics (PR 1/6) #298)
  • Verified --closed_loop_sites_npz_root as a .json manifest through the real, unmodified train_run.py CLI entrypoint (not just a direct function call): resumed a checkpoint, ran 10 real epochs, closed-loop fired at the checkpoint-save epoch against a real 2-site JSON manifest, all 4 (site, object-mode) combinations completed with correct metrics, videos and W&B log uploaded successfully

@kosuke55
kosuke55 force-pushed the pr5-standalone-multi-site-cli branch from 4a59adc to f12871b Compare July 24, 2026 12:15
@kosuke55
kosuke55 force-pushed the pr6-training-loop-wiring branch 2 times, most recently from c8644d9 to 656273b Compare July 24, 2026 12:46
@kosuke55
kosuke55 force-pushed the pr5-standalone-multi-site-cli branch from f12871b to 2419286 Compare July 24, 2026 13:33
kosuke55 added 2 commits July 24, 2026 22:33
closed_loop_validate() (train.py) now runs one PR2 FullRouteClosedLoopEvaluation
per (site, object-mode) pair: closed_loop_npz_root (single arbitrary path) and
closed_loop_sites_npz_root (auto-discovered multi-site root, PR3's
site_discovery) fire independently -- not if/else -- so an ad-hoc
override-scene test and a full per-site sweep can run in the same call, each
contributing its own rows to the combined episode table / cross-site
aggregate (PR4's wandb_closed_loop). Each source picks its own object modes
via --closed_loop_npz_object_modes (default objects-only) / --closed_loop_
sites_object_modes (default objects+noobj, for the comparison).

train_config.py/train_predictor.py/train_grpo_predictor.py gain the
corresponding CLI/config fields (site roots, object modes, seg_len, the
abort_*/unstick_* RolloutParams knobs, wandb video-pick/colormap-metrics/
report-base-url). train_run.py forwards --closed_loop_sites_npz_root
alongside the existing --closed_loop_npz_root.

Closes out the stacked series: rollout engine (#298) -> evaluation
framework (#299) -> site discovery/colormap (#300) -> W&B dashboard/HTML
report (#301) -> standalone CLI (#302) -> this.
When closed_loop_sites_npz_root ends in .json, sites are grouped via PR3's
discover_sites_from_json instead of discover_sites -- lets training-time
site closed-loop validation use the same curated path-list manifest
(path_list_closed_loop_x2.json) as --closed_loop_npz_root's existing
JSON-list support, instead of recursively evaluating every route under
each site's split dir.

Also stopped forcing npz_root through str() before passing it to
FullRouteClosedLoopEvaluation, since a JSON-manifest site's route pool may
now be a list of curated roots (PR2's multi-root support) rather than a
single path.

Verified end-to-end with a real 2-site JSON manifest via a direct
closed_loop_validate() call (bypassing full DDP training): both sites ran
with correct real metrics, matching a standalone run of the same data.
@kosuke55
kosuke55 force-pushed the pr6-training-loop-wiring branch from 656273b to dd9deed Compare July 24, 2026 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant