fix(dashboard): isolate telemetry per training run - #83
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
<workspace>/run_<UTC timestamp>/instead of directly in the workspace root.python -m miles.dashboard.viewer --workspace <dir>resolves to the newestrun_*subdirectory; pass a specificrun_*directory to look at an older run.--miles-dashboard-workspacekeeps its meaning (it is the base directory) and is never rewritten behind the user's back.Why
DashboardStorepartitions each stream by UTC hour and appends (phases/20260728_22.jsonl, ...), andmeta.jsonis written once per collector. Relaunching training with the same--miles-dashboard-workspacetherefore appended the new run's phases, trajectories and GPU samples into the previous run's files and overwrotemeta.json, so the viewer drew two runs as a single timeline — the reported symptom.Two design points worth a look during review:
args.miles_dashboard_workspace(that would silently change a user-supplied path, and every worker would inherit the mutated value), the collector owns the run directory and exposes it throughDashboardCollector.workspace(). The rank-0GpuUtilSampler— the only worker that writes telemetry files itself — asks the collector for it; phase and trajectory events already flow through the collector, so nothing else needs the path.resolve_run_dirrelies on to pick "newest".For reference, miles core does not handle this inside the dashboard either: its telemetry directory is
{dump_details}/dashboardwith--dump-detailsasserted as required, and per-run uniqueness comes from the launcher (--dump-details {output_dir}/{run_id}/dump_details,run_id= UTC timestamp plus a random suffix). miles-diffusion has no such layer — no example or script passes--dump-details— and its viewer reads only the JSONL streams, so deriving the workspace from--dump-detailswould force heavyweight per-rollout.ptdumping for no benefit. This PR is the same idea one level down: one timestamped directory per launch, owned by the dashboard instead of the launcher.Known limits, deliberately not addressed here:
DashboardStore.write_metawithexist_ok=True. Guarding it means having the driver create the directory withexist_ok=Falseand fall back to a-Nsuffix.run_*directory exists the viewer resolves to that instead, so point--workspaceat the old directory itself to read it.Files
miles/dashboard/store.py—run_dir()(launch timestamp to directory) andresolve_run_dir()(workspace to newest run).miles/dashboard/backend.py— the driver points the collector config at this launch's run directory.miles/dashboard/collector.py—workspace()accessor so a worker can locate the run directory.miles/dashboard/hooks.py— rank-0 GPU sampler writes into the collector's run directory.miles/dashboard/viewer.py—--workspaceresolves to the newest run.miles/utils/arguments.py— help text for--miles-dashboard-workspace.tests/fast/dashboard/test_run_isolation.py— new: a relaunch does not merge metrics, newest-run resolution (including a workspace that holds no runs), andinit_dashboardleaving the workspace arg untouched.tests/fast/dashboard/test_async_collector.py— drop a stalemonkeypatch.setattr(backend, "_is_primary", ...); the attribute no longer exists, so that test was erroring onmainas well.Validation
pytest tests/fast/dashboard -q→ 11 passed (8 existing, 3 new).--workspace <base>rendered only the newest run's phase span,--workspace <base>/run_<older>rendered the older one.tests/fastsuite andtrain_diffusion.py --help— this checkout has noray/torch, so collection fails at import for both. Both need CI or a GPU box.Checklist
pre-commit run --all-filespytest -xis greenpython3 train.py --helpstill parses — not run (needsray); no flag was added or removed, only the help text of an existing one