Skip to content

fix(dashboard): isolate telemetry per training run - #83

Merged
Rockdu merged 1 commit into
radixark:mainfrom
Rockdu:fix/dashboard-run-isolation
Jul 30, 2026
Merged

fix(dashboard): isolate telemetry per training run#83
Rockdu merged 1 commit into
radixark:mainfrom
Rockdu:fix/dashboard-run-isolation

Conversation

@Rockdu

@Rockdu Rockdu commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

What

  • Dashboard telemetry now lands in a per-launch subdirectory <workspace>/run_<UTC timestamp>/ instead of directly in the workspace root.
  • python -m miles.dashboard.viewer --workspace <dir> resolves to the newest run_* subdirectory; pass a specific run_* directory to look at an older run.
  • No flag added or removed. --miles-dashboard-workspace keeps its meaning (it is the base directory) and is never rewritten behind the user's back.

Why

DashboardStore partitions each stream by UTC hour and appends (phases/20260728_22.jsonl, ...), and meta.json is written once per collector. Relaunching training with the same --miles-dashboard-workspace therefore appended the new run's phases, trajectories and GPU samples into the previous run's files and overwrote meta.json, so the viewer drew two runs as a single timeline — the reported symptom.

Two design points worth a look during review:

  • The run directory is derived on the driver only. Instead of rewriting args.miles_dashboard_workspace (that would silently change a user-supplied path, and every worker would inherit the mutated value), the collector owns the run directory and exposes it through DashboardCollector.workspace(). The rank-0 GpuUtilSampler — the only worker that writes telemetry files itself — asks the collector for it; phase and trajectory events already flow through the collector, so nothing else needs the path.
  • Run directory names are second-granular and sort lexicographically, which is what resolve_run_dir relies on to pick "newest".

For reference, miles core does not handle this inside the dashboard either: its telemetry directory is {dump_details}/dashboard with --dump-details asserted as required, and per-run uniqueness comes from the launcher (--dump-details {output_dir}/{run_id}/dump_details, run_id = UTC timestamp plus a random suffix). miles-diffusion has no such layer — no example or script passes --dump-details — and its viewer reads only the JSONL streams, so deriving the workspace from --dump-details would force heavyweight per-rollout .pt dumping for no benefit. This PR is the same idea one level down: one timestamped directory per launch, owned by the dashboard instead of the launcher.

Known limits, deliberately not addressed here:

  • Two launches whose start timestamps land in the same UTC second compute the same run directory and merge. A sequential relaunch cannot realistically hit this; two drivers started concurrently can. There is no arbitration today: the driver only derives the path, the directory itself is created lazily by DashboardStore.write_meta with exist_ok=True. Guarding it means having the driver create the directory with exist_ok=False and fall back to a -N suffix.
  • Telemetry written into the workspace root before this change stays there; once any run_* directory exists the viewer resolves to that instead, so point --workspace at the old directory itself to read it.

Files

  • miles/dashboard/store.pyrun_dir() (launch timestamp to directory) and resolve_run_dir() (workspace to newest run).
  • miles/dashboard/backend.py — the driver points the collector config at this launch's run directory.
  • miles/dashboard/collector.pyworkspace() accessor so a worker can locate the run directory.
  • miles/dashboard/hooks.py — rank-0 GPU sampler writes into the collector's run directory.
  • miles/dashboard/viewer.py--workspace resolves to the newest run.
  • miles/utils/arguments.py — help text for --miles-dashboard-workspace.
  • tests/fast/dashboard/test_run_isolation.py — new: a relaunch does not merge metrics, newest-run resolution (including a workspace that holds no runs), and init_dashboard leaving the workspace arg untouched.
  • tests/fast/dashboard/test_async_collector.py — drop a stale monkeypatch.setattr(backend, "_is_primary", ...); the attribute no longer exists, so that test was erroring on main as well.

Validation

  • pytest tests/fast/dashboard -q → 11 passed (8 existing, 3 new).
  • Manual viewer check: created two run directories under one workspace; --workspace <base> rendered only the newest run's phase span, --workspace <base>/run_<older> rendered the older one.
  • Not run locally: the full tests/fast suite and train_diffusion.py --help — this checkout has no ray/torch, so collection fails at import for both. Both need CI or a GPU box.

Checklist

  • pre-commit run --all-files
  • Added/updated tests for new behaviour
  • pytest -x is green
  • If launch flags changed, python3 train.py --help still parses — not run (needs ray); no flag was added or removed, only the help text of an existing one
  • If a public flag was added, it appears in the CLI reference docs — n/a, no new flag
  • If an example was added, it has a real walkthrough — n/a, no example

@Rockdu
Rockdu marked this pull request as ready for review July 30, 2026 05:15
@Rockdu
Rockdu merged commit e9e6f5e into radixark:main Jul 30, 2026
29 of 30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant