Skip to content

feat(serve): surface a coarse startup phase while a service comes up#107

Open
michaelroy-amd wants to merge 4 commits into
fix/dash-instance-statusfrom
feat/serve-startup-phase
Open

feat(serve): surface a coarse startup phase while a service comes up#107
michaelroy-amd wants to merge 4 commits into
fix/dash-instance-statusfrom
feat/serve-startup-phase

Conversation

@michaelroy-amd

Copy link
Copy Markdown
Member

Stacked on #106 (EAI-7350). Review/merge #106 first; this PR's base is fix/dash-instance-status, so its own diff is only the startup-phase work.

Summary

While a managed service is coming up, the dashboard showed only a generic STARTING label (or a spinner) with no indication of what it was doing — pulling weights, loading them onto the device, or warming up. A slow first launch was therefore indistinguishable from a hang. This PR surfaces a coarse startup phase (Downloading / Loading / Warmup) end to end.

Root Cause

The serve supervisor (apps/rocmd) already redirects the engine's stdout/stderr into the per-service log file, but wait_for_service_ready only polled a healthcheck RPC and never inspected that output. ManagedServiceRecord had no field to carry a startup stage, and the dashboard's InstanceStatus::Starting was a unit variant with nowhere to put one.

Changes

  • rocm-core: add ManagedServiceRecord.startup_phase: Option<String> (#[serde(default)] so older on-disk records still deserialize), cleared back to None once the service reaches ready.
  • rocmd: wait_for_service_ready now tails the service log while polling and reports each phase transition, which the caller persists to the record. Parsing is split into small pure, unit-tested helpers:
    • last_cr_segment collapses \r progress redraws (pip/tqdm/HF), the same rule the dashboard job console uses.
    • classify_startup_phase matches the common vLLM / llama.cpp / Hugging Face startup vocabulary into a downloading/loading/warmup token.
  • rocm-dash-core: add a fieldless StartupPhase enum and change InstanceStatus::Starting to Starting { phase: Option<StartupPhase> }. StartupPhase is Copy, so InstanceStatus stays Copy — the only ripple is pattern shape (Starting { .. }), not a Copy-removal cascade. A new InstanceStatus::label() centralizes the status text (READY / DOWNLOADING / LOADING / WARMUP / …).
  • rocm-dash-daemon (registry): map the record's startup_phase onto Starting { phase } for starting/recovering records; Docker discovery and the scrape-success promotion keep phase: None.
  • rocm-dash-tui: status_meta/status_role and the services-manager list now render via label(), so the phase shows on instance cards, the detail pane, and the services list. Note: serving.rs is a static verb menu with no per-instance status rendering, so the live phase display lives in instances.rs + services_manager.rs (the actual instance-status surfaces).

Scope / Notes

  • Phase detection is best-effort log scraping (no structured engine signal exists today); an unrecognized phase token degrades gracefully to a plain STARTING.
  • The Starting serde wire shape changes from the bare string "starting" to an externally-tagged object; this only affects the in-version daemon socket/replay stream, and the round trip is covered by a new test.

Test Plan

  • cargo build (workspace)
  • cargo clippy --all-targets --all-features -- -D warnings (workspace, clean)
  • cargo test -p rocm-core -p rocm-dash-core -p rocm-dash-collectors -p rocm-dash-daemon — green, incl. new tests: instance_status_label_surfaces_startup_phase, instance_status_starting_round_trips_through_serde, startup_phase_from_token_round_trips_labels, discovered_from_record_surfaces_startup_phase_while_starting
  • cargo test -p rocmd --lib — 109 green, incl. last_cr_segment_keeps_final_progress_redraw, classify_startup_phase_maps_engine_vocabulary, classify_startup_phase_emits_only_dashboard_known_tokens, read_new_log_phase_advances_and_tracks_latest
  • cargo test -p rocm-dash-tui -- --test-threads=1 — 543 + 16 + 5 green

Relates to EAI-7355

A service that is still coming up showed only a generic STARTING/spinner
with no indication of whether it was pulling weights, loading them, or
warming up — so a slow first-token launch was indistinguishable from a
hang. The serve supervisor already streams the engine's stdout/stderr to
the service log file but never inspected it, and the dashboard had no
field to carry a startup stage.

Add a coarse startup phase (Downloading/Loading/Warmup) end to end:

- rocm-core: add ManagedServiceRecord.startup_phase (Option<String>,
  serde-default so old on-disk records still load), cleared once the
  service reaches ready.
- rocmd: tail the service log during wait_for_service_ready and classify
  each line into a phase, persisting transitions to the record. Parsing
  is split into small pure helpers (last_cr_segment collapses \r progress
  redraws the same way the dashboard job console does; classify_startup_phase
  matches the vLLM/llama.cpp/HF startup vocabulary) so it is unit-testable.
- rocm-dash-core: add a StartupPhase enum and turn InstanceStatus::Starting
  into Starting { phase: Option<StartupPhase> }. StartupPhase is fieldless
  and Copy, so InstanceStatus stays Copy and the only ripple is pattern
  shape. A new InstanceStatus::label() owns the status text (READY /
  DOWNLOADING / LOADING / WARMUP / …).
- rocm-dash-daemon registry: read the record's startup_phase and map it
  onto Starting { phase } (recovering/starting only); Docker discovery and
  the scrape-success promotion keep phase None.
- rocm-dash-tui: status_meta/status_role and the services list render via
  label(), so the phase shows on instance cards, the detail pane, and the
  services manager. (serving.rs is a static verb menu with no per-instance
  status, so the live rendering lives in instances.rs + services_manager.rs.)

Stacked on #106 (EAI-7350).

Relates to EAI-7355

Signed-off-by: Michael Roy <michael.roy@amd.com>
…ing`

supervise_service (apps/rocmd/src/lib.rs) flips a managed-service record's
on-disk status to "running" immediately after spawning the engine child --
before it starts polling and classifying startup_phase from the serve log --
and only clears startup_phase once the service reaches "ready". So a record
can sit at status="running" with startup_phase=Some(...) for the entire
download/load/warmup window.

registry::instance_status_for_record only read startup_phase in the
"starting"/"recovering" arm, so that overlap fell straight through to a bare
InstanceStatus::Running and DOWNLOADING/LOADING/WARMUP never rendered during
a real `rocm serve <large-model>` cold start. The feature was inert
end-to-end despite the plumbing all being in place.

Chose the lower-risk fix: teach instance_status_for_record to also honor
startup_phase on the "running" arm, rather than delaying the "running"
write in supervise_service until the service is ready (rejected: that would
leave the record at status="starting" for the whole download, and
manifest_service_recovery_reason's 5-minute staleness check keys off
"starting"/"recovering" -- a slow cold start could then get misclassified as
a stale/stuck launch and trigger an unwanted recovery restart, a regression
supervise_service's current "running"-first ordering was implicitly
guarding against).

The existing registry test for this path constructed a status="starting" +
startup_phase record -- a shape the real producer never writes -- which
masked the bug. Replaced it with a case driven through the actual
load_service_records + discovered_from_record path using the real
status="running" + startup_phase shape, plus a starting/recovering
parametrized case and a running-with-no-phase steady-state case.

Also bump PROTOCOL_VERSION: InstanceStatus::Starting went from a unit
variant to a struct variant carrying `phase` in the prior commit on this
branch, which changes Event::InstanceDiscovered/Snapshot's external-tag JSON
shape for that variant; the version bump was missing.

Regenerated THIRD_PARTY_NOTICES.txt (pre-existing ordering drift, unrelated
to this change).

Relates to EAI-7355

Signed-off-by: Michael Roy <michael.roy@amd.com>
…o dep change)

Signed-off-by: Michael Roy <michael.roy@amd.com>
…g instances serving on unknown phase

- Add a cross-crate serde contract test: serialize rocm-core's real
  ManagedServiceRecord (with startup_phase set) and deserialize it as the
  daemon's ServiceRecord, so a future #[serde(rename)] on either side fails
  the test instead of silently dropping the phase (previously the field-name
  match rested only on a hand-written JSON literal). Adds rocm-core as a
  workspace-local dev-dependency (no third-party dep, TPN unaffected).
- Harden the running-status mapping: a 'running' record with no phase OR an
  unrecognized/foreign phase token now stays Running (serving); only the
  pinned tokens (downloading/loading/warmup) map to Starting{phase}. Prevents
  wrongly downgrading a genuinely-serving instance to non-serving STARTING.

Signed-off-by: Michael Roy <michael.roy@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant