hpc status: detail rendering only inspects first task of array / het jobs

[ultimatile]

Summary

hpc status <id> on an array job or a heterogeneous job reports the accounting fields of the first parent row only. The polling path fixed in #10 already aggregates across rows, so hpc submit --wait correctly waits until every task is terminal — but the new detail path added in #5 has no equivalent treatment.

Reproduction

For an array job whose tasks finish with mixed outcomes (e.g. one OOM, two COMPLETED), the user sees:

Job 12345_0: COMPLETED
  ExitCode: 0:0
  Elapsed:  00:01:23
  MaxRSS:   1024K
  ReqMem:   16Gn

…with no indication that another task was OUT_OF_MEMORY. Same shape for hetjob leaders: only the first component's accounting is rendered.

Proposal

Render either:

1. one block per task / het component (probably gated by an opt-in flag to avoid noise on large arrays), with the parent row identified by the canonical JobID (e.g. 12345_0, 12345_1 for array tasks; 12345+0, 12345+1 for het components), or
2. an aggregate summary line (e.g. 3 tasks: 2 COMPLETED, 1 OUT_OF_MEMORY) with per-task drill-down on demand.

The exact UX shape is part of this issue.

Out of scope

• Changes to the polling / wait_for_job path (already handled in fix(scheduler): aggregate Slurm sacct rows for array job wait #10).
• PJM array-equivalent rendering — see the companion PJM-detail issue.

[ultimatile]

Implementation plan — aggregate-aware hpc status detail for array / het jobs

Closes #16.

Slurm.parse_detail returns the accounting of the first parent row only, so hpc status on an array / het job hides every task after the first — a mixed-outcome array shows COMPLETED even when another task was OUT_OF_MEMORY. The polling path (#10) already aggregates; this brings the detail path (#5) to parity.

Decisions (settled)

• Default (multi-task): an aggregate summary line that always surfaces mixed outcomes (Job 12345: 3 tasks — 2 COMPLETED, 1 OUT_OF_MEMORY). Single-task jobs render exactly as today.
• Opt-in per-task detail: --detail {summary,tasks} (default summary), a symmetric enum option (mirrors --scheduler; extends cleanly to a future json etc.). tasks adds one accounting block per task / component.
• Scope: Slurm only. PJM array rendering stays out of scope — PJM has no detail_cmd and is tracked separately.

Checklist of changes

src/hpc/scheduler.py

• JobDetail: add a leading job_id: str field (the canonical sacct JobID, e.g. 12345_0 / 12345+0), so per-task rows can be labeled and grouped. Field order: job_id, state, exit_code, elapsed, max_rss, req_mem (matches the --format column order).
• Scheduler.parse_detail: change the contract to return list[JobDetail] (was JobDetail | None). Keep it a concrete (non-@abstractmethod) base method whose body returns []; update only the return-type annotation and docstring. (Adding @abstractmethod would make PJM() uninstantiable, contradicting "PJM inherits the base".)
• Slurm.parse_detail: return one JobDetail per parent row ("." not in JobID), in sacct order. For each parent P, aggregate max_rss over only its own substep rows (JobID.startswith(P + ".")), not globally — per-task RSS must not bleed across tasks. Skip parent rows with empty state; if none remain, return [].
• PJM.parse_detail: inherits the base [] (still never reached — detail_cmd is None).

src/hpc/job.py

• get_job_detail: return type becomes list[JobDetail] | None — None when the scheduler has no detail_cmd (PJM, unsupported), else the (possibly empty) list from parse_detail.

src/hpc/cli.py

• Add class DetailMode(str, Enum) with summary / tasks (mirrors SchedulerChoice).
• status: add detail: DetailMode = typer.Option(DetailMode.summary, "--detail", ...).
• Render logic:
  • not details (None unsupported, or [] no row) → existing single-line / "accounting not ready" fallback.
  • len == 1 → today's single block (Job {job_id}: {state} + 4 fields when terminal). Unchanged output.
  • len > 1 → aggregate line Job {job_id}: {n} {unit} — {k} STATE, ... (counts grouped by state, unit = tasks for _ ids / components for + ids). When detail == tasks, follow with one indented block per task keyed by JobDetail.job_id, fields shown when that task's state is terminal (reuse _normalize_sacct_state / _TERMINAL_SACCT_STATES).

Impact list

• parse_detail callers: only JobManager.get_job_detail (job.py:286). Return-type widening is compile-irrelevant in Python but semantically changes None->[] for the no-row case — handled at the single CLI consumer.
• get_job_detail callers: only cli.status (cli.py:408). Rewritten for the list.
• JobDetail constructions: Slurm.parse_detail (src) and ~6 test sites — all gain job_id.
• No change to the polling / wait_for_job path (out of scope, untouched).

Test plan

Invariants:

• A multi-task result never collapses to a single task's state; every parent row is represented.
• Per-task max_rss is drawn only from that task's substep rows (no cross-task bleed).
• Single-task output is byte-for-byte unchanged from today.

tests/test_scheduler.py:

• Slurm.parse_detail on a 2-task array (12345_0 COMPLETED + substeps, 12345_1 OUT_OF_MEMORY + substeps) -> list of 2 JobDetails with correct per-task job_id, state, and per-task max_rss.
• het input (12345+0, 12345+1) -> 2 components.
• single job (12345 + .batch) -> list of 1 (unchanged accounting).
• empty / header-only / no-parent-row -> [].
• existing single-job assertions updated for the list return + job_id field.

tests/test_job.py:

• get_job_detail Slurm -> list; PJM -> None (no detail_cmd, no SSH call — existing test updated).

tests/test_cli.py:

• single-task job -> unchanged block (regression pin).
• multi-task default (summary) -> aggregate line with the mixed-outcome breakdown; no per-task fields.
• multi-task --detail tasks -> per-task blocks labeled by canonical JobID, terminal fields per task.
• --detail summary (explicit) == default.
• PJM / no-row -> fallback line (unchanged).

Implementation guards

• Per-task max_rss grouping keyed by JobID.startswith(parent + ".") — asserted by the cross-task bleed test (distinct RSS per task).
• Single-task output regression pinned by a byte-for-byte CLI test, so the list refactor cannot silently change the common-case rendering.

Derivations

• sacct row taxonomy (Slurm spec): array task JobIDs are <jid>_<n>, het components <jid>+<n>; both lack .. Substeps are always <parent>.<step> (.batch, .extern, .0), always containing .. Therefore "." not in JobID selects exactly the per-task/component parent rows, and a task's substep rows are exactly those whose JobID is parent + "." + .... No dot-free substep and no dotted parent exist in sacct output, so the partition is total and disjoint.

Inconclusive / Deferred items

• deferred — runtime confirmation of sacct's array/het JobID column shape on a live Slurm cluster, including whether a pending array emits an extra dotless aggregate/range row (e.g. 12345_[2-9]) alongside the launched per-task rows. reason: no Slurm cluster is available this session (the reachable cluster is PJM/Fugaku). resolution-point: next Slurm cluster access. Non-blocking: the row taxonomy is standard, documented Slurm behavior already relied on by the existing .-based substep filter and the existing parse_detail tests; the change only extends that same convention from "first parent" to "all parents". Fail-safe for a dotless range row: it is .-free, so it renders as one additional (typically PENDING) entry in the aggregate — verbose, not wrong; it never hides a terminal task.