Integrate sacct info for completed jobs into hpc status

[ultimatile]

Summary

For completed jobs, hpc status <id> should also show the following fields from sacct:

• State (COMPLETED / FAILED / OUT_OF_MEMORY / TIMEOUT / CANCELLED, etc.)
• ExitCode
• Elapsed
• MaxRSS
• ReqMem

Motivation

To diagnose why a job failed (OOM vs. timeout, etc.), users currently have to SSH in and run sacct by hand.
We are not going to build generalized "tuning suggestions" (the old hpc tune idea), but just surfacing the raw sacct numbers is already useful on its own.

Implementation notes

• Run sacct -j <jobid> --format=State,ExitCode,Elapsed,MaxRSS,ReqMem --noheader -P over SSH.
• Slurm-only feature. PJM should be handled separately (or skipped for now).
• If the job is still queued or running, fall back to the existing squeue-based display.
• Structured output (e.g. --json) can be revisited together with MCP integration; a human-readable table is fine for now.

Out of scope

• Custom log parsing for failure modes — Slurm already reports OUT_OF_MEMORY etc., which is enough.

Priority

Medium. Good cost/benefit: makes triage of failed jobs noticeably easier.

[ultimatile]

Implementation plan

Design constraints

• Do not touch the existing Scheduler.status_cmd / parse_status / JobManager.get_job_status / wait_for_job polling path.
  Widening the format would break the parser (lines[0].strip().rstrip("+") on "COMPLETED|0:0|..." falls through to JobStatus.FAILED and the wait loop terminates incorrectly).
• Do not extend the JobStatus enum.
  Display the raw sacct State string verbatim in the new detail path (so OUT_OF_MEMORY, CANCELLED+, etc. are shown literally) without affecting the polling enum.
• Add a separate detail path: new dataclass + new scheduler method pair + new JobManager method, consumed only by cli.status.

Files to change

1. src/hpc/scheduler.py

   • New module-level dataclass:

     @dataclass
     class JobDetail:
         state: str
         exit_code: str
         elapsed: str
         max_rss: str
         req_mem: str

   • Base Scheduler gains default no-op:

     def detail_cmd(self, job_id: str) -> list[str] | None: return None
     def parse_detail(self, output: str) -> JobDetail | None: return None

   • Slurm overrides:
     • detail_cmd: ["sacct", "-j", job_id, "--format=JobID,State,ExitCode,Elapsed,MaxRSS,ReqMem", "--noheader", "-P"].
       JobID is included so the parser can disambiguate the parent row from .batch / .extern / srun step rows.
     • parse_detail:
       1. splitlines() and drop empty lines.
       2. For each line, split("|"); if the last field is empty, drop one trailing empty (defensive against -p/--parsable output that adds a trailing |; -P/--parsable2 itself does not).
       3. Skip rows with fewer than 6 fields.
       4. Parent rows = those whose JobID does not contain .. Take the first parent row's state / exit_code / elapsed / req_mem.
       5. MaxRSS: prefer the .batch row's value (JobID.endswith(".batch")); otherwise scan all rows and pick the first non-empty value.
       6. If no parent row was found, or state is empty, return None so the caller can fall back.
   • PJM does not override; both methods return None (PJM is out of scope for detail rendering, falls back to single-line display).

2. src/hpc/job.py

   • Import JobDetail from .scheduler (annotations are evaluated eagerly under Python 3.11+ without from __future__ import annotations, so the symbol must be in scope).
   • New method:

     def get_job_detail(self, job_id: str) -> JobDetail | None:
         cmd = self.scheduler.detail_cmd(job_id)
         if cmd is None:
             return None
         result = self.ssh_manager.run_command(cmd[0], cmd[1:])
         return self.scheduler.parse_detail(result.stdout)

   • get_job_status and wait_for_job are unchanged.

3. src/hpc/cli.py (status command)

   • Try job_manager.get_job_detail(job_id) first.
   • If the result is None or detail.state is empty, fall back to the existing single-line output via get_job_status (covers PJM and the "sacct has not yet recorded the job" case for newly-submitted jobs).
   • Otherwise print Job {job_id}: {detail.state} (raw state).
   • If the normalized state token is in the terminal-state allowlist, also print indented ExitCode / Elapsed / MaxRSS / ReqMem lines, with empty fields shown as -.
     • Normalization: state.split()[0].rstrip("+") (handles CANCELLED+ and CANCELLED by 12345).
     • Allowlist: COMPLETED, FAILED, CANCELLED, TIMEOUT, OUT_OF_MEMORY, BOOT_FAIL, NODE_FAIL, PREEMPTED, DEADLINE, REVOKED, SPECIAL_EXIT.
   • Non-terminal states (PENDING / RUNNING / unrecognized) print only the single state line.

Test plan

• tests/test_scheduler.py:
  • Slurm.detail_cmd includes --format=JobID,State,ExitCode,Elapsed,MaxRSS,ReqMem, --noheader, and -P.
  • Slurm.parse_detail on a 3-row sample (parent + .batch + .extern) returns a JobDetail with parent-row fields and .batch-row MaxRSS.
  • Slurm.parse_detail on output with a trailing | on each row (defensive -p compatibility) still returns the same fields.
  • Slurm.parse_detail preserves OUT_OF_MEMORY and CANCELLED+ as raw strings.
  • Slurm.parse_detail on empty output returns None.
  • Slurm.parse_detail on output with no parent row returns None.
  • PJM.detail_cmd and PJM.parse_detail return None.
• tests/test_job.py:
  • JobManager.get_job_detail on a Slurm config invokes sacct over SSH and returns a JobDetail.
  • JobManager.get_job_detail on a PJM config returns None and does not invoke SSH.
• tests/test_cli.py:
  • status on a terminal Slurm job (mocked detail) prints all five fields.
  • status on a RUNNING Slurm job prints only the single state line.
  • status on a PJM job prints only the single state line.

Implementation guards

• parse_detail defensively rejects rows with fewer columns than expected — silent corruption from a malformed sacct output should produce None (graceful fallback) rather than incorrect fields.
• Terminal-state detection runs against the normalized token, not the raw string.
• JobDetail carries raw strings only; no unit conversion or enum mapping is done in this issue.

Derivation — sacct -P output shape

• Slurm sacct man page:
  • -P / --parsable2 → pipe-delimited, no trailing |.
  • -p / --parsable → pipe-delimited, with trailing |.
  • --format=A,B,C → columns appear in the requested order.
  • --noheader → suppresses the header line.
  • -j JOBID → returns the job and its accounting steps.
• For --format=JobID,State,ExitCode,Elapsed,MaxRSS,ReqMem -P --noheader, each non-empty line is exactly 6 pipe-separated fields. State values are enum-like strings (COMPLETED, CANCELLED+, CANCELLED by <uid>, OUT_OF_MEMORY, ...) and never contain |; MaxRSS / ReqMem are numeric+suffix and never contain |. Therefore positional parsing by column index is sound.
• Counterexample search: a State value containing | would falsify the layout. None of the documented Slurm State strings contain |, so the layout holds.

Inconclusive / Deferred items

• deferred — JSON / structured output (sacct --json).
  reason: issue itself defers this until the MCP integration cycle; a human-readable table is sufficient for now.
  resolution-point: when MCP integration begins.
• deferred — PJM detail rendering (e.g. pjstat -E, accounting log).
  reason: issue says "PJM should be handled separately (or skipped for now)".
  resolution-point: a separate PJM-specific issue.
• deferred — JobStatus enum extensions (OUT_OF_MEMORY, etc.).
  reason: this issue's scope is raw sacct string display; extending the enum would also affect the polling path.
  resolution-point: when a downstream consumer (e.g. wait_for_job distinguishing OOM vs timeout) requires it.
• deferred — heterogeneous jobs (hetjob) and array jobs.
  reason: the parent row may legitimately appear multiple times (one per het component or array task). This issue handles the typical single-job layout by taking the first parent row.
  resolution-point: a follow-up issue to model multi-component status output.
• note (not deferred) — the issue body's phrase "fall back to the existing squeue-based display" is slightly inaccurate: the existing implementation is also sacct-based (single --format=State). The behavioral intent (keep the current single-line output for non-terminal states) is preserved.