Skip to content

wait/job-output: report scheduler job-id rejection distinctly instead of UNKNOWN / tail -F hang #52

Description

@ultimatile

#50 made the scheduler layer classify an affirmative job-id rejection as JobIdRejectedError (a SchedulerError subtype), and status now reports it distinctly: an unknown id gets the Run not found report, and a known run whose recorded job id is rejected prints Job <job_id>: the scheduler rejected this job id (run <run_id> metadata may be stale or corrupt) with exit 1.

The other commands that query the scheduler with a recorded job id still conflate the deterministic rejection with transient states:

  • wait and submit --wait: since status: run-id-shaped argument from the wrong project root raises SSHError instead of the Run not found report on Slurm #50, wait_for_job returns UNKNOWN immediately on the first poll for a rejection, so the wait loop terminates rather than hanging — but the CLI prints final state unknown (scheduler stopped returning data), a transient-sounding message that misdirects the user into retrying, and overwrites run.status with unknown.
  • job-output --follow: the pre-tail status check swallows the rejection via its generic except SSHError and falls through to tail -F on an output path that can never exist, hanging indefinitely.

These paths are reachable only with stale or corrupt run metadata (recorded job ids come from scheduler acceptance, so a live submission never produces them), so #50 was deliberately scoped to status, the one command that also accepts raw user-supplied ids; this issue extends the distinct report to the remaining commands.

Proposal

  1. Let wait / submit --wait distinguish a rejection: either surface JobIdRejectedError from wait_for_job to the CLI, or return a distinct terminal marker, and print the same rejected-id diagnosis status gives (exit non-zero, without clobbering run.status).
  2. Make job-output --follow's pre-tail status check treat JobIdRejectedError as fatal instead of falling through to tail -F.
  3. Reuse the recorded rejection fixtures in tests/rejection_fixtures.py for the new tests, mocking at the SSHManager layer.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions