submit: JOB_TEMPLATE emits Slurm-form --output= / --error= directives for PJM jobs

[ultimatile]

Summary

JOB_TEMPLATE in src/hpc/job.py injects scheduler-bookkeeping output and error directives in Slurm form (--output=..., --error=...) for both Slurm and PJM jobs. PJM (pjsub) uses -o <path> and -e <path> instead, so the emitted lines are not valid PJM directives.

Repro

hpc.toml with [cluster] scheduler = "pjm", then hpc submit "echo hi" and inspect the rendered job script (uploaded under <workdir>/.hpc/runs/<run_id>/job.sh):

#PJM -L node=12
...
#PJM --output=/.../job-%j.out
#PJM --error=/.../job-%j.err

The last two lines are Slurm-style; PJM's pjsub does not honor them. Job stdout / stderr therefore land at PJM's default locations rather than under <workdir>/.hpc/runs/<run_id>/job-<job_id>.out, which breaks hpc job-output <run_id> (it reads from the hpc-tracked path).

Expected

For PJM, the bookkeeping lines should be emitted as #PJM -o <path> and #PJM -e <path>, with the path / wildcard substitution adjusted to PJM conventions (%j is Slurm's; PJM uses %j in some configurations and the job-id substitution differs across PJM versions — needs investigation).

Notes

• This was identified during research for a separate issue but kept out of scope to keep that fix minimal.
• The fix likely needs a per-scheduler hook for emitting output / error directives, since the syntax and substitution conventions diverge.
• hpc job-output resolution (get_job_output in src/hpc/job.py) assumes the hpc-controlled path layout, so this needs to stay aligned with whatever PJM actually writes.

[ultimatile]

Plan

Design — per-scheduler directive/path pair

Introduce two abstract methods on Scheduler in src/hpc/scheduler.py so directive emission and path read-back stay in sync per scheduler:

@abstractmethod
def output_directives(self, run_dir: str) -> list[str]: ...

@abstractmethod
def output_path(
    self, run_dir: str, job_id: str, error: bool = False
) -> str: ...

Here run_dir is the per-run directory JobManager.submit_run already creates under <workdir>/.hpc/runs/<run_id>/ — the same directory hpc job-output <run_id> resolves to.

Slurm (preserves current behavior):

• output_directives(run_dir) returns ["#SBATCH --output={run_dir}/job-%j.out", "#SBATCH --error={run_dir}/job-%j.err"].
• output_path(run_dir, job_id, error) returns {run_dir}/job-{job_id}.{ext}.

PJM (fixed names):

• output_directives(run_dir) returns ["#PJM -o {run_dir}/job.out", "#PJM -e {run_dir}/job.err"].
• output_path(run_dir, job_id, error) returns {run_dir}/job.{ext} (the job_id arg is ignored).

hpc submit generates a fresh run_id per invocation in RunManager.create_run, so each PJM job has its own run_dir and a fixed job.out / job.err cannot collide across submissions. This sidesteps the question of how Fugaku pjsub handles a Slurm-style %j substitution in -o / -e, which is not part of the option's documented behavior.

Checklist

1. src/hpc/scheduler.py

   • Add output_directives / output_path abstract methods on Scheduler.
   • Concrete impls on Slurm and PJM per the table above.
   • Keep Scheduler.directive_prefix() — still consumed by JobManager._build_directives (Slurm option rendering) and by _extract_prologue_directives (user-script directive hoisting). Only the directive_prefix().split()[0] use inside JOB_TEMPLATE goes away.

2. src/hpc/job.py

   • Replace the two template lines that currently emit {{ scheduler.directive_prefix().split()[0] }} --output=... and the matching --error=... line with a Jinja loop:

     {% for d in output_directives %}
     {{ d }}
     {% endfor %}

   • In JobManager._render_job_script, compute run_dir = f"{workdir}/.hpc/runs/{run.run_id}" and pass output_directives=self.scheduler.output_directives(run_dir) to template.render(...).
   • In legacy JobManager.submit_job, same with run_id="job" (so run_dir = f"{workdir}/.hpc/runs/job"). Add one line to its docstring: "For PJM, fixed output filenames mean repeated calls overwrite previous output." (For Slurm, %j substitution already disambiguates by job id.) No other behavior change.
   • In JobManager.get_job_output and JobManager.tail_job_output, replace the hand-built f"{workdir}/.hpc/runs/{run_id}/job-{job_id}.{ext}" with self.scheduler.output_path(run_dir, job_id, error=error).

3. Tests — tests/test_scheduler.py

   • TestSlurmOutput: verify the output_directives strings and output_path for both error=False and error=True.
   • TestPJMOutput: same shape, asserting that output_path ignores job_id (different job_id returns the same path).
   • Abstract contract test (both schedulers): the path emitted by output_directives(run_dir) — with Slurm's %j substituted by job_id — equals output_path(run_dir, job_id, error).

4. Tests — tests/test_job.py

   • test_render_job_script_pjm_emits_pjm_output_directives: render with a PJM config, assert that #PJM -o .../job.out and #PJM -e .../job.err are present, and that --output= and --error= are absent (disconfirming check).
   • test_get_job_output_pjm_uses_fixed_filename: PJM get_job_output reads from .../job.out.
   • test_tail_job_output_pjm_uses_fixed_filename: PJM tail -F target is .../job.out.
   • Existing Slurm assertions in TestJobManagerTemplate and the directive-hoisting tests pass unchanged, because the Slurm directive string is byte-identical.

Impact

• Slurm directive strings are byte-identical; no Slurm behavior change.
• PJM is the actual fix — the directive form changes from the Slurm-shaped #PJM --output=... (which pjsub does not honor) to #PJM -o ..., and JobManager.get_job_output / tail_job_output resolve the new fixed path.
• CLI surface unchanged (hpc submit, hpc job-output).
• Config schema unchanged.

Derivations (compressed)

Self-consistency of output_directives / output_path. Each scheduler owns both methods, so the path written by the directive and the path read by output_path share one source of truth per scheduler.

• Slurm: directive --output={run_dir}/job-%j.out; Slurm substitutes %j with job_id; on-disk path is {run_dir}/job-{job_id}.out, which equals output_path(run_dir, job_id, error=False).
• PJM: directive -o {run_dir}/job.out (literal, no substitution); on-disk path is {run_dir}/job.out, which equals output_path(run_dir, _, error=False).

A discrepancy is possible only if a scheduler implementation chooses different conventions for the two methods — caught by the abstract contract test above.

Inconclusive / Deferred items

• deferred — whether current Fugaku pjsub accepts %j / %J substitution in -o / -e reason: this plan uses fixed names, so the answer does not change the design. resolution-point: separate issue if output naming is revised to include the job id.

• deferred — SSHError in SSHManager.run_command includes only stderr and drops stdout reason: independent concern. It made the symptom of this issue harder to diagnose (pjsub error text was lost when emitted on stdout), but it is not the directive-emission bug fixed here. resolution-point: separate issue.

• deferred — legacy JobManager.submit_job does not mkdir -p its remote .hpc/runs/job/ before submission reason: pre-existing behavior in a code path the CLI no longer invokes (CLI submission flows through submit_run). Out of scope for this fix. resolution-point: separate issue if the legacy path is retained.

[ultimatile]

Resolved in #20 (commit 4fe2465).

Summary of the change:

• Scheduler.output_directives(run_dir) + Scheduler.output_path(run_dir, job_id, error) introduced as a paired abstraction; Slurm keeps the --output=...job-%j.out form, PJM emits -o ...job.out / -e ...job.err with fixed filenames.
• JOB_TEMPLATE, _render_job_script, legacy submit_job, get_job_output, and tail_job_output all routed through the new pair.
• Side-find during review: legacy submit_job had a latent missing mkdir -p on its run_dir that this PR's own change activated — fixed in the same PR.
• Contract test added (TestSchedulerOutputDirectiveContract) enforces that the path emitted in directives equals the path JobManager reads back, per scheduler.

Note: the %j / %J portability question called out in this issue is sidestepped (fixed names per run, not substituted). A separate issue can revisit if a future feature needs job-id in the output path.