hpc job-output --follow for real-time log tailing

[ultimatile]

Summary

Add a --follow / -f option to hpc job-output <id> that streams the running job's output, equivalent to tail -F.

Motivation

• hpc wait blocks until completion but does not show output.
• hpc job-output can only be used after the fact.
• For long-running jobs, watching progress (training loss, progress indicators) from the local side currently requires manually running ssh myhpc tail -F /path/to/output.

Implementation notes

• A thin implementation that just streams ssh ... tail -F <output_path> is sufficient.
• The output path can be pulled from RunManager metadata or from the stdout/stderr paths assigned by the scheduler.
• While the job is pending, the target file does not yet exist (tail -F retries, so this is largely a non-issue).
• Confirm that SSH closes cleanly when the user exits with Ctrl-C.

Priority

Low. ssh ... tail -F is a working substitute, so this can wait until it becomes a frequent need.

[ultimatile]

Implementation plan (research-eg)

Checklist of changes

1. src/hpc/ssh.py — add SSHManager.run_streaming

• Signature: run_streaming(self, cmd: str, args: Optional[list[str]] = None) -> int
• Validate command name via _validate_command_name (mirroring run_command).
• Apply shlex.quote to every arg.
• Build SSH command via _build_ssh_command(quoted_string).
• Call subprocess.run(ssh_cmd) without capture_output so stdout/stderr stream directly to the terminal.
• Return result.returncode.

2. src/hpc/job.py — add JobManager.tail_job_output (status-aware)

• Signature: tail_job_output(self, run_id: str, job_id: str, error: bool = False) -> int

• Sketch:

  def tail_job_output(self, run_id, job_id, error=False) -> int:
      from .ssh import SSHError
      try:
          status = self.get_job_status(job_id)
      except SSHError:
          status = None  # fall through to tail (retries on missing file)
  
      terminal = {JobStatus.COMPLETED, JobStatus.FAILED,
                  JobStatus.CANCELLED, JobStatus.TIMEOUT}
      if status in terminal:
          # Job already finished; no more output is coming. Print existing file.
          # get_job_output re-raises SSHError when the file is genuinely missing.
          print(self.get_job_output(run_id, job_id, error=error), end="")
          return 0
  
      workdir = _resolve_home_path(self.ssh_manager, self.config.cluster.workdir)
      ext = "err" if error else "out"
      output_path = f"{workdir}/.hpc/runs/{run_id}/job-{job_id}.{ext}"
      return self.ssh_manager.run_streaming("tail", ["-F", output_path])

• Rationale: a naive tail -F would retry forever on a missing output file when the job has already terminated. The status-aware gate funnels terminal-state cases through the existing get_job_output path, which surfaces the same SSHError behaviour the non-follow command already has.

3. src/hpc/cli.py — add --follow to job_output

• New option: follow: bool = typer.Option(False, "--follow", "-f", help="Stream output as the job runs (tail -F equivalent)")
• When follow=True: call job_manager.tail_job_output(run.run_id, run.job_id, error=error); raise typer.Exit(returncode) if non-zero.
• When follow=False: behaviour unchanged (get_job_output → print).

Impact list

• JobManager.get_job_output: unchanged, existing callers unaffected.
• SSHManager.run_command / run_script: unchanged.
• cli.job_output: --error stays orthogonal to --follow (-f -e follows stderr).
• A --follow request issued while the job is PENDING waits silently for the output file to appear (tail -F retry semantics).

Test plan

tests/test_ssh.py — new TestSSHManagerRunStreaming class:

• test_run_streaming_does_not_capture_output: subprocess.run kwargs do not contain capture_output.
• test_run_streaming_returns_exit_code: returncode passes through.
• test_run_streaming_validates_command: malformed command name raises ValueError.
• test_run_streaming_quotes_args: shlex.quote applied (e.g., paths with spaces).
• test_run_streaming_includes_control_master: ControlMaster options inherited.
• test_run_streaming_builds_tail_command: tail, -F, and the path appear as the SSH trailing argument.

tests/test_job.py:

• test_tail_job_output_active_uses_tail_F: status RUNNING ⇒ run_streaming("tail", ["-F", ...]) is called.
• test_tail_job_output_terminal_falls_back_to_get_job_output: status COMPLETED ⇒ run_streaming not called; cat-based get_job_output output is printed (verified via capsys).
• test_tail_job_output_unknown_status_streams: get_job_status raises SSHError ⇒ falls through to tail -F.
• test_tail_job_output_error_flag: error=True ⇒ path uses the .err extension.
• test_tail_job_output_returns_exit_code: streaming path passes the exit code through.

tests/test_cli.py:

• test_job_output_follow_in_help: --follow / -f appears in --help.
• test_job_output_follow_passes_error_flag: mock JobManager.tail_job_output, invoke hpc job-output -f -e <id>, assert it is called with error=True.
• test_job_output_follow_propagates_exit_code: when tail_job_output returns non-zero, the CLI exits non-zero.

Implementation guards

• run_streaming MUST validate the command name and quote args (defence against shell injection).
• tail_job_output MUST NOT call tail -F for terminal-state jobs (gate via get_job_status first).
• Unknown status (SSHError from get_job_status) falls through to tail -F — tail's own retry is the safe-side default.
• No explicit Python signal handling for Ctrl-C — rely on canonical SSH propagation (foreground PG SIGINT → ssh client exit → sshd disconnect → remote tail SIGHUP).

Derivations

• tail -F retry on missing files — man tail defines -F as --follow=name --retry, and --retry is "keep trying to open a file even when it is or becomes inaccessible". A non-existent path is inaccessible by definition, so tail -F /not/yet/existing retries until the file is created. No counterexample in GNU coreutils.
• Default tail line count — GNU tail default is -n 10. For an empty / short / not-yet-existing file, only the available lines are emitted, then -F continues to follow as new lines appear. This matches the live-progress use case described in this issue.

Inconclusive / Deferred items

• inconclusive — Ctrl-C cleanup of remote tail process. Standard SSH semantics (foreground process group SIGINT → ssh client exits → sshd disconnect → remote tail receives SIGHUP) suggest this works canonically, but the claim cannot be verified inside the repository.
  • probe: after implementation, run hpc job-output -f <id> on a real HPC cluster, send Ctrl-C, then verify with pgrep tail on the remote host.
  • expected branches:
    • clean exit ⇒ ship as-is.
    • residual remote tail ⇒ either allocate a TTY with ssh -t for SIGINT propagation, or add an explicit Python SIGINT handler. Either route is a re-plan trigger.