Add job cancellation: CLI cannot stop a running job (unlike a directly-run app)

[Sootopolis]

Problem

There is no way to cancel a running job submitted via the CLI. Once ccas recruit (or any job subcommand) submits, the work runs to completion no matter what the operator does — Ctrl-C in the CLI only detaches the log follow; the server-side job fiber runs on.

This is asymmetric with running an app directly (e.g. RecruitmentApp in IntelliJ), where the app is the JVM's root fiber and the Stop button (SIGINT) interrupts it, running its .onInterrupt finalizers.

Why the CLI can't cancel today

The CLI is a separate process talking to the server over HTTP. ccas recruit POSTs /api/jobs/recruitment, gets a jobId, then merely follows the log stream. The real work runs as a fiber forkIn(layerScope) inside the server JVM (JobRunner.scala:184), owned by the server's layer scope — not the terminal. Ctrl-C interrupts only the CLI's follow fiber and sends nothing to the server (by design — the detach messaging says "job keeps running, reattach with ccas logs", JobFollower.scala:155-169).

The gap (concrete missing parts)

1. No JobRunId → Fiber handle. submit discards the result of forkIn(layerScope) (JobRunner.scala:147-184). The two Refs keyed by JobRunId hold a completion Promise and a progress BarChannel only (JobRunner.scala:74,77) — never the fiber. There is no address to interrupt.
2. No cancel method on the JobRunner trait (submit/status/recentJobs/logStream/progressStream).
3. No terminal state for cancellation. JobRunStatus = Running | Completed | Failed (JobRunStatus.scala:5-7). No Cancelled.
4. Interrupt is never observed as a terminal transition. runJob ends in foldZIO(onFailure, onSuccess) (JobRunner.scala:214), which catches only the typed Throwable channel. Interruption is a Cause, bypasses both branches, and the job_run row is left stuck at Running until the next server boot's JobRun.markOrphansAsFailed sweep (JobRunner.scala:87, JobRun.markOrphansAsFailed). The .ensuring(release) at JobRunner.scala:179 fires on interrupt but only closes the file sink / completion promise / channel Refs — it never touches the DB row.
5. No HTTP endpoint, no CLI subcommand.

Recruitment: the invite prompt is already handled, and cancel reuses it

Worth calling out because it removes work. When RecruitmentApp is interrupted, .onInterrupt runs finalizeRun(interrupted = true) (RecruitmentApp.scala:326-340). Note that this path skips the interactive invite prompt — line 382 short-circuits if (interrupted || found.isEmpty) … — and leaves the candidates found so far persisted as Deferred rows. The Y/n promptConfirmation only fires on normal completion, and it reads process stdin (StdIn.readLine, RecruitmentApp.scala:446), which cannot cross the CLI→server boundary anyway.

That is fine, because the Deferred candidates left by an interrupt feed the existing async delivery flow: GET /api/jobs/{id}/recruitment/found + POST .../recruitment/confirm. So cancel-a-recruit-job + existing confirm endpoint ≈ the graceful-stop-then-invite behaviour, decoupled from stdin — no new prompt machinery required. The recruit interrupt finalizer is also already stdin-free, so it is safe to trigger server-side.

Proposed approach

• (a) Retain fiber handle. Add fibers: Ref[Map[JobRunId, Fiber.Runtime[?, Unit]]] next to completions/jobChannels. Capture the forkIn(layerScope) result inside the existing uninterruptibleMask and register it; add fibers removal to the existing release block (JobRunner.scala:159-164) so the map self-cleans on normal exit.
• (b) cancel method. def cancel(id: JobRunId): ZIO[…, ?, Boolean]. Look up the fiber; present → fork fiber.interrupt (don't block the handler on the interrupt landing), then mark the row terminal. Absent → return false (already terminal, or owned by another server — see caveats).
• (c) New Cancelled state (preferred over reusing Failed — keeps operator-initiated stops distinct from real failures in recentJobs and the "Service restarted" orphan sweep). Add to the enum; persists via EnumSql. Companion fix: the terminal write must come from an interrupt-observing hook, not foldZIO. Switch runJob's tail to .onExit and, on Exit.Failure(cause) where cause.isInterrupted, write Cancelled + completed_at. Add JobRun.markCancelled mirroring markOrphansAsFailed.
• (d) HTTP. POST /api/jobs/{jobId}/cancel (POST, not DELETE — the row is retained; this is a state transition). Mirror the GET /api/jobs/{jobId} handler shape: 200 with a small CancelResult, 404 when unknown.
• (e) CLI. A ccas cancel <id> subcommand (new CliCommand case + Dispatcher branch). Optionally, offer cancel on Ctrl-C-during-follow via a JobFollower.onInterrupt prompt — but keep detach-as-default (an accidental Ctrl-C must not kill a long run); gate behind --cancel-on-interrupt or an explicit y/N.

Caveats

• "Request interrupt", not "instant kill". DB ops are ZIO.attemptBlocking (not …Interrupt) in PostgresClient, so an in-flight statement runs to completion; interrupt is honoured between statements (or when the Neon socketTimeout fires). HistoryApp.scala:353 has a bounded .uninterruptible finalize block (see HistoryApp Phase 4 finalization is not interrupt-safe — interrupt after the wave block strands a non-terminal history_run #137). UX copy should say "cancellation requested."
• Multi-server. The fibers map is per-process; fiber.interrupt only reaches this JVM's fiber. Fine under the documented single-server-per-DB model (JobRunner orphan-marking has no instance ownership — a 2nd server kills the 1st's live jobs #110 / Rate limiter is per-process: N instances = N× the Chess.com per-IP request rate #111). If the fiber is absent locally, return 404 / "not owned here" — do not blindly mark Cancelled, or a row still genuinely Running on another server would be flipped.

Scope

• Minimal (~half day): reuse Failed (no schema change), add fiber Ref + capture, cancel + onExit terminal write, POST .../cancel, ccas cancel <id>. ~4 files (JobRunner, JobRoutes, CliCommand/Dispatcher, JobRun).
• Full (~1–1.5 days): distinct Cancelled state + surfacing in JobStatusResponse, Ctrl-C-during-follow prompt, tests. Fiddly parts are only the fiber capture and the interrupt-vs-foldZIO terminal write; the rest is mechanical and follows existing route/CLI/table patterns. No new deps.

Related

• HistoryApp Phase 4 finalization is not interrupt-safe — interrupt after the wave block strands a non-terminal history_run #137 — HistoryApp Phase 4 interrupt-safety (same class of "interrupt strands a non-terminal run row" problem; the onExit terminal-write fix here is the general form).
• JobRunner orphan-marking has no instance ownership — a 2nd server kills the 1st's live jobs #110 / Rate limiter is per-process: N instances = N× the Chess.com per-IP request rate #111 — single-server-per-DB assumptions this inherits.

[Sootopolis]

Shipped in #169 (squash-merged as 0b61977). Adds cancel(id) on JobRunner (per-process fiber handles + cancelRequested gate distinguishing operator-cancel from shutdown-interrupt), the Cancelled state, POST /api/jobs/{jobId}/cancel, and the ccas cancel <job-id> subcommand; also unifies completed_at writers on Clock.instant. Main CI green.

The minimal + Cancelled-state slice landed. Deferred to a follow-up (as planned): Ctrl-C-during-ccas logs-follow → optional prompt-to-cancel.