fix(server): retry FileSink writes instead of latching off on first failure

[Sootopolis]

What

FileSink (per-job log sink) latched permanently on the first file-write failure, silently dropping every subsequent line for the rest of the job (the stdout tee kept working). One transient blip — brief I/O hiccup, dir momentarily missing — killed file logging for the whole job: a real data-loss window in the per-job log, now visible to operators because the file-tail logs endpoint (#47) is wired.

This replaces the permanent kill-switch with a suppress-and-retry gate + a drop counter (issue #52 options 1 + 3):

• On a write/open failure the sink enters a suppressed state, counts the dropped line, and reopens a fresh CREATE,APPEND writer — gated by retryAfterLines (50) or retryAfterNanos (5s), whichever first, so a permanent cause (disk-full) doesn't reopen per line and stderr sees one trace per failure episode.
• On recovery it writes a file logging resumed after N dropped line(s) marker; on close() a N log line(s) dropped summary. Both flow through the File-tail logs endpoint GET /api/jobs/{id}/logs #47 tail for free. droppedLineCount exposes the running total.
• Recovers transient open failures that later clear (the logDir reappears) and write/flush failures that later clear (disk freed, perms restored).
• Out of scope, documented: file deleted under an open fd (Linux keeps writing to the unlinked inode — no failure fires), FileTail offset desync after delete-recreate, disk-full mid-flush partial line.

Internals: the primary constructor is now private[jobs] with injected openWriter / nowNanos / thresholds — the deterministic test seam. make/close signatures are unchanged, so JobRunner integration is untouched, and a clean run produces byte-identical file output to before. All mutable state moved to vars guarded by the existing fileLock monitor (every access under the lock; init safely published via ZIO fiber handoff).

Fixes

Resolves #52.

Testing

• New ccas.server.jobs.TestFileSink (6 sleep-free tests, count-gate as the deterministic trigger): transient open-failure recovery, mid-job flush-failure recovery + stderr-once, counter accuracy/no-double-count, close-summary (file + stderr fallback + idempotent), no-reopen-after-close.
• Reworded 2 now-misleading tests in ccas.utils.TestJobLogSink (latch→closed, disabled→suppressed); assertions unchanged.
• ccas.server.jobs.TestJobRunner (FileSink integration) green; full sbt test green (960 passed).

Follow-up (not in this PR)

Surface droppedLineCount via a job_run column so the count shows in the jobs API, not only in-file — needs a JobRun case-class + sql/ migration change. Deferred.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ETHm6wCYdzFuB3kNEBayjF