feat: real-time observability for GPU transcription pipeline

[ojfbot]

Summary

• Add ProgressReporter to transcribe_episodes.py — writes structured progress.json at each of 8 pipeline stages with GPU metrics and segment heartbeats
• Add preflight() to aws_runner.py — validates model cache, CUDA/cuDNN, pyannote, GPU memory, and swap before transcription starts
• Replace 60s blind PID polling with 15s _poll_with_progress() that reads progress.json, logs stage/GPU info, and detects stalls with per-stage thresholds
• Add --strict-diarize flag to fail hard on diarization load failures (no silent fallback)
• Add --skip-preflight flag for known-good instances

Stacked on #38 (torch pin fix) — merge that first.

Motivation

During a transcription run on 2026-04-10, the whisper model download hung silently for 60+ minutes across 2 episodes. The old polling loop (kill -0 PID every 60s) had zero visibility into what was happening. A cuDNN mismatch also silently disabled diarization. Both issues were only discovered by manual SSH investigation.

What this changes

Remote side (transcribe_episodes.py)

Feature	Detail
ProgressReporter class	Writes progress.json atomically (tmp + rename) at each stage transition
8 stage names	model_loading → transcription → intro_outro_tagging → topic_tagging → diarization → speaker_embedding → chapter_generation → write_outputs
Segment heartbeats	Updates progress every 50 segments during transcription
GPU metrics	nvidia-smi utilization/VRAM via subprocess, cached 5s
--progress-file	Explicit path; defaults to {episode_dir}/progress.json for single-episode runs
--strict-diarize	Raises RuntimeError if pyannote/cuDNN fails instead of silent fallback

Local side (aws_runner.py)

Feature	Detail
preflight()	Checks model cache (>3GB, no .incomplete), CUDA tensor alloc, pyannote import, GPU memory (>4GB), swap (>8GB)
_poll_with_progress()	15s interval, reads progress.json, logs Stage: X (detail) | GPU: Y% ZMB | N/8 stages
Stall detection	Per-stage warn/kill thresholds (e.g. model_loading: 5m/10m, transcription: 10m/20m, diarization: 10m/25m)
Backward compat	Falls back to PID-alive polling if progress.json unavailable

Test plan

• Provision instance, verify preflight checks all pass (model cache ✓, CUDA ✓, pyannote ✓, GPU mem ✓, swap ✓)
• Run transcription and verify progress.json stage tracking appears in logs every 15s
• Verify segment heartbeat count advances during transcription stage
• Verify GPU metrics are reported (utilization %, VRAM MB)
• Test --strict-diarize catches diarization failures
• Test --skip-preflight bypasses checks
• Test backward compat: run against instance without progress.json support, verify fallback to PID polling

🤖 Generated with Claude Code