v0.4.8 — speaker diarization via pyannote.audio

Speaker diarization (pyannote.audio)

New spectator audio diarize subcommand and --diarize flag on audio transcribe for speaker diarization. Runs in the same audio-venv as Whisper.

After spectator audio install (with the default --with-diarize):

• spectator audio diarize call.mp3 --target <gpu-machine> — standalone; writes <stem>.diar.{rttm,json}.
• spectator audio transcribe call.mp3 --target <gpu-machine> --diarize — Whisper + pyannote + merge in one tmux session; produces <stem>.diarized.{json,txt} (Whisper segments with speaker labels via maximum-overlap voting against pyannote turns).

Hugging Face access is needed at run time only, not at install time. See the new README section and REFERENCE.md → audio diarize for the three-form HF setup, the multi-field-form-vs-checkbox gotcha, and the merge algorithm.

Notes for new-silicon hosts

On Blackwell GB10 (sm_121) with the cu128 PyTorch wheels, the audio diarize runtime probes for nvrtc support and surgically patches Tensor.abs to compute magnitude manually (sqrt(real² + imag²)) for complex CUDA tensors only. Without the patch, pyannote's wespeaker fbank trips nvrtc: error: invalid value for --gpu-architecture (-arch). The patch costs nothing on properly-supported hardware. The proper fix lands when PyTorch ships nvrtc with sm_121 in its allowlist (cu129+ or torch nightlies on cu13).

Commits

• be9f59d Add HUGGING_FACE_HUB_TOKEN to .creds and ensure_var_block helper
• 361908d Add speaker diarization via pyannote.audio (audio diarize)
• e3adc04 Document speaker diarization and bump version to 0.4.8

Full diff: v0.4.7...v0.4.8

v0.4.7 — \`--workdir ~/...\` footgun + audio status symlink follow

Closes Bug A and Bug E from the 2026-05-09 final bug report (workspace/Outbox/spectator-bug-report-final-2026-05-09.md). Other bugs from that report were already addressed: B in v0.3.6, C in v0.4.2, D mitigated in v0.4.1.

Bug A — `--workdir ~/...` unquoted gets local-tilde-expanded

Symptom on a Mac terminal driving a Linux remote:

$ ./spectator audio install --target spark --workdir ~/.co-sa/spectator
install failed
mkdir: cannot create directory '/Users': Permission denied

The local Bash expanded ~/.co-sa/spectator to /Users/misha/.co-sa/spectator before Spectator's typer parser saw the argument. Spectator stored that absolute Mac path in cfg.workdir, shipped it to the Linux remote, which failed because /Users/... doesn't exist there.

Fix: when --target is set AND cfg.workdir starts with the local user's home directory, rewrite the prefix back to $HOME so the REMOTE shell expands it. New helper config.normalize_workdir_for_remote(workdir, target). Plumbed into every verb that takes both --target and --workdir (preflight, install, uninstall, deploy, rsync, up, down, status, logs, audio install / transcribe / status / fetch, system cache-cleaner-start). The CLI emits a one-line heads-up when the rewrite kicks in, with a hint to single-quote the tilde next time:

heads-up: --workdir /Users/misha/.spectator looks like your LOCAL home
($HOME on this machine), but --target spark is set. Rewriting to
$HOME/.spectator so the remote shell expands it to the remote's $HOME.
Quote your tilde (--workdir '~/...' with single quotes) to avoid this
round-trip.

ui-server start is NOT rewritten — its --target is for job-routing, the workdir is local to the UI server, so cross-platform translation would be wrong there.

Bug E — `audio status` doesn't follow audio-out/ symlink

spectator audio status used bare find (no -L), so when $workdir/audio-out/ is a directory symlink, completed transcripts in subdirs of the symlink target weren't listed. One-character fix: find -L.

Tests

69/69 (was 67, +2). One pins the rewrite behavior for tilde-expanded local-home prefixes plus the no-rewrite cases (unrelated absolute paths, local-only invocations, already-tilde-quoted invocations). One renders audio.status's bash and asserts find -L is present.

Migration

None — both fixes are defensive additions. v0.3.6+ users have been single-quoting --workdir '~/...' as a workaround; that still works (the rewrite is a no-op when no rewrite is needed). v0.4.7 just makes Spectator forgiving when the user forgets to quote.

v0.4.6 — Fix tilde-quoting in preflight's .creds probe

Hot-fix on top of v0.4.5. The probe [ -f \"~/.spectator/.creds\" ] does not tilde-expand inside double quotes, so it always fell to the else branch even when the file existed. v0.4.5's preflight therefore continued to report keys as missing on hosts where .creds was correctly populated.

Fix: translate a leading ~ in workdir to $HOME before substituting into the bash test:

# Before (v0.4.5, broken):
[ -f "~/.spectator/.creds" ]
# After (v0.4.6):
[ -f "$HOME/.spectator/.creds" ]

Same translation pattern as the existing audio._expand_tilde / install._bash_workdir / stack._bash_workdir helpers. Will likely be centralized in _creds.py as a shared helper in a future release; for v0.4.6 just localized.

Tests: 67/67. The existing test_preflight_finds_creds_in_workdir was updated to assert the bash-expanded $HOME/... form in the captured probe (the only string assertion that would have caught this bug).

v0.4.5 — preflight aware of $workdir/.creds

Follow-up to v0.4.4. After the canonical credential store moved to $workdir/.creds, spectator preflight was still only checking the driving-shell env and the legacy ~/.ngc/api_key location. So a host with v0.4.4-populated .creds but no env vars in the running shell would preflight as if the keys were missing — even though every actual bash payload would have found them via the source-block.

preflight now ssh-probes $workdir/.creds first (set -a; . .creds; set +a; echo \"${NGC_CLI_API_KEY:+SET}\"). Three priority-ordered sources are now reflected in the rendered table:

Source	Reported as
.creds on target	in <workdir>/.creds on target (v0.4.4+; sourced by every bash payload)
Driving-shell env ($NGC_CLI_API_KEY / $NVIDIA_API_KEY)	set in driving shell (forwarded over SSH; first install will persist to .creds)
Legacy ~/.ngc/api_key (NGC only)	~/.ngc/api_key present on target

spectator preflight gained a --workdir / -w flag so the .creds probe targets the right path; spectator install already had --workdir and now threads it through to collect_checks too.

Tests: 67/67 (was 65, +2): one mocks _exec to return NGC=SET / NVIDIA=SET from the .creds probe and verifies both checks flip to ok with the right detail; one mocks empty .creds and verifies the env-var fallback kicks in for NGC and NVIDIA correctly fails.

Migration

None. Existing v0.4.4 .creds files are picked up immediately by the new preflight probe.

v0.4.4 — Persistent creds store at $workdir/.creds

Adds a credential-store layer at $workdir/.creds. Read priority (highest first):

1. $workdir/.creds — sourced by every bash payload that needs creds.
2. SSH env / process env — --ngc-key / --nvidia-key / shell exports.
3. None — caller errors.

Deliberately backwards from the usual CLI > env > config: .creds is the persistent source of truth that you set up once per target. CLI / env are useful for the FIRST install (they're what gets captured into .creds); after that the file is authoritative.

How it works

# First install captures keys from env / flags into $workdir/.creds:
$ export NGC_CLI_API_KEY=nvapi-... NVIDIA_API_KEY=nvapi-...
$ ./spectator install --target myspark1-local
...
==== wrote /home/misha/.spectator/.creds (creds persisted; future calls source it) ====

# Subsequent calls don't need keys in env — sourced from .creds:
$ unset NGC_CLI_API_KEY NVIDIA_API_KEY
$ ./spectator up --target myspark1-local   # works, sources from .creds

# Rotate keys: edit the file directly.
$ ssh myspark1-local 'vi ~/.spectator/.creds'

File format

Shell-source-able, one export VAR=VALUE per line, value quoted via printf %q. chmod 600 so other users on the host can't read it. Excluded from rsync (the file never travels off the host it was written on).

Spectator-managed vars (the "vars" tuple)

• NGC_CLI_API_KEY — always needed (docker pull from nvcr.io).
• NVIDIA_API_KEY — needed for remote LLM auth (default endpoint is build.nvidia.com).
• LLM_ENDPOINT_URL — only if self-hosting the LLM.

Future contributions can add to _creds.CREDS_VARS and the round-trip is automatic.

Tests

65/65 (was 60, +5): source-block shape pin, save-block creates file with 0600 perms + skips unset vars + is idempotent, source overrides env in a real bash subprocess, install renders both blocks, up renders the source block.

Migration

None. Existing installs upgrade in place — running ./spectator install --target HOST once after upgrading creates the .creds file from current env. After that, subsequent runs don't need the env vars set.

v0.4.3 — Wrapper stamp validity by pyproject content hash, not mtime

Fixes the wrapper-UX wart that surfaced during today's testing on myspark1-local.

What was broken

The v0.3.8 wrapper fast-path skipped require_uv when .venv looked healthy, but the staleness check was an mtime comparison:

[ ! "pyproject.toml" -nt "$INSTALL_STAMP" ]

rsync bumps the receiving file's mtime on every push, even when the content is byte-identical. So every

laptop$ ./spectator rsync --target HOST
laptop$ ssh HOST 'cd ~/.spectator/Spectator && ./spectator <verb>'

re-entered bootstrap on the remote → fired require_uv → died with "uv is required" on hosts where uv had been deleted post-deploy or where ~/.local/bin wasn't on the non-interactive SSH PATH. Even though the deps were unchanged and .venv was perfectly fine.

Fix

Stamp file now stores the SHA256 of pyproject.toml's content. The wrapper's bootstrap() compares that hash against the current pyproject.toml's hash:

State	Action
Hash matches	Fast path (no uv needed)
Hash differs	Sync (uv required — deps actually changed)
Stamp empty (legacy v0.3.9-v0.4.2 touch'd stamp)	Fast path + write the hash for next time (self-healing migration)

Hashing tries shasum -a 256 first (in Perl's macOS default distribution), then sha256sum (GNU coreutils on Linux), then falls back to touch (the legacy-empty-stamp path will self-heal on next bootstrap).

deploy.py writes the hash directly during the remote uv-sync instead of just touching, so freshly deploy'd targets skip the legacy-empty-stamp branch entirely.

Tests

60/60 (was 57, +3). All three new tests sabotage PATH (no uv available) and run ./spectator install (which calls bootstrap for real, unlike help which short-circuits before bootstrap):

• Hash matches + post-rsync mtime → fast path → success.
• Hash mismatch → bootstrap correctly demands uv → fails with the right error.
• Empty stamp (legacy) → self-heals → success + writes the new hash.

Migration

None. Existing v0.3.9-v0.4.2 installs upgrade in place: their empty stamp triggers the self-healing path on first invocation, which writes the hash.

v0.4.2 — Shell-escape audio fetch's remote rsync path

Closes bug 3 from the 2026-05-09 bug report.

spectator audio fetch --target HOST --only "stem with spaces" failed because the remote path wasn't shell-escaped before going through SSH to remote rsync. Symptom on a stem like MY-NIMA 1_1 0507:

fetch failed
rsync: link_stat "/home/misha/.co-sa/spectator/audio-out/MY-NIMA" failed: No such file or directory (2)
rsync: link_stat "/home/misha/1_1" failed: No such file or directory (2)
rsync: change_dir "/home/misha//0507" failed: No such file or directory (2)

Real-world hit because macOS meeting recordings often have spaces in basenames (zoom default "YYYY-MM-DD HH.MM Meeting Name.mp3"). Same hazard with parens, brackets, $, backticks — anything the remote shell interprets.

Fix: src/audio.py:fetch() now wraps the remote path with shlex.quote() before joining onto the host: prefix. The remote shell sees one argument, rsync forwards it intact.

Tests: 57/57 (was 55, +2). One feeds a stem combining spaces / parens / brackets / $ / backticks (real-world torture) and verifies shlex.split round-trips the rendered rsync argv to a single token. One pins that the no-host (local-only) branch does NOT add quoting (would create a literal-quote-character filename).

Workaround on older versions

If you can't upgrade right now, plain rsync with the inner-quoting trick works:

rsync -avh --progress \
    'host:".test/audio-out/My Recording/"' \
    /tmp/dest/

v0.4.1 — Skip MPS in audio auto-detect (openai-whisper crashes on Apple Silicon GPU)

Closes bug 2 from the 2026-05-09 bug report.

openai-whisper × torch >= 2.x crashes on Apple Silicon GPU for the entire large-v3 model family — and all four of Spectator's quality presets (studio / meeting / phone / extreme) use large-v3 or large-v3-turbo. So a fresh Apple Silicon Mac with spectator audio transcribe meeting.mp3 (auto-detect → MPS) crashes with:

TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework
doesn't support float64. Please use float32 instead.

Tracked upstream at openai/whisper#2151.

v0.4.1 mitigation

_detect_device now downgrades probe-detected mps → cpu by default, with a one-line console warning explaining the override. Two ways to bypass:

• Per-call: --device mps (always honored — the user is asking explicitly).
• Globally: SPECTATOR_ALLOW_MPS_AUTO=1 in your shell. Useful if you've patched whisper locally or upstream has shipped a fix.

audio install's verify step now prints the same heads-up on Apple Silicon so the operator sees the situation at install time too.

Real-world impact for Apple Silicon users

Mode	RT factor on M-series for meeting preset
Default (CPU after MPS auto-skip)	real-time to 2× slower
--device mps (explicit, may crash on large-v3)	2-4× faster when working
--target <cuda-host> over SSH	10-30× faster

Still tractable for occasional use on Apple Silicon CPU; heavy jobs should still go to a CUDA target.

Tests

55/55 (was 53, +2): test_detect_device_skips_mps_by_default confirms probe-mps + no env var → cpu + warning containing the issue link, test_detect_device_respects_mps_opt_in_env_var confirms SPECTATOR_ALLOW_MPS_AUTO=1 re-enables MPS and silences the warning.

Bug 1 from the same report

Already fixed in v0.3.6 (literal $HOME directory created under cwd when --workdir ~/...). v0.4.1 closes only bug 2.

v0.4.0 — spectator uninstall (inverse of install; rm -rf workdir)

Adds a first-class spectator uninstall verb. spectator down was always meant to be non-destructive (stops processes / containers / tmux, leaves the install on disk for next bring-up); v0.4.0 makes the cleanup-the-whole-thing case a first-class verb instead of "manually rm -rf and hope you typed the right path".

Behavior

$ ./spectator uninstall --target myspark1-local
This will stop everything Spectator launches on myspark1-local
and remove ~/.spectator/ (the entire workdir, including VSS clone,
audio-venv, ui-server state, and any transcripts in audio-out/).
Proceed? [y/N]:

1. Confirms (defaults to No so a stray Enter doesn't nuke anything; pass --force / -y for scripts).
2. Calls stack.down() first so VSS containers + tmux sessions exit cleanly. Best-effort — if down already ran, that's fine.
3. rm -rf $workdir/ with safety guards: the bash payload refuses to operate when the workdir resolves to empty / / / $HOME.
4. Prints a checklist of categories deliberately left alone, with the canonical commands to clean each manually.

Left untouched (and why)

• Docker images cached locally (~30 GB). Re-pulling is slow (~30-45 min) and the cache may be useful for adjacent NIM-based tooling. List: docker images | grep nvcr.io. Remove: docker rmi <image-id>.
• ~/.docker/config.json NGC login entry. May coexist with other registries the user uses. Revoke: docker logout nvcr.io.
• --apply-system mutations (nvidia-ctk runtime config, docker group membership). These were opt-in via an explicit flag, so reversal is also explicit.
• ~/.local/bin/uv. The user installed it and may use it for other projects.

Tests

53/53 (was 50, +3): bash payload pinned for safety guards + each "left alone" category mention; stack.down is called before rm -rf (verified for both local and remote paths via monkey-patched call-order capture); --help advertises --target / --workdir / --force and the destructive-action language.

down vs uninstall — when to use which

	down	uninstall
Stops VSS containers	yes	yes (via down internally)
Kills tmux sessions	yes	yes (via down internally)
Removes $workdir/	no	yes
Use case	between work sessions	tearing down a host you're done with

v0.3.9 — deploy() creates the wrapper install stamp + syncs dev deps

Pairs with v0.3.8. After v0.3.8 the ./spectator wrapper takes a fast path skipping require_uv when .venv/ looks healthy, but on a freshly deploy'd target two things kept that fast path from triggering:

1. deploy.py ran uv sync (runtime only) over SSH; the wrapper's deps_ready() probe checks for pytest + ruff and refused.
2. The wrapper's install-stamp file (.venv/.spectator-installed) was never created on the target; the wrapper's mtime check refused the fast path even when deps_ready would have passed.

Net effect: ./spectator <verb> on a deploy'd Spark exploded with "uv is required" on every invocation over a non-interactive SSH session (where ~/.local/bin typically isn't on PATH), even though the venv was fully populated.

Fix: deploy.deploy() now uses uv sync --extra dev and touch .venv/.spectator-installed after the sync. Adds ~30 MB of dev-deps to the target (pytest + ruff + transitives), negligible vs the multi-GB VSS docker images.

50/50 tests pass (was 49, +1: test_deploy_remote_uv_sync_includes_dev_and_writes_install_stamp introspects deploy.deploy source to pin both new lines).