fix(audio): defer capture rebuilds during route changes#236
Conversation
📝 WalkthroughWalkthroughThe PR refactors capture-environment event handling in ChangesDeferred Capture Restart Flow
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
db97c55 to
0ec8b73
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.changeset/4dca4969.md:
- Line 5: Append the GitHub issue/PR reference to the changeset summary line
"Defer Fast Mode capture rebuilds until audio route changes settle" by appending
the tracking token in the required format, e.g. " (`#236`)" so the fragment
matches release-note conventions and includes the user-facing impact plus the
PR/issue number in parentheses.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 1408268c-af43-42b6-8ed0-8724cad4767c
📒 Files selected for processing (3)
.changeset/4dca4969.mdHex/Clients/RecordingClient.swiftHex/Clients/SuperFastCaptureController.swift
| "hex-app": patch | ||
| --- | ||
|
|
||
| Defer Fast Mode capture rebuilds until audio route changes settle |
There was a problem hiding this comment.
Add PR/issue reference to the changeset summary.
Please append the tracking reference in the required format, e.g. (#236), so this fragment matches release-note conventions.
Suggested edit
-Defer Fast Mode capture rebuilds until audio route changes settle
+Defer Fast Mode capture rebuilds until audio route changes settle (`#236`)As per coding guidelines: “Include user-facing impact and GitHub issue/PR number in the format 'Improve Fn hotkey stability (#89)'.”
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Defer Fast Mode capture rebuilds until audio route changes settle | |
| Defer Fast Mode capture rebuilds until audio route changes settle (`#236`) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.changeset/4dca4969.md at line 5, Append the GitHub issue/PR reference to
the changeset summary line "Defer Fast Mode capture rebuilds until audio route
changes settle" by appending the tracking token in the required format, e.g. "
(`#236`)" so the fragment matches release-note conventions and includes the
user-facing impact plus the PR/issue number in parentheses.
FluidAudio's main moved the batch ASR tree under ASR/Parakeet/SlidingWindow/TDT/ and changed the AsrManager surface: initialize(models:) was removed in favor of loadModels(_:), and transcribe(url) now requires a caller-managed inout TdtDecoderState. Pin to 9c20d9bc (the revision that also carries StreamingNemotronMultilingualAsrManager, needed for upcoming streaming work) and adapt ParakeetClient accordingly. For one-shot file transcription we hand transcribe(_:decoderState:) a fresh TdtDecoderState sized to the loaded model's decoder layer count; FluidAudio carries context across its own internal chunks via that state. Only ParakeetClient references FluidAudio, so the blast radius is contained; import paths are unchanged because Swift module namespaces are flat. Verified: Release build succeeds and Parakeet v3 downloads + transcribes correctly at runtime. Groundwork for the Nemotron streaming integration (kitlangton#236). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FluidAudio's main moved the batch ASR tree under ASR/Parakeet/SlidingWindow/TDT/ and changed the AsrManager surface: initialize(models:) was removed in favor of loadModels(_:), and transcribe(url) now requires a caller-managed inout TdtDecoderState. Pin to 9c20d9bc (the revision that also carries StreamingNemotronMultilingualAsrManager, needed for upcoming streaming work) and adapt ParakeetClient accordingly. For one-shot file transcription we hand transcribe(_:decoderState:) a fresh TdtDecoderState sized to the loaded model's decoder layer count; FluidAudio carries context across its own internal chunks via that state. Only ParakeetClient references FluidAudio, so the blast radius is contained; import paths are unchanged because Swift module namespaces are flat. Verified: Release build succeeds and Parakeet v3 downloads + transcribes correctly at runtime. Groundwork for the Nemotron streaming integration (kitlangton#236). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduce a FluidEngine discriminator (.tdtBatch / .nemotronStreaming) on ParakeetModel as the source of truth for transcription routing, with derived isStreaming, and add the nemotron-3.5-asr-streaming-0.6b-coreml case plus its curated models.json entry. capabilityLabel / recommendationLabel / the curated badge gain a branch for the new case. The streaming model has no client or routing yet (those land in kitlangton#236 Phase 2), so selecting it would fail to download. Gate it out of all three model-exposure surfaces (curated list, availableModels, getAvailableModels) via a temporary ParakeetModel.isSelectable flag so it stays invisible until streaming is wired; Phase 2 deletes the flag and its call sites. ParakeetClient.asrVersion is now optional and ensureLoaded throws a clear error if a non-batch model ever reaches it. No changeset: this is fully gated and ships no user-facing change. The user-facing changeset lands with the phase that removes the gate. Verified: Debug build succeeds and HexCore swift test passes (62 tests, 5 suites). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add NemotronStreamingClient, an actor wrapping FluidAudio's StreamingNemotronMultilingualAsrManager: ensureLoaded (downloadVariant + loadModels with progress), availability/delete against the on-disk layout (<repo folderName>/<latin|multilingual>/<chunkMs>ms), and the live session methods startUtterance / feed / finishUtterance. Mirrors ParakeetClient's canImport(FluidAudio) split and NSError conventions. Wire routing into TranscriptionClient: a TranscriptionUpdate enum (.partial/.final), isStreamingModel(_:), and transcribeStreaming which loads the variant, forwards the manager's cumulative partial callback, feeds sample chunks through a single serial consumer to preserve frame order, and emits a terminal .final when the input ends. Notes from verifying the FluidAudio API at the pinned revision: - process(samples:) always returns "" - partials surface only via the callback, which delivers the cumulative running transcript (not a delta). - on-disk folder is Repo.folderName "nemotron-multilingual" (not the long repo name), and languageDirectory collapses en/es/fr/it/pt/de to "latin". - the stream is AsyncThrowingStream (not AsyncStream) so load/decode failures (e.g. Intel unsupportedPlatform) reach the Phase 4 caller instead of being swallowed. Nothing calls transcribeStreaming yet (Phase 4 inverts the recording flow) and the model stays UI-gated via isSelectable, so main remains shippable. Part of the Nemotron streaming integration (kitlangton#236), Phase 2. Verified: Debug build succeeds and HexCore swift test passes (62 tests, 5 suites). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-ups from testing the Nemotron live flow: - Indicator: move the live-transcript bubble below the recording dot (clear of the MacBook notch); it grows horizontally to ~35% screen width, then wraps downward up to 10 lines, head-truncated so the newest words stay visible. - Chunk size: switching tiers re-checks the new variant's on-disk availability (each chunk is a separate ~600 MB download) and refreshes model readiness, so an un-downloaded tier prompts a download instead of silently downloading at record time and hanging. Switching tiers mid-download cancels the abandoned download silently. - Streaming settings: inline download status (ready / not-downloaded + Download / progress + Cancel) under the chunk picker, and an info popover on streaming model rows explaining the per-chunk download. - Model list: per-model "Reveal in Finder" button shown only for downloaded models, resolving each engine's actual on-disk folder (handles WhisperKit, Parakeet's -coreml-stripped folder, and Nemotron's repo folder). Part of the Nemotron streaming integration (kitlangton#236). Verified: Debug build succeeds and HexCore swift test passes (62 tests, 5 suites). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add `autoDownloadChunkOnChange` (default off). When enabled, switching to a Nemotron chunk tier that isn't on disk starts its ~600 MB download automatically after the availability re-check; when off (the default), the chunk change just surfaces the inline Download prompt as before. Default off so an accidental chunk change can't kick off a large download on a limited connection. Exposed as a checkbox under the chunk-size picker. Part of the Nemotron streaming integration (kitlangton#236). Verified: Debug build succeeds and HexCore swift test passes (62 tests, 5 suites). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the Nemotron streaming model end-to-end and expose it in the UI. This inverts the record-then-transcribe flow for streaming models: audio is decoded live while recording, with an optional running transcript in the indicator. - Audio tee: SuperFastCaptureController gains an optional converted-samples sink (fed on the processing queue in capture order, including pre-roll); RecordingClient exposes observeAudioSamples() and finishes the stream on stop. - Flow inversion: TranscriptionFeature starts a streaming effect at record-start that installs the sample sink before recording (so pre-roll isn't clipped), pipes samples through transcribeStreaming, and shows cumulative partials. Stop finishes the sample stream -> terminal .final; a URL/text rendezvous then reuses the shared result path (word remap/removal, paste, history). - Settings: nemotronChunkMs (560/1120/2240/4480, default 2240) and showLivePartials, plus a "Streaming" settings section shown for streaming models (chunk-size help text notes the 0.56s punctuation tradeoff, #687). - Routing: TranscriptionClient routes download/availability/delete for streaming models to NemotronStreamingClient; file-based transcribe rejects them. - Remove the Phase 1 isSelectable gate so Nemotron is selectable. Hardening from self-review: a streaming error mid-recording now stops the mic and clears state (no hot-mic hang); record-start cancels a stale finalize so a late rendezvous half can't pair with a new recording's audio; the sample sink is installed before startRecording to avoid dropping pre-roll. Known limitations: streaming requires the capture-engine backend (the AVAudioRecorder fallback has no live PCM tap, yielding an empty transcript); changing chunk size or language after download triggers an on-demand variant download at the next record-start. Completes the Nemotron streaming integration (kitlangton#236). Verified: Debug build succeeds and HexCore swift test passes (62 tests, 5 suites). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-ups from testing the Nemotron live flow: - Indicator: move the live-transcript bubble below the recording dot (clear of the MacBook notch); it grows horizontally to ~35% screen width, then wraps downward up to 10 lines, head-truncated so the newest words stay visible. - Chunk size: switching tiers re-checks the new variant's on-disk availability (each chunk is a separate ~600 MB download) and refreshes model readiness, so an un-downloaded tier prompts a download instead of silently downloading at record time and hanging. Switching tiers mid-download cancels the abandoned download silently. - Streaming settings: inline download status (ready / not-downloaded + Download / progress + Cancel) under the chunk picker, and an info popover on streaming model rows explaining the per-chunk download. - Model list: per-model "Reveal in Finder" button shown only for downloaded models, resolving each engine's actual on-disk folder (handles WhisperKit, Parakeet's -coreml-stripped folder, and Nemotron's repo folder). Part of the Nemotron streaming integration (kitlangton#236). Verified: Debug build succeeds and HexCore swift test passes (62 tests, 5 suites). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add `autoDownloadChunkOnChange` (default off). When enabled, switching to a Nemotron chunk tier that isn't on disk starts its ~600 MB download automatically after the availability re-check; when off (the default), the chunk change just surfaces the inline Download prompt as before. Default off so an accidental chunk change can't kick off a large download on a limited connection. Exposed as a checkbox under the chunk-size picker. Part of the Nemotron streaming integration (kitlangton#236). Verified: Debug build succeeds and HexCore swift test passes (62 tests, 5 suites). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a "Transcribe" tab for importing audio/video files via drag-and-drop or the file picker, transcribing them with the currently selected model. Adapted from kitlangton#232 for this fork's dual-engine setup. - Works with every engine. WhisperKit and Parakeet batch use the existing file-based transcribe; the Nemotron streaming model gains a one-shot transcribeFile path that feeds the whole file through the streaming manager (process(audioBuffer:) + finish()) in ~5s blocks. TranscriptionClient routes streaming models here instead of rejecting file-based transcribe. - Per-job progress: determinate percentage + bar for the Nemotron file path (real framesFed/total), indeterminate bar for batch engines that don't expose a decode fraction, plus a live elapsed timer for every job. - Multi-file import, per-job copy/remove, Clear Finished, and clear errors for unsupported or silent files. Imported transcripts save to History (preserving the original file extension) when history is enabled. Part of the Nemotron streaming integration (kitlangton#236). Verified: Debug build succeeds and HexCore swift test passes (62 tests, 5 suites); file transcription runtime-tested across engines with live progress. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Summary
AVAudioEngineteardown/recreation while AirPods handoff or Bluetooth profile changes may still be dispatching Core Audio callbacksAVAudioRecorderinstances after export instead of re-priming them while idleWhy
Issue #226 includes an AirPods handoff crash where an
AVAudioIOUnitproperty callback racesAVAudioEnginedeallocation. This is a containment patch: it narrows teardown timing risk without disabling Fast Mode or changing its normal low-latency path.Verification
git diff --checkcd HexCore && swift testxcodebuild -scheme Hex -configuration Debug buildFollow-up
This does not claim to fully solve Bluetooth A2DP-to-HFP stabilization. A later PR should add explicit capture outcomes and generation-scoped callback handling so route interruptions are surfaced rather than silently producing invalid audio.
Stack
Depends on #235.
Stack
Summary by CodeRabbit
New Features
Bug Fixes