Skip to content

fix(audio): defer capture rebuilds during route changes#236

Merged
kitlangton merged 1 commit into
mainfrom
fix/audio-route-containment
Jun 4, 2026
Merged

fix(audio): defer capture rebuilds during route changes#236
kitlangton merged 1 commit into
mainfrom
fix/audio-route-containment

Conversation

@kitlangton

@kitlangton kitlangton commented Jun 4, 2026

Copy link
Copy Markdown
Owner

Summary

  • keep Fast Mode warm-buffer behavior unchanged during ordinary idle and repeated dictation
  • when Core Audio route events occur, clear stale pre-roll and defer capture-engine rebuilds until the next explicit warm-up or recording request
  • avoid eager AVAudioEngine teardown/recreation while AirPods handoff or Bluetooth profile changes may still be dispatching Core Audio callbacks
  • release fallback AVAudioRecorder instances after export instead of re-priming them while idle
  • keep standard-mode capture startup on demand after environment changes

Why

Issue #226 includes an AirPods handoff crash where an AVAudioIOUnit property callback races AVAudioEngine deallocation. This is a containment patch: it narrows teardown timing risk without disabling Fast Mode or changing its normal low-latency path.

Verification

  • git diff --check
  • cd HexCore && swift test
  • xcodebuild -scheme Hex -configuration Debug build

Follow-up

This does not claim to fully solve Bluetooth A2DP-to-HFP stabilization. A later PR should add explicit capture outcomes and generation-scoped callback handling so route interruptions are surfaced rather than silently producing invalid audio.

Stack

Depends on #235.

Stack

  1. fix: harden microphone and recording reliability #235
  2. fix(audio): defer capture rebuilds during route changes #236 👈 current

Summary by CodeRabbit

  • New Features

    • Fast Mode capture rebuilds now defer until audio route changes finish settling, improving handling of audio input/output switches.
  • Bug Fixes

    • Enhanced stability when connecting/disconnecting audio devices or switching system audio routes.

@coderabbitai

coderabbitai Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The PR refactors capture-environment event handling in RecordingClientLive to defer capture-engine restarts in super-fast mode until audio-route changes settle. The forceRestart parameter is removed, restart decisions centralize through deferred state (captureControllerNeedsRestartReason), and all system observers (wake, device, hardware) enqueue non-forcing changes. A new ensureCaptureControllerReadyAfterDeferredRestart helper applies deferred restarts at recording start and warm-up. Fallback stop is simplified to always attempt duplicate + release.

Changes

Deferred Capture Restart Flow

Layer / File(s) Summary
Environment-change handler refactoring
Hex/Clients/RecordingClient.swift
Removes forceRestart parameter; enqueueCaptureEnvironmentChange debounces and calls the updated handleCaptureEnvironmentChange(reason:). Handler now decides whether to defer rebuild (super-fast mode: set captureControllerNeedsRestartReason, clear warm buffer, release recorder) or apply immediately (standard mode: adjust inputs, stop/release).
Deferred restart support infrastructure
Hex/Clients/RecordingClient.swift, Hex/Clients/SuperFastCaptureController.swift
New ensureCaptureControllerReadyAfterDeferredRestart(for:reason:) helper applies deferred restarts by checking and clearing captureControllerNeedsRestartReason. New clearWarmBuffer() method on SuperFastCaptureController synchronously clears ring buffer when no recording is active.
Observer callback updates
Hex/Clients/RecordingClient.swift
Capture engine configuration changes, wake/screen-wake events, AV device connect/disconnect, and CoreAudio hardware property observers all enqueue environment changes using simplified signature (no force restart).
Recording start and warm-up integration
Hex/Clients/RecordingClient.swift
startRecording and warmUpRecorder now arm capture controller via ensureCaptureControllerReadyAfterDeferredRestart instead of direct call, so restart decisions incorporate deferred-restart state.
Fallback stop path simplification
Hex/Clients/RecordingClient.swift
Recorder fallback stop logic no longer conditionally primes recorder based on copy success; always attempts to duplicate current recording (best-effort) then immediately releases recorder with fallback-stop reason.
Release notes
.changeset/4dca4969.md
Patch release entry: Fast Mode capture rebuilds are deferred until audio route changes settle.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Possibly related PRs

  • kitlangton/Hex#235: Both PRs modify Hex/Clients/RecordingClient.swift's recording lifecycle to handle deferred capture restart more reliably—this PR introduces the deferred environment-change restart flow via super-fast restart state, while the related PR hardens fallback stop/cleanup to flush that deferred restart path.

Poem

🐰 A capture's dance, now swift and sure,
Defers its rebuild, lets changes endure,
No forced restart blocks the way,
Warm buffers clear, recorders play.
Audio routes may shift and bend,
But captures flow from start to end. 🎙️

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'fix(audio): defer capture rebuilds during route changes' directly summarizes the main change: deferring capture-engine rebuilds when audio routes change, which is the central objective across all three modified files.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/audio-route-containment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@kitlangton kitlangton changed the base branch from fix/microphone-recording-reliability to main June 4, 2026 15:35
@kitlangton kitlangton force-pushed the fix/audio-route-containment branch from db97c55 to 0ec8b73 Compare June 4, 2026 15:35

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.changeset/4dca4969.md:
- Line 5: Append the GitHub issue/PR reference to the changeset summary line
"Defer Fast Mode capture rebuilds until audio route changes settle" by appending
the tracking token in the required format, e.g. " (`#236`)" so the fragment
matches release-note conventions and includes the user-facing impact plus the
PR/issue number in parentheses.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 1408268c-af43-42b6-8ed0-8724cad4767c

📥 Commits

Reviewing files that changed from the base of the PR and between c00a91d and 0ec8b73.

📒 Files selected for processing (3)
  • .changeset/4dca4969.md
  • Hex/Clients/RecordingClient.swift
  • Hex/Clients/SuperFastCaptureController.swift

Comment thread .changeset/4dca4969.md
"hex-app": patch
---

Defer Fast Mode capture rebuilds until audio route changes settle

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Add PR/issue reference to the changeset summary.

Please append the tracking reference in the required format, e.g. (#236), so this fragment matches release-note conventions.

Suggested edit
-Defer Fast Mode capture rebuilds until audio route changes settle
+Defer Fast Mode capture rebuilds until audio route changes settle (`#236`)

As per coding guidelines: “Include user-facing impact and GitHub issue/PR number in the format 'Improve Fn hotkey stability (#89)'.”

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
Defer Fast Mode capture rebuilds until audio route changes settle
Defer Fast Mode capture rebuilds until audio route changes settle (`#236`)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.changeset/4dca4969.md at line 5, Append the GitHub issue/PR reference to
the changeset summary line "Defer Fast Mode capture rebuilds until audio route
changes settle" by appending the tracking token in the required format, e.g. "
(`#236`)" so the fragment matches release-note conventions and includes the
user-facing impact plus the PR/issue number in parentheses.

@kitlangton kitlangton merged commit 71878b7 into main Jun 4, 2026
1 check passed
sperka added a commit to sperka/Hex that referenced this pull request Jun 26, 2026
FluidAudio's main moved the batch ASR tree under ASR/Parakeet/SlidingWindow/TDT/
and changed the AsrManager surface: initialize(models:) was removed in favor of
loadModels(_:), and transcribe(url) now requires a caller-managed inout
TdtDecoderState. Pin to 9c20d9bc (the revision that also carries
StreamingNemotronMultilingualAsrManager, needed for upcoming streaming work) and
adapt ParakeetClient accordingly.

For one-shot file transcription we hand transcribe(_:decoderState:) a fresh
TdtDecoderState sized to the loaded model's decoder layer count; FluidAudio
carries context across its own internal chunks via that state. Only
ParakeetClient references FluidAudio, so the blast radius is contained; import
paths are unchanged because Swift module namespaces are flat.

Verified: Release build succeeds and Parakeet v3 downloads + transcribes
correctly at runtime. Groundwork for the Nemotron streaming integration (kitlangton#236).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sperka added a commit to sperka/Hex that referenced this pull request Jun 26, 2026
FluidAudio's main moved the batch ASR tree under ASR/Parakeet/SlidingWindow/TDT/
and changed the AsrManager surface: initialize(models:) was removed in favor of
loadModels(_:), and transcribe(url) now requires a caller-managed inout
TdtDecoderState. Pin to 9c20d9bc (the revision that also carries
StreamingNemotronMultilingualAsrManager, needed for upcoming streaming work) and
adapt ParakeetClient accordingly.

For one-shot file transcription we hand transcribe(_:decoderState:) a fresh
TdtDecoderState sized to the loaded model's decoder layer count; FluidAudio
carries context across its own internal chunks via that state. Only
ParakeetClient references FluidAudio, so the blast radius is contained; import
paths are unchanged because Swift module namespaces are flat.

Verified: Release build succeeds and Parakeet v3 downloads + transcribes
correctly at runtime. Groundwork for the Nemotron streaming integration (kitlangton#236).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sperka added a commit to sperka/Hex that referenced this pull request Jun 26, 2026
Introduce a FluidEngine discriminator (.tdtBatch / .nemotronStreaming) on
ParakeetModel as the source of truth for transcription routing, with derived
isStreaming, and add the nemotron-3.5-asr-streaming-0.6b-coreml case plus its
curated models.json entry. capabilityLabel / recommendationLabel / the curated
badge gain a branch for the new case.

The streaming model has no client or routing yet (those land in kitlangton#236 Phase 2),
so selecting it would fail to download. Gate it out of all three model-exposure
surfaces (curated list, availableModels, getAvailableModels) via a temporary
ParakeetModel.isSelectable flag so it stays invisible until streaming is wired;
Phase 2 deletes the flag and its call sites. ParakeetClient.asrVersion is now
optional and ensureLoaded throws a clear error if a non-batch model ever reaches
it.

No changeset: this is fully gated and ships no user-facing change. The
user-facing changeset lands with the phase that removes the gate.

Verified: Debug build succeeds and HexCore swift test passes (62 tests, 5 suites).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sperka added a commit to sperka/Hex that referenced this pull request Jun 26, 2026
Add NemotronStreamingClient, an actor wrapping FluidAudio's
StreamingNemotronMultilingualAsrManager: ensureLoaded (downloadVariant +
loadModels with progress), availability/delete against the on-disk layout
(<repo folderName>/<latin|multilingual>/<chunkMs>ms), and the live session
methods startUtterance / feed / finishUtterance. Mirrors ParakeetClient's
canImport(FluidAudio) split and NSError conventions.

Wire routing into TranscriptionClient: a TranscriptionUpdate enum
(.partial/.final), isStreamingModel(_:), and transcribeStreaming which loads
the variant, forwards the manager's cumulative partial callback, feeds sample
chunks through a single serial consumer to preserve frame order, and emits a
terminal .final when the input ends.

Notes from verifying the FluidAudio API at the pinned revision:
- process(samples:) always returns "" - partials surface only via the callback,
  which delivers the cumulative running transcript (not a delta).
- on-disk folder is Repo.folderName "nemotron-multilingual" (not the long repo
  name), and languageDirectory collapses en/es/fr/it/pt/de to "latin".
- the stream is AsyncThrowingStream (not AsyncStream) so load/decode failures
  (e.g. Intel unsupportedPlatform) reach the Phase 4 caller instead of being
  swallowed.

Nothing calls transcribeStreaming yet (Phase 4 inverts the recording flow) and
the model stays UI-gated via isSelectable, so main remains shippable. Part of
the Nemotron streaming integration (kitlangton#236), Phase 2.

Verified: Debug build succeeds and HexCore swift test passes (62 tests, 5 suites).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sperka added a commit to sperka/Hex that referenced this pull request Jun 27, 2026
Follow-ups from testing the Nemotron live flow:

- Indicator: move the live-transcript bubble below the recording dot (clear of
  the MacBook notch); it grows horizontally to ~35% screen width, then wraps
  downward up to 10 lines, head-truncated so the newest words stay visible.
- Chunk size: switching tiers re-checks the new variant's on-disk availability
  (each chunk is a separate ~600 MB download) and refreshes model readiness, so
  an un-downloaded tier prompts a download instead of silently downloading at
  record time and hanging. Switching tiers mid-download cancels the abandoned
  download silently.
- Streaming settings: inline download status (ready / not-downloaded + Download
  / progress + Cancel) under the chunk picker, and an info popover on streaming
  model rows explaining the per-chunk download.
- Model list: per-model "Reveal in Finder" button shown only for downloaded
  models, resolving each engine's actual on-disk folder (handles WhisperKit,
  Parakeet's -coreml-stripped folder, and Nemotron's repo folder).

Part of the Nemotron streaming integration (kitlangton#236).

Verified: Debug build succeeds and HexCore swift test passes (62 tests, 5 suites).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sperka added a commit to sperka/Hex that referenced this pull request Jun 27, 2026
Add `autoDownloadChunkOnChange` (default off). When enabled, switching to a
Nemotron chunk tier that isn't on disk starts its ~600 MB download automatically
after the availability re-check; when off (the default), the chunk change just
surfaces the inline Download prompt as before.

Default off so an accidental chunk change can't kick off a large download on a
limited connection. Exposed as a checkbox under the chunk-size picker.

Part of the Nemotron streaming integration (kitlangton#236).

Verified: Debug build succeeds and HexCore swift test passes (62 tests, 5 suites).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sperka added a commit to sperka/Hex that referenced this pull request Jun 27, 2026
Wire the Nemotron streaming model end-to-end and expose it in the UI. This
inverts the record-then-transcribe flow for streaming models: audio is decoded
live while recording, with an optional running transcript in the indicator.

- Audio tee: SuperFastCaptureController gains an optional converted-samples sink
  (fed on the processing queue in capture order, including pre-roll);
  RecordingClient exposes observeAudioSamples() and finishes the stream on stop.
- Flow inversion: TranscriptionFeature starts a streaming effect at record-start
  that installs the sample sink before recording (so pre-roll isn't clipped),
  pipes samples through transcribeStreaming, and shows cumulative partials. Stop
  finishes the sample stream -> terminal .final; a URL/text rendezvous then
  reuses the shared result path (word remap/removal, paste, history).
- Settings: nemotronChunkMs (560/1120/2240/4480, default 2240) and
  showLivePartials, plus a "Streaming" settings section shown for streaming
  models (chunk-size help text notes the 0.56s punctuation tradeoff, #687).
- Routing: TranscriptionClient routes download/availability/delete for streaming
  models to NemotronStreamingClient; file-based transcribe rejects them.
- Remove the Phase 1 isSelectable gate so Nemotron is selectable.

Hardening from self-review: a streaming error mid-recording now stops the mic
and clears state (no hot-mic hang); record-start cancels a stale finalize so a
late rendezvous half can't pair with a new recording's audio; the sample sink
is installed before startRecording to avoid dropping pre-roll.

Known limitations: streaming requires the capture-engine backend (the
AVAudioRecorder fallback has no live PCM tap, yielding an empty transcript);
changing chunk size or language after download triggers an on-demand variant
download at the next record-start.

Completes the Nemotron streaming integration (kitlangton#236).

Verified: Debug build succeeds and HexCore swift test passes (62 tests, 5 suites).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sperka added a commit to sperka/Hex that referenced this pull request Jun 27, 2026
Follow-ups from testing the Nemotron live flow:

- Indicator: move the live-transcript bubble below the recording dot (clear of
  the MacBook notch); it grows horizontally to ~35% screen width, then wraps
  downward up to 10 lines, head-truncated so the newest words stay visible.
- Chunk size: switching tiers re-checks the new variant's on-disk availability
  (each chunk is a separate ~600 MB download) and refreshes model readiness, so
  an un-downloaded tier prompts a download instead of silently downloading at
  record time and hanging. Switching tiers mid-download cancels the abandoned
  download silently.
- Streaming settings: inline download status (ready / not-downloaded + Download
  / progress + Cancel) under the chunk picker, and an info popover on streaming
  model rows explaining the per-chunk download.
- Model list: per-model "Reveal in Finder" button shown only for downloaded
  models, resolving each engine's actual on-disk folder (handles WhisperKit,
  Parakeet's -coreml-stripped folder, and Nemotron's repo folder).

Part of the Nemotron streaming integration (kitlangton#236).

Verified: Debug build succeeds and HexCore swift test passes (62 tests, 5 suites).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sperka added a commit to sperka/Hex that referenced this pull request Jun 27, 2026
Add `autoDownloadChunkOnChange` (default off). When enabled, switching to a
Nemotron chunk tier that isn't on disk starts its ~600 MB download automatically
after the availability re-check; when off (the default), the chunk change just
surfaces the inline Download prompt as before.

Default off so an accidental chunk change can't kick off a large download on a
limited connection. Exposed as a checkbox under the chunk-size picker.

Part of the Nemotron streaming integration (kitlangton#236).

Verified: Debug build succeeds and HexCore swift test passes (62 tests, 5 suites).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sperka added a commit to sperka/Hex that referenced this pull request Jul 1, 2026
Add a "Transcribe" tab for importing audio/video files via drag-and-drop or the
file picker, transcribing them with the currently selected model. Adapted from
kitlangton#232 for this fork's dual-engine setup.

- Works with every engine. WhisperKit and Parakeet batch use the existing
  file-based transcribe; the Nemotron streaming model gains a one-shot
  transcribeFile path that feeds the whole file through the streaming manager
  (process(audioBuffer:) + finish()) in ~5s blocks. TranscriptionClient routes
  streaming models here instead of rejecting file-based transcribe.
- Per-job progress: determinate percentage + bar for the Nemotron file path
  (real framesFed/total), indeterminate bar for batch engines that don't expose
  a decode fraction, plus a live elapsed timer for every job.
- Multi-file import, per-job copy/remove, Clear Finished, and clear errors for
  unsupported or silent files. Imported transcripts save to History (preserving
  the original file extension) when history is enabled.

Part of the Nemotron streaming integration (kitlangton#236).

Verified: Debug build succeeds and HexCore swift test passes (62 tests, 5 suites);
file transcription runtime-tested across engines with live progress.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant