Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,7 +246,11 @@ in the `audio` multipart part plus a JSON `config` part (`sample_rate`, `channel
empty `llm` block). No model header — the service pins the STT model server-side. The `prompt`
(built per utterance by `TranscriptionPrompt`) steers _transcription_; the `llm` block asks the
service to run its default LLM cleanup rewrite (remove disfluencies, fix punctuation) over the
verbatim transcript, all inside the same request. The response carries both `text` (verbatim) and
verbatim transcript, all inside the same request. The block rides along while **enhanced
transcripts** are enabled (`EnhancedTranscriptsStore`, on by default, read per request via the
transcriber's injected `enhancedTranscripts` closure); with the setting off the config omits `llm`
entirely, the service skips the rewrite, and the verbatim transcript is pasted as spoken. The
response carries both `text` (verbatim) and
`llm_response` (the rewrite); the transcriber returns the rewrite and falls back to `text` when
`llm_response` is null — the rewrite is best-effort (5 s server-side budget), so a rewrite failure
(`llm_error`) is a logged degradation, never a user-facing error.
Expand Down Expand Up @@ -405,6 +409,8 @@ Engine-side stores, all `UserDefaults`-backed value types with the same shape:
- **`TriggerKeyStore`** (`BlurtTriggerKeyCode`), **`SoundPackStore`** (`BlurtSoundPack`),
**`KeyTermsStore`** (the user's domain vocabulary, re-read at every press via the session's
`keyTermsProvider`), **`DeveloperModeStore`** (`BlurtDeveloperMode`, off by default),
**`EnhancedTranscriptsStore`** (`BlurtEnhancedTranscripts`, **on** by default — unset reads as
enabled; gates the dictation request's `llm` cleanup-rewrite block, re-read at every request),
**`OverlayOriginStore`** (the pill's dragged origin, x/y), **`LastUpdateCheckStore`**
(`BlurtLastUpdateCheck`, the stamp throttling the automatic launch update check).
- **`PersistedSettings.allDefaultsKeys`** is the roster of every key those stores write, and
Expand Down
34 changes: 32 additions & 2 deletions App/Blurt/Blurt/Wizard/SettingsWindowRoot.swift
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import BlurtEngine
import SwiftUI

/// Root view of the `Settings` scene. A `TabView` at the root of a `Settings`
Expand Down Expand Up @@ -66,19 +67,48 @@ private struct GeneralSettingsTab: View {
}
}

/// The occasional stuff: checking for an update and the developer-mode log
/// toggle. Kept out of General so the common pane stays short.
/// The occasional stuff: the enhanced-transcripts switch, checking for an
/// update, and the developer-mode log toggle. Kept out of General so the
/// common pane stays short.
private struct AdvancedSettingsTab: View {
let updateModel: UpdateCheckModel

var body: some View {
SettingsPane {
TranscriptionSection()
UpdateSection(model: updateModel)
DeveloperSection()
}
}
}

/// The Transcription section of the Settings window: the enhanced-transcripts
/// switch. While on (the default), every dictation request asks AssemblyAI's
/// dictation API for its server-side cleanup rewrite, so the pasted text is
/// the polished version; turned off, the request omits the rewrite and the
/// verbatim transcript is pasted exactly as spoken. The transcriber reads the
/// same default this toggle writes at every request, so a change applies to
/// the next dictation. Settings-only — not a wizard step, since it never
/// gates setup.
private struct TranscriptionSection: View {
@AppStorage(EnhancedTranscriptsStore.defaultsKey) private var enhancedTranscripts = true

var body: some View {
Section {
Toggle(isOn: $enhancedTranscripts) {
Label("Enhanced transcripts", systemImage: "wand.and.stars")
}
.accessibilityIdentifier(UITestIdentifiers.enhancedTranscriptsToggle)
} header: {
Text("Transcription")
} footer: {
Text(
"Polishes each dictation before pasting — removing filler words and fixing punctuation. "
+ "Turn off to paste your words exactly as spoken.")
}
Comment on lines +105 to +108

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wording here is deliberate, so leaving it as-is:

  • "Filler words" over "disfluencies": this is user-facing settings copy, and "filler words" is the plainer term; "disfluencies" stays in the engine docs where precision matters.
  • The best-effort fallback (a rare llm_error/null rewrite degrading to the verbatim transcript) is an internal degradation path, not something the setting controls — enumerating it in a two-sentence settings footer would add noise without helping the user decide whether to flip the toggle. It's documented in AGENTS.md/BLURTENGINE.md and covered by tests.

Also holding off on copy churn since the PR is green and already in the merge queue.


Generated by Claude Code

}
}

/// The Updates section of the Settings window: the running version and a
/// "Check for Updates" button that runs the check and reports the result in a
/// modal (see `UpdateCheckModel`). The same check is reachable from the
Expand Down
1 change: 1 addition & 0 deletions App/Blurt/Shared/UITestIdentifiers.swift
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ enum UITestIdentifiers {
static let hotkeyPicker = "settings.hotkey.picker"
static let soundPicker = "settings.sound.picker"
static let developerToggle = "settings.developer.toggle"
static let enhancedTranscriptsToggle = "settings.enhancedTranscripts.toggle"
static let updateCheck = "settings.update.check"

/// The dictation overlay pill (`OverlayView`).
Expand Down
4 changes: 2 additions & 2 deletions BLURTENGINE.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ press() ──▶ MicCapture.start() release() ──▶ MicCapture.s
Key properties of the design, which your integration can rely on:

- **One request per utterance, no streaming.** The dictation API returns the complete transcript — and its LLM-rewritten form — in the response body: no upload step, no job polling, no incremental deltas, no second request for the cleanup. `TranscriberProtocol.transcribe` is a single `async throws -> String`. UIs should show a "transcribing…" state and then the whole result; there is nothing to stream.
- **Cleanup happens server-side.** The request's empty `llm` block asks the service for its default cleanup rewrite (remove disfluencies, fix punctuation), applied to the verbatim transcript inside the same call; the per-utterance `config.prompt` (built by `TranscriptionPrompt` from the captured context) primes the _transcription_. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one.
- **Cleanup happens server-side, and it's optional.** The request's empty `llm` block asks the service for its default cleanup rewrite (remove disfluencies, fix punctuation), applied to the verbatim transcript inside the same call; the per-utterance `config.prompt` (built by `TranscriptionPrompt` from the captured context) primes the _transcription_. The block is gated by the **enhanced transcripts** setting (`EnhancedTranscriptsStore`, on by default): turned off, the config omits `llm` and the verbatim transcript is pasted as spoken. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one.
- **Latency is pre-paid where possible.** `press()` fires a detached `warmUp()` at the transcriber (pre-opening the HTTPS connection while the user speaks, ~170 ms saved cold) and kicks off the cross-process accessibility read of the focused field without awaiting it — the read is then consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`, 500 ms), so an unresponsive frontmost app costs the transcript its priming, never a multi-second stall — and never delays the recording indicator. On the way out, `release()` flips the phase to `.transcribing` _before_ reading the recorded audio back, so a host's stop cue fires at key-up rather than after the disk read.
- **A held trigger auto-releases.** `DictationSession` stops recording after `maxRecordingSeconds` (default `SyncSTTLimits.autoReleaseSeconds`, 115 s) so audio never exceeds what the endpoint accepts, and transcribes what it has. Clips shorter than `SyncSTTLimits.minPCMBytes` (~100 ms of audio — an accidental tap) are dropped as a silent no-op rather than sent to earn a 400.

Expand Down Expand Up @@ -133,7 +133,7 @@ func transcribe(pcm: Data, sampleRate: Int, context: TranscriptionContext?) asyn
func warmUp() async // optional; no-op default
```

`AssemblyAITranscriber` is a stateless `Sendable` struct. One `POST https://dictation.assemblyai.com/transcribe` per utterance: the audio as raw S16LE PCM (the `pcm` blob, byte-for-byte) in the `audio` multipart part, plus a JSON `config` part (`sample_rate`, `channels`, the rendered `prompt`, and an empty `llm` block requesting the service's default cleanup rewrite), with the API key in `Authorization` (no model header — the service pins the STT model server-side). The response carries the verbatim `text` and the rewritten `llm_response`; the transcriber returns the rewrite and falls back to `text` when it is null (the rewrite is best-effort — `llm_error` is logged, never surfaced as a failure). Its initializer takes an `apiKeyProvider` closure (defaults to `APIKeyStore.current`), a `baseURL`, and an `HTTPTransport` — inject a fake transport (see `Tests/BlurtEngineTests/Stubs/FakeHTTPTransport.swift`) to test against canned responses. `warmUp()` fires a throwaway GET at the host root to pre-pool the connection; it never throws and any failure just means the real request pays connection setup as before.
`AssemblyAITranscriber` is a stateless `Sendable` struct. One `POST https://dictation.assemblyai.com/transcribe` per utterance: the audio as raw S16LE PCM (the `pcm` blob, byte-for-byte) in the `audio` multipart part, plus a JSON `config` part (`sample_rate`, `channels`, the rendered `prompt`, and — while enhanced transcripts are enabled, the default — an empty `llm` block requesting the service's default cleanup rewrite), with the API key in `Authorization` (no model header — the service pins the STT model server-side). The response carries the verbatim `text` and the rewritten `llm_response`; the transcriber returns the rewrite and falls back to `text` when it is null (the rewrite is best-effort — `llm_error` is logged, never surfaced as a failure). Its initializer takes an `apiKeyProvider` closure (defaults to `APIKeyStore.current`), a `baseURL`, an `HTTPTransport` — inject a fake transport (see `Tests/BlurtEngineTests/Stubs/FakeHTTPTransport.swift`) to test against canned responses — and an `enhancedTranscripts` closure deciding, per request, whether the `llm` block is sent (nil, the default, reads `EnhancedTranscriptsStore`). `warmUp()` fires a throwaway GET at the host root to pre-pool the connection; it never throws and any failure just means the real request pays connection setup as before.

The model's limits live in `SyncSTTLimits` (16 kHz sample rate, ~0.1 s–120 s audio, and the auto-release math — the sync STT model behind the dictation service) — the single source shared by the mic, the session, and the request so recorded and declared geometry can't drift.

Expand Down
29 changes: 29 additions & 0 deletions Sources/BlurtEngine/Config/EnhancedTranscriptsStore.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import Foundation

/// Persists the "enhanced transcripts" switch in `UserDefaults`. On by
/// default; the Settings window's Transcription section flips it. While on,
/// every dictation request carries the `llm` block asking the dictation API
/// for its server-side cleanup rewrite (remove disfluencies, fix punctuation);
/// turned off, the request omits the block and the verbatim transcript is
/// pasted exactly as spoken. `AssemblyAITranscriber` reads this at each
/// request, so a change applies to the very next dictation.
/// Same shape as `DeveloperModeStore` / `SoundPackStore`.
public struct EnhancedTranscriptsStore {
/// UserDefaults key holding the switch. Public so SwiftUI views can observe
/// it directly (e.g. `@AppStorage`) and re-render on change.
public static let defaultsKey = "BlurtEnhancedTranscripts"
private let defaults: UserDefaults

init(defaults: UserDefaults = .standard) {
self.defaults = defaults
}

/// Unset means **on** — the cleanup rewrite is the product's default
/// behavior, so only an explicit opt-out disables it. That inverts the
/// usual `bool(forKey:)` shape (which reads a missing key as false), hence
/// the presence check.
var isEnabled: Bool {
get { defaults.object(forKey: Self.defaultsKey) as? Bool ?? true }
nonmutating set { defaults.set(newValue, forKey: Self.defaultsKey) }
}
}
4 changes: 3 additions & 1 deletion Sources/BlurtEngine/Config/PersistedSettings.swift
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
import Foundation

/// The roster of `UserDefaults` keys the engine's settings stores persist:
/// trigger key, sound pack, key terms, developer mode, overlay origin, and the
/// trigger key, sound pack, key terms, developer mode, enhanced transcripts,
/// overlay origin, and the
/// timestamp throttling the automatic update check. Owned
/// here — next to the stores — so adding a store and adding it to every "reset
/// to a clean state" sweep (e.g. the app's UI-test launch reset) are the same
Expand All @@ -19,6 +20,7 @@ public enum PersistedSettings {
SoundPackStore.defaultsKey,
KeyTermsStore.defaultsKey,
DeveloperModeStore.defaultsKey,
EnhancedTranscriptsStore.defaultsKey,
OverlayOriginStore.xDefaultsKey,
OverlayOriginStore.yDefaultsKey,
LastUpdateCheckStore.defaultsKey,
Expand Down
35 changes: 26 additions & 9 deletions Sources/BlurtEngine/STT/AssemblyAITranscriber.swift
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,10 @@ private let transcriberLog = Logger(subsystem: BlurtIdentity.subsystem, category
/// A single `POST dictation.assemblyai.com/transcribe` carries the captured
/// audio (raw S16LE PCM, exactly the bytes the mic recorded — there is no
/// re-encoding pass) plus a JSON `config` part, and the response body carries
/// both the verbatim transcript and — because the config requests one via its
/// `llm` block — an LLM-rewritten version with disfluencies removed and
/// punctuation fixed. No upload step, no job submission, no polling — one
/// both the verbatim transcript and — when the config requests one via its
/// `llm` block (the "enhanced transcripts" setting, on by default) — an
/// LLM-rewritten version with disfluencies removed and punctuation fixed.
/// No upload step, no job submission, no polling — one
/// request per utterance covers transcription *and* cleanup. The service picks
/// the STT model server-side and handles audio from ~80 ms up to 120 s; the
/// rewrite is best-effort with a ~5 s server-side deadline, so a rewrite
Expand All @@ -23,6 +24,7 @@ public struct AssemblyAITranscriber: TranscriberProtocol {
private let apiKeyProvider: @Sendable () -> String?
private let baseURL: URL
private let transport: any HTTPTransport
private let enhancedTranscriptsEnabled: @Sendable () -> Bool

/// Idle timeout for the transcribe round trip — `URLRequest.timeoutInterval` is
/// reset each time data moves, so this bounds *stalls*, not total elapsed time.
Expand All @@ -33,14 +35,22 @@ public struct AssemblyAITranscriber: TranscriberProtocol {
/// stuck on "Transcribing…" indefinitely.
private static let requestTimeoutSeconds: TimeInterval = 90

/// `enhancedTranscripts` decides, per request, whether the config carries
/// the `llm` cleanup-rewrite block. Read at every `transcribe` so a settings
/// change applies to the next dictation without rebuilding the transcriber.
/// `nil` (the default) reads `EnhancedTranscriptsStore` — spelled as an
/// optional rather than a default closure because a public default argument
/// can't reference the store's internal `isEnabled`.
public init(
apiKeyProvider: @escaping @Sendable () -> String? = { APIKeyStore.current },
baseURL: URL = URL(staticString: "https://dictation.assemblyai.com"),
transport: any HTTPTransport = URLSession.shared
transport: any HTTPTransport = URLSession.shared,
enhancedTranscripts: (@Sendable () -> Bool)? = nil
) {
self.apiKeyProvider = apiKeyProvider
self.baseURL = baseURL
self.transport = transport
self.enhancedTranscriptsEnabled = enhancedTranscripts ?? { EnhancedTranscriptsStore().isEnabled }
}

// MARK: - Dictation request
Expand Down Expand Up @@ -107,16 +117,20 @@ public struct AssemblyAITranscriber: TranscriberProtocol {

/// Builds the JSON `config` part sent alongside the audio. The context
/// `prompt` is included only when non-empty; a nil or blank prompt omits the
/// field so the server applies its default prompt. The `llm` block always
/// rides along — see `DictationConfig.llm`. Internal so tests can assert the
/// field so the server applies its default prompt. The `llm` block rides
/// along while enhanced transcripts are enabled (the default) and is omitted
/// entirely when the user has turned them off, so the service skips the
/// rewrite and the verbatim transcript is what gets pasted — see
/// `DictationConfig.llm`. Internal so tests can assert the
/// prompt wiring without inspecting the multipart upload body (which
/// `URLProtocol` mocks can't observe reliably for `upload(from:)`).
func makeConfigData(sampleRate: Int, prompt: String?) throws -> Data {
try JSONEncoder().encode(
DictationConfig(
sampleRate: sampleRate,
channels: 1,
prompt: prompt.trimmedNonEmpty()
prompt: prompt.trimmedNonEmpty(),
llm: enhancedTranscriptsEnabled() ? LLMRewrite() : nil
)
)
}
Expand Down Expand Up @@ -202,11 +216,14 @@ public struct AssemblyAITranscriber: TranscriberProtocol {
/// it falls back to the server's default prompt. Steers *transcription*;
/// the cleanup rewrite is the `llm` block's job.
let prompt: String?
/// The rewrite request. An empty object selects the service's default
/// The rewrite request, present only while enhanced transcripts are
/// enabled (nil — the synthesized `encode` omits it — asks for no rewrite,
/// so the response's `llm_response` is null and the verbatim `text` is
/// used). An empty object selects the service's default
/// cleanup instruction; per the API's `instruction`-mode rules, output
/// format and don't-answer-the-text safeguards are enforced server-side,
/// so nothing rides along here.
let llm = LLMRewrite()
let llm: LLMRewrite?
enum CodingKeys: String, CodingKey {
case sampleRate = "sample_rate"
case channels
Expand Down
Loading