Stop sending anything about the running app to the LLM - #100
Merged
Conversation
`config.llm.instruction` carried a formatting clause derived from the
frontmost app: `AppKindPriming` recognized the app's bundle ID as a kind
(terminal, code editor, Slack, Obsidian) and sent "Format the result as a
shell command with no trailing period.", "… as Python code." (language
inferred from the window title's filename), and so on.
Remove the whole path rather than just the last link, so nothing is left
dead for periphery to flag:
- `AppKindPriming` and its tests, deleted.
- `TranscriptionSteering.Fields.rewriteInstruction`, gone — `build` now
renders exactly two fields, `conversation_context` and `keyterms_prompt`.
- `LLMRewrite` encodes as `{}`. The `llm` block still rides on every
request (it is what asks the service to rewrite at all), so the service's
own default cleanup instruction now runs for every utterance. That is the
same wire shape an unrecognized app already produced.
- `TranscriptionContext.bundleID` and the `CapturedFocus` bundle-ID capture,
removed — the clause was their only consumer. `windowTitle` stays: it
anchors `KeyInjector`'s same-window paste separator.
- `DictationLog`'s `llm_instruction` column, dropped.
The steering and context test tables keep a case per app family that used
to earn a clause, each pinned to `.empty`, so reintroducing app-kind
steering fails there; the log's "omits every steering field" test pins
`llm_instruction` never being written.
Verified with `scripts/check.sh` (425 unit tests, 23 UI tests, sanitizers,
leaks, periphery, linters) — green apart from a prettier reflow of
AGENTS.md, applied here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR removes all “app-kind” priming so nothing identifying or describing the destination/frontmost app is sent to AssemblyAI’s LLM rewrite path. It simplifies the steering model to only send prior-cursor text (conversation_context) and user key terms (keyterms_prompt), while always requesting the default server-side cleanup rewrite via an empty llm block.
Changes:
- Deleted
AppKindPriming(engine + tests) and removed bundle-ID capture/plumbing (CapturedFocus,TranscriptionContext,DictationSession). - Simplified
TranscriptionSteering.Fieldsto onlyconversationContext+keyterms, and ensuredAssemblyAITranscriberalways encodesllmas{}. - Updated tests and docs to pin “no app-derived steering/logging” behavior (and removed
llm_instructionfrom the dictation log).
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| Tests/BlurtEngineTests/TranscriptionSteeringTests.swift | Updates steering expectations to .empty for app-family contexts; removes instruction-based assertions. |
| Tests/BlurtEngineTests/TranscriptionContextTests.swift | Removes bundleID-driven non-emptiness behavior; adds carry-only cases that must steer nothing. |
| Tests/BlurtEngineTests/DictationLogTests.swift | Removes llm_instruction decoding/assertions; pins that app-identifying strings never land in logs. |
| Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift | Pins llm == {} for multiple steering shapes and removes instruction assertions. |
| Tests/BlurtEngineTests/AppKindPrimingTests.swift | Deletes tests for removed AppKindPriming. |
| Sources/BlurtEngine/STT/TranscriptionSteering.swift | Drops rewrite-instruction field and removes app-kind clause generation. |
| Sources/BlurtEngine/STT/TranscriptionContext.swift | Removes bundleID from captured context and updates invariants/docs accordingly. |
| Sources/BlurtEngine/STT/AssemblyAITranscriber.swift | Always encodes llm as an empty object ({}) and removes instruction wiring. |
| Sources/BlurtEngine/STT/AppKindPriming.swift | Deletes AppKindPriming implementation. |
| Sources/BlurtEngine/Pipeline/DictationSession.swift | Stops capturing/passing bundle IDs into TranscriptionContext. |
| Sources/BlurtEngine/Pipeline/DictationLog.swift | Removes llm_instruction from on-disk JSONL entries. |
| Sources/BlurtEngine/FocusCapture/FocusCapture.swift | Stops capturing the frontmost app’s bundle identifier. |
| README.md | Updates high-level architecture note to remove “format” steering mention. |
| BLURTENGINE.md | Updates engine API/design docs to reflect removal of app-kind formatting and llm_instruction logging. |
| AGENTS.md | Updates repo guide/settled decisions and steering docs to match new privacy posture. |
| .claude/skills/project-guardrails/SKILL.md | Adds/updates guardrail explicitly forbidding sending frontmost-app info. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
53
to
55
| /// Nothing to customize — every field omitted, so the service applies its | ||
| /// managed default prompt and its default cleanup rewrite. Also the value | ||
| /// to compare against for "does this utterance customize anything?" — |
alexkroman
enabled auto-merge
July 29, 2026 18:53
dylan-duan-aai
pushed a commit
to dylan-duan-aai/blurt
that referenced
this pull request
Aug 3, 2026
Rolls the tree back to e08a9ce (v0.1.34) by reverting everything merged since, newest first: - AssemblyAI#101 chore: bump to v0.1.36 - AssemblyAI#100 feat(stt): stop sending anything about the running app to the LLM - AssemblyAI#99 chore: bump to v0.1.35 - AssemblyAI#98 Add app-kind guidance to transcription prompts AssemblyAI#98 and AssemblyAI#100 partly cancelled — AssemblyAI#100 removed the app-kind clause AssemblyAI#98 added — but AssemblyAI#98 carried much more than that, and all of it is undone here. Restored as a result: - `config.prompt` is sent again, built by `TranscriptionPrompt` from the focused app, window, and field plus the user's key terms and the text around the cursor. - `TranscriptionSteering` is gone, and with it the split into `conversation_context`, `keyterms_prompt`, and `llm.instruction`. Key terms are back to a `Keywords: a, b, c.` clause inside the prompt; prior-cursor text is contextual priming in the prompt rather than a conversation turn. - `DictationLog` records the assembled prompt again, alongside the frontmost app's bundle ID. - `AppKindPriming` returns with AssemblyAI#98 and is not re-removed: AssemblyAI#100's removal was reverted too, so the bundle-ID → app-kind table is live again. Docs (AGENTS.md, BLURTENGINE.md, README.md, the project-guardrails skill) go back to describing that design. `CFBundleShortVersionString` is 0.1.34. `git diff e08a9ce` is empty, so this is an exact rollback rather than an approximation. Verified: swift test (424 tests, 67 suites) and the Debug app build both pass. scripts/check.sh as a whole does not pass on this machine — it aborts in the XCUITest suite with the "harness window was not presented" failures that predate all of this work. Co-authored-by: Alex Kroman <alex@assemblyai.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
config.llm.instructioncarried a formatting clause derived from the frontmost app.AppKindPrimingrecognized the app's bundle ID as a kind — terminal, code editor, Slack, Obsidian — and sent a matching imperative:"Format the result as a shell command with no trailing period.""Format the result as Python code."(language inferred from the window title's filename)"Format the result as a casual Slack message, using Slack emoji where they fit.""Format the result as markdown."This removes that entirely. Nothing identifying or describing the destination app reaches the service.
How
The whole chain goes, not just the last link — a partial removal would leave dead code for periphery to fail on:
AppKindPriming+ its testsTranscriptionSteering.Fields.rewriteInstructionbuildnow renders exactly two fields:conversation_context,keyterms_promptTranscriptionContext.bundleIDCapturedFocus.bundleIDNSWorkspace.frontmostApplicationDictationLog'sllm_instructioncolumnLLMRewritenow encodes as{}. Thellmblock still rides on every request — it's what asks the service to run the cleanup rewrite at all — so the service's own default cleanup instruction applies to every utterance. That's the same wire shape an unrecognized app already produced, so this is not a new request shape.windowTitlestays: it anchorsKeyInjector's same-window paste-separator fallback.Regression pins
TranscriptionSteeringTestskeeps a case per app family that used to earn a clause (terminal, code editor with amain.pytitle, Slack, Obsidian), each expecting.empty.TranscriptionContextTestsmoves those contexts into the carry-only set: non-empty (worth carrying for the injector) but steering nothing.AssemblyAITranscriberTestspinsllm == {}for every steering shape, not just the empty one.DictationLogTestspins thatllm_instructionis never written, and that"Terminal"never lands in the log.Docs
AGENTS.md(start-here summary, repo map, settled-decisions table, transcription-steering section),BLURTENGINE.md,README.md, and.claude/skills/project-guardrailsall updated. Both the settled-decisions table and the guardrails skill gain a "never send anything about the frontmost app" row so this doesn't come back.Verification
scripts/check.sh— 425 unit tests / 67 suites, 23 UI tests, TSan + ASan, leaks, xcodegen drift, periphery, swift-format, swiftlint, actionlint, markdownlint, shellcheck: all green. Prettier flagged anAGENTS.mdtable reflow, applied here. (Its two other warnings are gitignored.remember/scratch files, untouched by this change.)🤖 Generated with Claude Code