Skip to content

Stop sending anything about the running app to the LLM - #100

Merged
alexkroman merged 1 commit into
mainfrom
remove-app-context-from-llm
Jul 29, 2026
Merged

Stop sending anything about the running app to the LLM#100
alexkroman merged 1 commit into
mainfrom
remove-app-context-from-llm

Conversation

@alexkroman

Copy link
Copy Markdown
Collaborator

What

config.llm.instruction carried a formatting clause derived from the frontmost app. AppKindPriming recognized the app's bundle ID as a kind — terminal, code editor, Slack, Obsidian — and sent a matching imperative:

  • "Format the result as a shell command with no trailing period."
  • "Format the result as Python code." (language inferred from the window title's filename)
  • "Format the result as a casual Slack message, using Slack emoji where they fit."
  • "Format the result as markdown."

This removes that entirely. Nothing identifying or describing the destination app reaches the service.

How

The whole chain goes, not just the last link — a partial removal would leave dead code for periphery to fail on:

Removed Note
AppKindPriming + its tests the bundle-ID table, extension→language map, and clause wording
TranscriptionSteering.Fields.rewriteInstruction build now renders exactly two fields: conversation_context, keyterms_prompt
TranscriptionContext.bundleID the clause was its only consumer
CapturedFocus.bundleID no longer read from NSWorkspace.frontmostApplication
DictationLog's llm_instruction column

LLMRewrite now encodes as {}. The llm block still rides on every request — it's what asks the service to run the cleanup rewrite at all — so the service's own default cleanup instruction applies to every utterance. That's the same wire shape an unrecognized app already produced, so this is not a new request shape.

windowTitle stays: it anchors KeyInjector's same-window paste-separator fallback.

Regression pins

  • TranscriptionSteeringTests keeps a case per app family that used to earn a clause (terminal, code editor with a main.py title, Slack, Obsidian), each expecting .empty.
  • TranscriptionContextTests moves those contexts into the carry-only set: non-empty (worth carrying for the injector) but steering nothing.
  • AssemblyAITranscriberTests pins llm == {} for every steering shape, not just the empty one.
  • DictationLogTests pins that llm_instruction is never written, and that "Terminal" never lands in the log.

Docs

AGENTS.md (start-here summary, repo map, settled-decisions table, transcription-steering section), BLURTENGINE.md, README.md, and .claude/skills/project-guardrails all updated. Both the settled-decisions table and the guardrails skill gain a "never send anything about the frontmost app" row so this doesn't come back.

Verification

scripts/check.sh — 425 unit tests / 67 suites, 23 UI tests, TSan + ASan, leaks, xcodegen drift, periphery, swift-format, swiftlint, actionlint, markdownlint, shellcheck: all green. Prettier flagged an AGENTS.md table reflow, applied here. (Its two other warnings are gitignored .remember/ scratch files, untouched by this change.)

🤖 Generated with Claude Code

`config.llm.instruction` carried a formatting clause derived from the
frontmost app: `AppKindPriming` recognized the app's bundle ID as a kind
(terminal, code editor, Slack, Obsidian) and sent "Format the result as a
shell command with no trailing period.", "… as Python code." (language
inferred from the window title's filename), and so on.

Remove the whole path rather than just the last link, so nothing is left
dead for periphery to flag:

- `AppKindPriming` and its tests, deleted.
- `TranscriptionSteering.Fields.rewriteInstruction`, gone — `build` now
  renders exactly two fields, `conversation_context` and `keyterms_prompt`.
- `LLMRewrite` encodes as `{}`. The `llm` block still rides on every
  request (it is what asks the service to rewrite at all), so the service's
  own default cleanup instruction now runs for every utterance. That is the
  same wire shape an unrecognized app already produced.
- `TranscriptionContext.bundleID` and the `CapturedFocus` bundle-ID capture,
  removed — the clause was their only consumer. `windowTitle` stays: it
  anchors `KeyInjector`'s same-window paste separator.
- `DictationLog`'s `llm_instruction` column, dropped.

The steering and context test tables keep a case per app family that used
to earn a clause, each pinned to `.empty`, so reintroducing app-kind
steering fails there; the log's "omits every steering field" test pins
`llm_instruction` never being written.

Verified with `scripts/check.sh` (425 unit tests, 23 UI tests, sanitizers,
leaks, periphery, linters) — green apart from a prettier reflow of
AGENTS.md, applied here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 29, 2026 18:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR removes all “app-kind” priming so nothing identifying or describing the destination/frontmost app is sent to AssemblyAI’s LLM rewrite path. It simplifies the steering model to only send prior-cursor text (conversation_context) and user key terms (keyterms_prompt), while always requesting the default server-side cleanup rewrite via an empty llm block.

Changes:

  • Deleted AppKindPriming (engine + tests) and removed bundle-ID capture/plumbing (CapturedFocus, TranscriptionContext, DictationSession).
  • Simplified TranscriptionSteering.Fields to only conversationContext + keyterms, and ensured AssemblyAITranscriber always encodes llm as {}.
  • Updated tests and docs to pin “no app-derived steering/logging” behavior (and removed llm_instruction from the dictation log).

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
Tests/BlurtEngineTests/TranscriptionSteeringTests.swift Updates steering expectations to .empty for app-family contexts; removes instruction-based assertions.
Tests/BlurtEngineTests/TranscriptionContextTests.swift Removes bundleID-driven non-emptiness behavior; adds carry-only cases that must steer nothing.
Tests/BlurtEngineTests/DictationLogTests.swift Removes llm_instruction decoding/assertions; pins that app-identifying strings never land in logs.
Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift Pins llm == {} for multiple steering shapes and removes instruction assertions.
Tests/BlurtEngineTests/AppKindPrimingTests.swift Deletes tests for removed AppKindPriming.
Sources/BlurtEngine/STT/TranscriptionSteering.swift Drops rewrite-instruction field and removes app-kind clause generation.
Sources/BlurtEngine/STT/TranscriptionContext.swift Removes bundleID from captured context and updates invariants/docs accordingly.
Sources/BlurtEngine/STT/AssemblyAITranscriber.swift Always encodes llm as an empty object ({}) and removes instruction wiring.
Sources/BlurtEngine/STT/AppKindPriming.swift Deletes AppKindPriming implementation.
Sources/BlurtEngine/Pipeline/DictationSession.swift Stops capturing/passing bundle IDs into TranscriptionContext.
Sources/BlurtEngine/Pipeline/DictationLog.swift Removes llm_instruction from on-disk JSONL entries.
Sources/BlurtEngine/FocusCapture/FocusCapture.swift Stops capturing the frontmost app’s bundle identifier.
README.md Updates high-level architecture note to remove “format” steering mention.
BLURTENGINE.md Updates engine API/design docs to reflect removal of app-kind formatting and llm_instruction logging.
AGENTS.md Updates repo guide/settled decisions and steering docs to match new privacy posture.
.claude/skills/project-guardrails/SKILL.md Adds/updates guardrail explicitly forbidding sending frontmost-app info.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 53 to 55
/// Nothing to customize — every field omitted, so the service applies its
/// managed default prompt and its default cleanup rewrite. Also the value
/// to compare against for "does this utterance customize anything?" —
@alexkroman
alexkroman enabled auto-merge July 29, 2026 18:53
@alexkroman
alexkroman added this pull request to the merge queue Jul 29, 2026
Merged via the queue into main with commit 7b78eb2 Jul 29, 2026
8 checks passed
@alexkroman
alexkroman deleted the remove-app-context-from-llm branch July 29, 2026 19:12
dylan-duan-aai pushed a commit to dylan-duan-aai/blurt that referenced this pull request Aug 3, 2026
Rolls the tree back to e08a9ce (v0.1.34) by reverting everything merged
since, newest first:

- AssemblyAI#101 chore: bump to v0.1.36
- AssemblyAI#100 feat(stt): stop sending anything about the running app to the LLM
- AssemblyAI#99  chore: bump to v0.1.35
- AssemblyAI#98  Add app-kind guidance to transcription prompts

AssemblyAI#98 and AssemblyAI#100 partly cancelled — AssemblyAI#100 removed the app-kind clause AssemblyAI#98 added —
but AssemblyAI#98 carried much more than that, and all of it is undone here. Restored
as a result:

- `config.prompt` is sent again, built by `TranscriptionPrompt` from the
  focused app, window, and field plus the user's key terms and the text
  around the cursor.
- `TranscriptionSteering` is gone, and with it the split into
  `conversation_context`, `keyterms_prompt`, and `llm.instruction`. Key terms
  are back to a `Keywords: a, b, c.` clause inside the prompt; prior-cursor
  text is contextual priming in the prompt rather than a conversation turn.
- `DictationLog` records the assembled prompt again, alongside the frontmost
  app's bundle ID.
- `AppKindPriming` returns with AssemblyAI#98 and is not re-removed: AssemblyAI#100's removal was
  reverted too, so the bundle-ID → app-kind table is live again.

Docs (AGENTS.md, BLURTENGINE.md, README.md, the project-guardrails skill) go
back to describing that design. `CFBundleShortVersionString` is 0.1.34.

`git diff e08a9ce` is empty, so this is an exact rollback rather than an
approximation.

Verified: swift test (424 tests, 67 suites) and the Debug app build both
pass. scripts/check.sh as a whole does not pass on this machine — it aborts
in the XCUITest suite with the "harness window was not presented" failures
that predate all of this work.

Co-authored-by: Alex Kroman <alex@assemblyai.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants