Voice spike for my-words, using OpenAI Realtime

[kcarnold]

A voice-native conversation mode for the My Words lab page, over OpenAI Realtime (WebRTC). The writer talks; the agent shapes their words in the real document.

Supersedes the old description ("doesn't obey the own-words validation, it's just a demo"). That is no longer true — see below.

The own-words constraint is enforced, including for voice

corpus.ts is the rule: the AI may edit the document but may never originate words. Anything it places must be lifted verbatim from the writer's corpus (document + scratchpad + their own chat/voice messages), joined only by punctuation and a closed set of glue words. Glue deliberately excludes content connectives ("because", "however") and the copula, which would let the model assert relationships the writer never made.

The voice path runs every tool call through the same validateOp against the live corpus, and applies it through the same durable edit path as the text tabs. The one voice-specific piece: the writer's rolling transcript feeds the corpus's userMessages slot, so words they just said aloud are shapeable — without it validateOp would reject the model for using them.

Control invariants, enforced in code

Five, in voice/session.ts, each because the writer's sense of control fails a specific way without it — covered by __tests__/voiceControl.test.ts:

1. One applied edit per writer turn. The budget resets when the writer starts speaking, not on model-response boundaries (this session requests a follow-up after every tool result, so a per-response budget would reset immediately and gate nothing). Only a successful apply consumes it, so a rejected edit stays retryable.
2. Speech during the veto window cancels the pending edit. The instinctive "no, wait" has to work; ~750ms is too short to reach for the ✕ chip. Implemented by holding the window's cancel rather than setting a flag, which would leak past the window and kill the next edit.
3. No silent failure. Every path that ends without the edit landing emits exactly one writer-visible notice. Word-bank rejections happen for mundane reasons (a mis-transcribed word), and silence reads as broken rather than principled.
4. Undo inverses captured at apply time, never re-derived, and freshness-checked, so a region the writer has since hand-edited is refused rather than clobbered.
5. Highlighting is not an edit — never costs or is blocked by the turn budget, and carries no veto beat.

These are enforced in code rather than asked for in the prompt; prompt.ts spends its words on stance.

Shape

• Six tools (view, str_replace, insert, move, highlight, undo), each taking target: 'document' | 'scratchpad'.
• Scratchpad — a second surface holding the writer's ideas in their own words, which the agent edits with the same tools and the same validation. Persisted via EditorAPI's document-setting pair, so it travels with the file in Word.
• Transport seam — session.ts is transport-agnostic; realtime.ts implements VoiceTransport over OpenAI Realtime. realtime.ts is deliberately not unit-tested (every bug it has had was an environment fact a mock reproduces wrongly by construction); the logic worth testing lives above it.
• Backend — one route, POST /api/openai/realtime/session, minting an ephemeral client secret. Audio and tool calls run browser → OpenAI directly.
• Demo harness — mywords-demo.html with scripted responders and scenarios, so the interaction can be exercised without a live model. Playwright spec included.

Status: experiment WIP, not ready to enable

The page ships behind a flag (enabled: () => isFlagEnabled('my-words')), so it's out of even the Labs menu. Reach it with ?ff=my-words on the standalone editor or dev server.

The blocker for turning it on is metering. Voice usage does not reach the llm_usage table, and can't be captured at the proxy layer: audio runs browser → OpenAI over WebRTC, so no tokens pass through our server to tee(). Doing it properly means reading usage off the client's response.done events and reporting them back. A voice session is billable but invisible to /api/usage_summary until then.

Fixed in this branch along the way: the realtime mint route was unauthenticated and called openaiApiKey() directly, bypassing attributeRequest — so anyone who could reach the server could mint a credential against the main key, and it skipped both demo-key routing and the fail-closed 401. It now runs the same gauntlet as a proxied generation (beta allowlist → 403, attribution → 401), and the client authorizes via the shared authorizedFetch.

Merge notes

Merged main and moved the scratchpad onto the getDocumentSetting/setDocumentSetting layer main introduced, deleting the bespoke loadScratchpad/saveScratchpad pair (Word's implementation was a literal copy of the generic one with the key hardcoded). The setting key is unchanged, so existing Word documents keep their scratchpads; browser-backed hosts reset, because those keys are now namespaced per document — previously the standalone editor shared one scratchpad across every document on the origin.

Also in the merge: both branches had widened EditorAPI independently, so each side's inline test mocks failed to satisfy the other's additions — added a shared noopEditorAPI fixture. And this branch's ai ^5→^7 bump outdated main's new generate.test.ts (MockLanguageModelV2 is gone; the v3 spec restructured finishReason and usage). Production is unaffected — ai v7 adapts v2 models via asLanguageModelV3.

Testing

• Frontend: 200/204 vitest (63 in my-words), tsc clean, eslint . 0 errors, build succeeds
• Backend: 106/106 vitest, tsc clean, build succeeds
• The 4 failures are pre-existing on main, not from this branch: documentSettings.test.ts fails under Node 26, whose built-in localStorage global shadows jsdom's, so the @vitest-environment jsdom pragma doesn't take. Verified by running them on a pristine origin/main worktree. Backend CLAUDE.md already specifies Node 24 LTS.

Design docs

docs/my-words-voice-native-research.md, docs/my-words-voice-scratchpad.md, docs/my-words-interaction-design.md.