You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Plugin hooks for speech: dictation engines, read-aloud and hands-free voice mode
#720
Some people need local or offline speech: a self-hosted Whisper-compatible server, or the speech engine on a phone.
CONTRIBUTING says optional capabilities belong in plugins. I agree that speech should be a plugin, not core. I'd like to discuss which extension points that would need.
I built all of this in my fork (links below) and use it every day on desktop and Android:
dictation through a local OpenAI-compatible /audio/transcriptions endpoint;
read-aloud with a per-reply speaker button, auto-read while a reply streams, maths spoken as words, and a speed control;
a hands-free voice mode: on the phone you talk, short voice commands send, reveal or move on, and the reply is read back.
It's written as core code today, which is why I'm asking about hooks rather than opening a PR.
What the current plugin API already covers
ctx.ui dictation widget, registerSettings, placeAction (sidebar, graph, top bar) and plugin shortcuts.
Backend handle() / frontend call() to reach a TTS or STT server without exposing keys to the page.
turn-settled events and the top-bar context.view snapshot.
What is missing
Shapes below are sketches to make the request concrete, not a fixed design.
1. Dictation engines
A plugin registers a speech-to-text engine. The host keeps everything it already owns: microphone capture, the level meter, the one-recording-per-window rule, Escape, the dictation bar and where the text goes.
ctx.registerDictationEngine({id: "local-whisper",name: "Local (Whisper-compatible)",status(): Promise<{ready: boolean;problem?: string}>,// e.g. "needs a URL", shown in the bartranscribe(audio: Blob/* 16 kHz mono WAV */,opts: {signal: AbortSignal}): Promise<string>,});
Settings → Dictation would list Deepgram plus the registered engines. A batch engine (one upload at stop) covers local servers and phone engines. Streaming engines could come later.
2. Live session view
Auto-read speaks a reply block by block while it streams. Today a plugin only sees context.view when the learner clicks its action. A subscription would fix that:
ctx.onSessionView((view: PluginSessionViewContext|null)=>{ ... });// the open session; null when none
Read-aloud also needs the open question with its options, which view.input may already carry.
3. Two more action placements
location: "turn": a small action under each assistant reply, called with { sessionId, messageIds }. This holds the speaker button.
location: "composer": a compact slot next to the dictation control, for toggles such as auto-read and voice mode. A widget mount here would also allow a speed control.
4. Audio coordination
Only one thing should make or take sound at a time. Today the core does this for dictation alone. A plugin that plays audio needs:
ctx.registerPlayback({active(): boolean,// Escape stops playback first while this is truestop(): void,// called when dictation starts or the learner sends});ctx.onDictation((phase: "idle"|"recording"|"stopping")=>{ ... });
5. Driving the open input (voice mode)
Voice mode turns each utterance into either text or a command. It needs the host's own input actions, with the host's usual checks (busy, compaction, nothing to send):
ctx.input.append(text: string): boolean;// into whichever input owns dictation: composer, question, practicectx.input.submit(): boolean;// the same as pressing Send therectx.input.reveal(): boolean;// open a check waiting behind its revealctx.input.next(): boolean;// move on between practice problems
Each returns false when the host refuses, so the plugin can give feedback. Hooks 1–4 are useful on their own, and 5 is only needed for hands-free use.
Alternatives
Read-aloud and voice mode in core. It's simpler for users, but it adds a lot of optional code to the core. The provider choice (Text-to-Speech or Voice Output Support #607 wants ElevenLabs, others want local engines) fits a plugin better.
A plugin on today's API only. It can't add a speaker button per reply, can't read while a reply streams, and can't stop when dictation starts.
Reference implementation
Working code in my fork. Today it's core code, which is what I'd move into a plugin:
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Use case
Several people want LearnHarness to speak and listen beyond the built-in Deepgram dictation:
CONTRIBUTING says optional capabilities belong in plugins. I agree that speech should be a plugin, not core. I'd like to discuss which extension points that would need.
I built all of this in my fork (links below) and use it every day on desktop and Android:
/audio/transcriptionsendpoint;It's written as core code today, which is why I'm asking about hooks rather than opening a PR.
What the current plugin API already covers
ctx.uidictation widget,registerSettings,placeAction(sidebar, graph, top bar) and plugin shortcuts.handle()/ frontendcall()to reach a TTS or STT server without exposing keys to the page.turn-settledevents and the top-barcontext.viewsnapshot.What is missing
Shapes below are sketches to make the request concrete, not a fixed design.
1. Dictation engines
A plugin registers a speech-to-text engine. The host keeps everything it already owns: microphone capture, the level meter, the one-recording-per-window rule, Escape, the dictation bar and where the text goes.
Settings → Dictation would list Deepgram plus the registered engines. A batch engine (one upload at stop) covers local servers and phone engines. Streaming engines could come later.
2. Live session view
Auto-read speaks a reply block by block while it streams. Today a plugin only sees
context.viewwhen the learner clicks its action. A subscription would fix that:Read-aloud also needs the open question with its options, which
view.inputmay already carry.3. Two more action placements
location: "turn": a small action under each assistant reply, called with{ sessionId, messageIds }. This holds the speaker button.location: "composer": a compact slot next to the dictation control, for toggles such as auto-read and voice mode. A widget mount here would also allow a speed control.4. Audio coordination
Only one thing should make or take sound at a time. Today the core does this for dictation alone. A plugin that plays audio needs:
5. Driving the open input (voice mode)
Voice mode turns each utterance into either text or a command. It needs the host's own input actions, with the host's usual checks (busy, compaction, nothing to send):
Each returns
falsewhen the host refuses, so the plugin can give feedback. Hooks 1–4 are useful on their own, and 5 is only needed for hands-free use.Alternatives
Reference implementation
Working code in my fork. Today it's core code, which is what I'd move into a plugin:
I'm happy to implement the agreed hooks as separate PRs, one hook per PR, starting with whichever you prefer.
All reactions