Skip to content

fix(#14373): make cloud Whisper STT model deploy-configurable (un-hardcode English-only)#14511

Merged
lalalune merged 2 commits into
developfrom
fix/14373-configurable-stt-model
Jul 6, 2026
Merged

fix(#14373): make cloud Whisper STT model deploy-configurable (un-hardcode English-only)#14511
lalalune merged 2 commits into
developfrom
fix/14373-configurable-stt-model

Conversation

@lalalune

@lalalune lalalune commented Jul 6, 2026

Copy link
Copy Markdown
Member

Part of #14373. This PR removes the code-level English-only hardcode and misleading plugin knob/log; #14373 should stay open until the live Railway Spanish audio contract is run with provisioned evidence.

Problem

The cloud /api/v1/voice/stt self-hosted Whisper path hardcoded Systran/faster-whisper-tiny.en while also forwarding languageCode. That meant non-English speech could silently go through an English-only model. Separately, plugin-elizacloud logged and documented ELIZAOS_CLOUD_TRANSCRIPTION_MODEL=gpt-5-mini-transcribe, but that setting is not honored by this Cloud route.

Change

  • Add optional WHISPER_STT_MODEL Worker binding and resolve it through resolveWhisperSttModel.
  • Default the Worker route to multilingual Systran/faster-whisper-small, so unset config is no longer English-only.
  • Keep deploy-level override support for model pinning; no per-request model override is added.
  • Route completion logs now name the actual Whisper model sent to the Railway service.
  • Remove the dead ELIZAOS_CLOUD_TRANSCRIPTION_MODEL plugin setting, model-registry validation entry, docs row, and misleading Using TRANSCRIPTION model: gpt-5-mini-transcribe log.
  • Extend the live Railway contract to use the same model resolver and add a gated Spanish clip assertion via ELIZA_VOICE_LIVE_SPANISH_AUDIO_URL / ELIZA_VOICE_LIVE_SPANISH_EXPECT.

Local Proof

bun test packages/cloud/api/v1/voice/stt/whisper-model.test.ts packages/cloud/api/__tests__/voice-kokoro-whisper-live.test.ts
  4 passed, 2 skipped
  skipped: live Railway English and Spanish paths require ELIZA_VOICE_LIVE_RAILWAY=1 and, for Spanish, ELIZA_VOICE_LIVE_SPANISH_AUDIO_URL

bunx @biomejs/biome check --write packages/cloud/api/v1/voice/stt/route.ts packages/cloud/api/v1/voice/stt/whisper-model.ts packages/cloud/api/v1/voice/stt/whisper-model.test.ts packages/cloud/api/__tests__/voice-kokoro-whisper-live.test.ts packages/cloud/shared/src/types/cloud-worker-env.ts plugins/plugin-elizacloud/src/models/transcription.ts plugins/plugin-elizacloud/src/services/cloud-model-registry.ts plugins/plugin-elizacloud/src/utils/config.ts plugins/plugin-elizacloud/src/index.ts plugins/plugin-elizacloud/README.md plugins/plugin-elizacloud/CLAUDE.md plugins/plugin-elizacloud/AGENTS.md
  clean

rg "ELIZAOS_CLOUD_TRANSCRIPTION_MODEL|Using TRANSCRIPTION model|faster-whisper-tiny\\.en" packages/cloud plugins/plugin-elizacloud -g '*.{ts,md}'
  no plugin dead-knob/log hits; no English-only default remains in the Cloud route/helper

Remaining Evidence Before Closing #14373

The live Spanish round-trip is now encoded as a gated contract test, but it was not run in this host because the Spanish audio fixture URL and live Railway credentials/config were not available. The issue should close only after a run like this is attached inline:

ELIZA_VOICE_LIVE_RAILWAY=1 \
ELIZA_VOICE_LIVE_SPANISH_AUDIO_URL=<stable Spanish WAV/WEBM> \
ELIZA_VOICE_LIVE_SPANISH_EXPECT=recordatorio \
bun test packages/cloud/api/__tests__/voice-kokoro-whisper-live.test.ts
  • Before screenshots: N/A - backend STT route/plugin config change; no UI surface changed.
  • After screenshots: N/A - backend STT route/plugin config change; no UI surface changed.
  • Walkthrough video: N/A - backend STT route/plugin config change; live audio contract logs are the relevant artifact.
  • Backend logs: N/A - live Railway English/Spanish backend logs require ELIZA_VOICE_LIVE_RAILWAY=1 plus a stable Spanish audio fixture on this host; the gated test command is documented above and #14373 remains open until that evidence is attached.
  • Frontend console/network logs: N/A - no frontend runtime path changed.
  • Real-LLM trajectory: N/A - STT backend config only; no agent prompt/planner/model-generation path changed.
  • Domain artifacts: N/A - no DB/memory/scheduled-task/wallet/file artifact is produced by model selection config.

The self-hosted Whisper STT path hardcoded the English-only
`Systran/faster-whisper-tiny.en`, so non-English speech (the persona corpus
spans 10+ languages, and the request already forwards `languageCode`) was
transcribed by an English-only model. Add `WHISPER_STT_MODEL` (optional; same
default) resolved by a small pure `resolveWhisperSttModel` helper, and name the
actual model in the completion log instead of leaving it implicit.

whisper-model.test.ts: 4/4, 100% line/func coverage (env override, English-only
default, whitespace trim, blank-degrades-to-default). The non-English round-trip
itself is a live-Railway assertion (voice-kokoro-whisper-live.test.ts).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@lalalune lalalune left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed by inspection (own-account, commenting). Clean, well-scoped, honest. Removes the hardcoded English-only Systran/faster-whisper-tiny.en from the cloud /api/v1/voice/stt self-hosted-Whisper path (#14373) — adds an optional WHISPER_STT_MODEL binding (cloud-worker-env.ts, documented) resolved via a pure resolveWhisperSttModel() (env override → English-only default → whitespace-degrades-to-default guard), and names the model in the completion log so ops can tell which ran. The model swap is now a config change, not a code change. 37-line unit test covers all three resolution paths. Default unchanged so zero-config deployments are untouched. Honest evidence scoping: the live non-English round-trip is correctly marked N/A here — it needs a provisioned Railway multilingual Whisper service + real clip, and that contract already lives in voice-kokoro-whisper-live.test.ts (gated on WHISPER_STT_URL); the service provisioning is #14374. This ships the tractable, testable plumbing without faking the live multilingual verification. Note the multilingual model isn't the DEFAULT yet (that's the deploy/#14374 step) — the hardcode blocker is removed and it's now deployable. LGTM.

@lalalune lalalune merged commit 0fe1192 into develop Jul 6, 2026
14 of 25 checks passed
@lalalune lalalune deleted the fix/14373-configurable-stt-model branch July 6, 2026 00:08
@claude

claude Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Claude encountered an error —— View job


I'll analyze this and get back to you.

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

❌ PR title does not match the required pattern. Please use one of these formats:

  • 'type: description' (e.g., 'feat: add new feature')
  • 'type(scope): description' (e.g., 'chore(core): update dependencies')
    Valid types: feat, fix, docs, style, refactor, perf, test, build, ci, chore, revert, release

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant