feat(mascot): locale-aware voice + multilingual LLM replies#2115
Merged
senamakel merged 7 commits intoMay 18, 2026
Merged
Conversation
Mascot voice settings now expose gender, ElevenLabs voice picker, and a "match the app language" toggle backed by a per-locale default voice map. Default voice is George (`JBFqnCBsd6RMkjVDRZzb`, multilingual) and all TTS requests now default to `eleven_multilingual_v2` so non-Latin scripts render correctly. Frontend chat send now carries the active UI locale to the core. In the web channel, a new `locale` RPC param threads through `start_chat` and `run_chat_task` into `build_session_agent`, which composes a "Respond in <language>" directive on top of the profile's existing system-prompt suffix for non-English locales.
Contributor
📝 WalkthroughWalkthroughMoves mascot voice controls from VoicePanel to MascotPanel, adds gender and locale-aware defaults and selectors, ensures frontend passes UI locale to chat RPCs, and composes locale-driven reply directives in the backend agent builder. ChangesMascot Voice and Chat Locale Integration
Sequence Diagram(s)sequenceDiagram
participant User as User (UI)
participant Conversations as Conversations.tsx
participant chatService as chatService
participant SocketIO as Socket.IO
participant WebProvider as Web Provider
participant SessionAgent as SessionAgent
User->>Conversations: send chat message
Conversations->>Conversations: uiLocale = locale.current ?? 'en'
Conversations->>chatService: chatSend({message, locale: uiLocale})
chatService->>SocketIO: channel_web_chat RPC (message, locale)
SocketIO->>WebProvider: start_chat(message, locale)
WebProvider->>SessionAgent: build_session_agent(locale)
SessionAgent->>SessionAgent: directive = locale_reply_directive(locale)
SessionAgent->>SessionAgent: suffix = compose_system_prompt_suffix(directive, profile_suffix)
SessionAgent->>SessionAgent: Agent configured with locale prompt
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
ESLint skipped: no ESLint configuration detected in root package.json. To enable, add Comment |
coderabbitai
Bot
added
the
feature
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (4)
src/openhuman/channels/providers/web.rs (2)
47-72: ⚖️ Poor tradeoffLocale omitted from session cache fingerprint — documented but worth tracking.
The
SessionCacheFingerprintdoesn't includelocale, so mid-conversation locale changes won't take effect until the session rebuilds for another reason. The PR documents this as intentional. Consider opening a follow-up issue to addlocaleto the fingerprint if user testing shows this causes confusion when switching app language.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/openhuman/channels/providers/web.rs` around lines 47 - 72, SessionCacheFingerprint currently omits locale, so add a new field (e.g., locale: Option<String> or String) to the SessionCacheFingerprint struct and include it wherever fingerprints are constructed/compared so locale changes invalidate the cache; update the struct declaration (SessionCacheFingerprint) and every place that builds instances of it (call sites that set model_override, temperature, target_agent_id, provider_binding) to populate the new locale field and ensure equality/clone behavior still works.
1672-1701: 💤 Low valueConsider adding a locale coverage test to prevent future drift.
The frontend
Localetype (app/src/lib/i18n/types.ts) and backendlocale_reply_directivemapping are currently in sync, but a test comparing both lists would help catch silent divergence as the UI adds new locales.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/openhuman/channels/providers/web.rs` around lines 1672 - 1701, Add a unit/integration test that asserts the backend locale mapping in the function locale_reply_directive covers every locale declared in the frontend Locale type; implement a test (e.g. test_locale_coverage_for_locale_reply_directive) that loads/parses the frontend i18n/types.ts Locale variants (or imports a canonical list if one is exported) and then verifies each frontend tag either maps to Some(...) by calling locale_reply_directive or is intentionally absent with a documented exception list; fail the test if any frontend locale is not accounted for so the backend mapping and the frontend Locale remain in sync.app/src/features/human/useHumanMascot.test.ts (1)
547-547: ⚡ Quick winUpdate test name to match the new assertion.
The test name says "omits the voice override" but the assertion now expects a voiceId to be present (
'JBFqnCBsd6RMkjVDRZzb'). The comment at lines 565-567 correctly explains the new behavior: the selector resolves the build-time default eagerly.📝 Proposed test name update
- it('omits the voice override when no preference is stored', async () => { + it('uses the default mascot voice when no preference is stored', async () => { mockMascotVoiceId = null;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@app/src/features/human/useHumanMascot.test.ts` at line 547, Update the test case name string for the it(...) block currently labelled "omits the voice override when no preference is stored" to reflect that the selector now resolves the build-time default eagerly (e.g., "defaults to build-time voice when no preference is stored"); keep the assertion that expects voiceId 'JBFqnCBsd6RMkjVDRZzb' unchanged so the test name matches the new behavior described in the surrounding comments and assertions.app/src/features/human/voice/ttsClient.test.ts (1)
36-36: ⚡ Quick winImport and use
MASCOT_VOICE_IDandMASCOT_VOICE_MODEL_IDconstants fromconfig.ts.The test hard-codes the voice ID
'JBFqnCBsd6RMkjVDRZzb'and model ID'eleven_multilingual_v2', which are already exported as constants inapp/src/utils/config.ts. Since the test name explicitly states it "falls back to the configured mascot voice + multilingual model", using the actual config constants would keep the test aligned with the real configuration and avoid brittleness if defaults change.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@app/src/features/human/voice/ttsClient.test.ts` at line 36, The test currently hard-codes the mascot voice and model values; import MASCOT_VOICE_ID and MASCOT_VOICE_MODEL_ID from the config module (the constants exported from app/src/utils/config.ts) into ttsClient.test.ts and replace the literal strings ('JBFqnCBsd6RMkjVDRZzb' and 'eleven_multilingual_v2') in the params object with those constants so the test uses the configured mascot voice and multilingual model; ensure the import is added at the top and the params line uses MASCOT_VOICE_ID and MASCOT_VOICE_MODEL_ID.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@app/src/components/settings/panels/MascotPanel.tsx`:
- Around line 120-123: The useEffect in MascotPanel currently calls
setVoiceDraft and setVoicePreviewError synchronously; change this to schedule
those updates asynchronously to satisfy the react-hooks/set-state-in-effect rule
— e.g., inside the useEffect that depends on storedVoiceId, capture const newId
= storedVoiceId ?? '' and then call the setters inside a microtask or timeout
(queueMicrotask(() => setVoiceDraft(newId)); queueMicrotask(() =>
setVoicePreviewError(null)); or setTimeout(..., 0)), ensuring you still
reference setVoiceDraft and setVoicePreviewError and retain the same dependency
on storedVoiceId.
---
Nitpick comments:
In `@app/src/features/human/useHumanMascot.test.ts`:
- Line 547: Update the test case name string for the it(...) block currently
labelled "omits the voice override when no preference is stored" to reflect that
the selector now resolves the build-time default eagerly (e.g., "defaults to
build-time voice when no preference is stored"); keep the assertion that expects
voiceId 'JBFqnCBsd6RMkjVDRZzb' unchanged so the test name matches the new
behavior described in the surrounding comments and assertions.
In `@app/src/features/human/voice/ttsClient.test.ts`:
- Line 36: The test currently hard-codes the mascot voice and model values;
import MASCOT_VOICE_ID and MASCOT_VOICE_MODEL_ID from the config module (the
constants exported from app/src/utils/config.ts) into ttsClient.test.ts and
replace the literal strings ('JBFqnCBsd6RMkjVDRZzb' and
'eleven_multilingual_v2') in the params object with those constants so the test
uses the configured mascot voice and multilingual model; ensure the import is
added at the top and the params line uses MASCOT_VOICE_ID and
MASCOT_VOICE_MODEL_ID.
In `@src/openhuman/channels/providers/web.rs`:
- Around line 47-72: SessionCacheFingerprint currently omits locale, so add a
new field (e.g., locale: Option<String> or String) to the
SessionCacheFingerprint struct and include it wherever fingerprints are
constructed/compared so locale changes invalidate the cache; update the struct
declaration (SessionCacheFingerprint) and every place that builds instances of
it (call sites that set model_override, temperature, target_agent_id,
provider_binding) to populate the new locale field and ensure equality/clone
behavior still works.
- Around line 1672-1701: Add a unit/integration test that asserts the backend
locale mapping in the function locale_reply_directive covers every locale
declared in the frontend Locale type; implement a test (e.g.
test_locale_coverage_for_locale_reply_directive) that loads/parses the frontend
i18n/types.ts Locale variants (or imports a canonical list if one is exported)
and then verifies each frontend tag either maps to Some(...) by calling
locale_reply_directive or is intentionally absent with a documented exception
list; fail the test if any frontend locale is not accounted for so the backend
mapping and the frontend Locale remain in sync.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 35121176-3b13-41ac-9f48-1256586fb689
📒 Files selected for processing (20)
app/src/components/settings/panels/MascotPanel.tsxapp/src/components/settings/panels/VoicePanel.tsxapp/src/components/settings/panels/__tests__/VoicePanel.test.tsxapp/src/components/settings/panels/elevenlabsVoicePresets.tsapp/src/features/human/useHumanMascot.test.tsapp/src/features/human/useHumanMascot.tsapp/src/features/human/voice/ttsClient.test.tsapp/src/features/human/voice/ttsClient.tsapp/src/lib/i18n/en.tsapp/src/pages/Conversations.tsxapp/src/pages/__tests__/Conversations.render.test.tsxapp/src/services/chatService.tsapp/src/services/webviewAccountService.tsapp/src/store/mascotSlice.tsapp/src/test/setup.tsapp/src/utils/config.tssrc/core/socketio.rssrc/openhuman/channels/bus.rssrc/openhuman/channels/providers/web.rssrc/openhuman/channels/providers/web_tests.rs
💤 Files with no reviewable changes (1)
- app/src/components/settings/panels/tests/VoicePanel.test.tsx
The new `settings.mascot.voice.*` keys landed in `en.ts` and `en-5.ts` but missed every per-locale chunk-5 file, so `pnpm i18n:check` reported 17 missing keys × 10 non-English locales. Translations added for ar / bn / es / fr / hi / id / it / pt / ru / zh-CN; runtime still falls back to English for any locale that ends up out of date.
Inline the draft + preview-error resets into the handlers that also dispatch `setMascotVoiceId(...)`, dropping the effect-based mirror flagged by `react-hooks/set-state-in-effect` (CodeRabbit review). All writes flow through this component, so an effect to "watch the slice" was redundant — handlers see the change before dispatch.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@app/src/components/settings/panels/MascotPanel.tsx`:
- Around line 117-126: The cleanup only stops existing audio but doesn't prevent
a pending synthesizeSpeech(...) from creating new Audio and updating state after
unmount; add a cancellation guard (e.g., a local isCancelled ref or
currentPreviewId token) that is set in the useEffect cleanup and checked before
creating/playing the Audio and before calling setState in the synthesizeSpeech
completion path (references: previewAudioRef, the useEffect cleanup block around
lines 117-126, and the synthesizeSpeech handler around lines 197-218); ensure
you also stop/clear any created Audio only when not cancelled and avoid calling
setPreviewing/setPreviewAudio after cancellation.
- Around line 154-156: The visiblePresets filter drops a curated preset that's
currently selected (effectiveVoiceId) when voiceGender changes, causing the
controlled <select> (value tied to effectiveVoiceId) to have no matching
<option>; update the filtering logic that builds visiblePresets from
ELEVENLABS_VOICE_PRESETS (and the similar occurrences) to also include any
preset whose id equals effectiveVoiceId regardless of gender or locales, e.g.
keep p if p.gender===voiceGender || p.locales.includes('*') ||
p.id===effectiveVoiceId so the active curated voice remains in the rendered
options.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 526fc0f9-8338-4914-9609-7a2e03235201
📒 Files selected for processing (1)
app/src/components/settings/panels/MascotPanel.tsx
CodeRabbit follow-up: - Add `previewRequestIdRef` so a `synthesizeSpeech` that resolves after unmount or another preview click bails before touching refs / state. The earlier audio-only cleanup couldn't catch that case. - Include the currently-selected preset in `visiblePresets` even when it doesn't match the active gender filter, so flipping gender can't leave the controlled `<select>` pointing at an id with no matching `<option>`.
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
app/src/components/settings/panels/MascotPanel.tsx (1)
224-226: 💤 Low valueConsider localizing the voice preview text.
The preview string is hardcoded in English. Since this PR introduces locale-aware voice behavior and uses
eleven_multilingual_v2, users may benefit from hearing the voice in their actual language to better evaluate it.💡 Suggested change
- const tts = await synthesizeSpeech("Hi, I'm your assistant. This is a voice preview.", { + const tts = await synthesizeSpeech(t('settings.mascot.voice.previewText'), { voiceId: effectiveVoiceId, });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@app/src/components/settings/panels/MascotPanel.tsx` around lines 224 - 226, The preview text is hardcoded in English; replace it with a localized string and pass that into synthesizeSpeech so users hear the preview in their locale. Locate the call to synthesizeSpeech in MascotPanel (the line using effectiveVoiceId) and retrieve the translated preview using your app's i18n utility (e.g., t('mascot.voicePreview') or useTranslation()/useLocale() helper), add a fallback string if translation is missing, then call synthesizeSpeech(localizedPreview, { voiceId: effectiveVoiceId }); also add the new translation key "mascot.voicePreview" to your locale files.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@app/src/components/settings/panels/MascotPanel.tsx`:
- Around line 224-226: The preview text is hardcoded in English; replace it with
a localized string and pass that into synthesizeSpeech so users hear the preview
in their locale. Locate the call to synthesizeSpeech in MascotPanel (the line
using effectiveVoiceId) and retrieve the translated preview using your app's
i18n utility (e.g., t('mascot.voicePreview') or useTranslation()/useLocale()
helper), add a fallback string if translation is missing, then call
synthesizeSpeech(localizedPreview, { voiceId: effectiveVoiceId }); also add the
new translation key "mascot.voicePreview" to your locale files.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: e87eba66-681e-4f85-b819-3ca8da8d42cc
📒 Files selected for processing (1)
app/src/components/settings/panels/MascotPanel.tsx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
JBFqnCBsd6RMkjVDRZzb), a multilingual ElevenLabs voice, and all TTS requests now default toeleven_multilingual_v2so non-Latin scripts render correctly."Respond in <language>"directive (non-English locales only).VoicePanelintoMascotPanel(single source of truth);VoicePanelkeeps a one-line pointer to the new location.Problem
The mascot was speaking English even when a user picked Arabic / Hindi / etc. in the UI. Two underlying issues:
There was no plumbing from the frontend's
localeSliceinto the Rust core, and the mascot voice picker lived under Voice settings (alongside Piper/Whisper, which confused which provider an id belonged to).Solution
Frontend (
app/)mascotSlice: addedvoiceGenderandvoiceUseLocaleDefault(both REHYDRATE-guarded); newselectEffectiveMascotVoiceId(state)resolves locale-default → manual override → build-time default so the UI and TTS path can never drift.MascotPanel.tsx: new Voice section — gender radio, preset dropdown filtered by gender, custom paste, locale-default checkbox showing the live<locale> → <voiceId>mapping, preview button, reset.elevenlabsVoicePresets.ts: each preset now declareslocales; addedDEFAULT_VOICE_BY_LOCALE[locale][gender]map +defaultVoiceIdForLocale()helper.ttsClient.ts: defaultsmodel_idtoeleven_multilingual_v2so non-English glyphs render natively.chatService.chatSend: new optionallocaleparam sent aslocalein the RPC payload;Conversations.tsxandwebviewAccountService.tsread the active locale from redux and pass it through.Rust core (
src/openhuman/channels/providers/web.rs)WebChatParams,channel_web_chat,start_chat,run_chat_task,build_session_agentnow threadlocale: Option<String>.locale_reply_directive(locale)maps BCP-47 →"Respond in <Language>"(no-op for English / unknown tags);compose_system_prompt_suffix()stitches that directive in front of the profile's own suffix before handing to the agent builder.localeinput.socketio.rs+channels/bus.rsupdated for the new signature.Submission Checklist
web_tests.rslocale helpers;mascotSliceselectors;MascotPanel,ttsClient,useHumanMascot,Conversationsrender tests). Run locally withpnpm test:coverageandpnpm test:rust.docs/TEST-COVERAGE-MATRIX.md.## Related.docs/RELEASE-MANUAL-SMOKE.md.Impact
eleven_multilingual_v2andeleven_monolingual_v1are priced the same on ElevenLabs, so TTS billing is unchanged.locale, so a thread that already has a warm agent keeps its current language until rebuilt (new threads pick up the new locale immediately). Documented; intentional for this PR.voiceUseLocaleDefaultdefaults tofalse,voiceGenderdefaults to'male'(matching the new default voice), and pre-Add configurable mascot voices with ElevenLabs voice IDs #1762 persisted blobs hydrate cleanly via the REHYDRATE guards.Related
localetoSessionCacheFingerprintso mid-thread locale changes invalidate the cached agent.AI Authored PR Metadata (required for Codex/Linear PRs)
Linear Issue
Commit & Branch
Validation Run
Validation Blocked
Behavior Changes
Parity Contract
Duplicate / Superseded PR Handling
Summary by CodeRabbit
New Features
Improvements