desktop: multi-language voice assistant — per-turn PTT language ID, provider hints, misdetect-proof chat bubbles#8948
Merged
Merged
Conversation
…ints)
Users can now select every language they speak (onboarding step +
Settings → Transcription card, primary first). Push-to-talk identifies
the spoken language per turn on-device (Parakeet v3 partial decode
during the hold + NLLanguageRecognizer, zero added latency at commit):
- OpenAI realtime: whisper input transcription gets an explicit
language hint before each commit
- Gemini Live (no language param exists): user languages are pinned in
the system instruction, and when the provider transcript comes back
in a language outside the user's set, the saved chat bubble falls
back to the on-device transcript (fixes Russian speech being saved
as an Italian bubble)
- fixes onboarding language normalization saving the literal typed
word ("russian") instead of the ISO code ("ru"), which also pinned
the ambient transcriber to a bogus single language; the language
step no longer touches ambient transcription settings at all
- adds `ptt_test_turn` automation action driving the real controller
turn path headlessly for E2E verification
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes from the multi-agent review (7 confirmed findings) and live E2E: - gate the whole feature on EXPLICITLY configured voice languages — default-config users keep provider auto-detect exactly as today (voiceLanguages fallback made every user "speaks ONLY English") - commit path is fully synchronous: the early language verdict lands mid-hold via a continuation and commit reads it without awaiting, so a turn can never be dropped or stalled on a cold model load; stale hints are cleared per turn (verified ru→en flip across turns) - feedAudio hops mic chunks to the main queue before touching turn state — chunks arrive on the CoreAudio IOProc thread where @mainactor is unenforced (data race / heap corruption risk on every turn) - value(of:timeoutMs:) now bounds RETURN time with unstructured racers (a task-group race still awaits the non-cancellable child at scope exit, making the 300/1500ms deadlines fictional) - setInputTranscriptionLanguage hops to the session's serial queue - voice-language edits post .voiceLanguagesDidChange; the hub prewarms the LID model and re-warms an idle session so the new languages line reaches the system instruction immediately - onboarding starts with an empty selection so the first pick truly defines the primary; typed-name normalization verified - provider transcript classified UNBIASED (hint-biasing masked the Italian misdetect as "en" in live testing); local transcript must still independently classify into the user's set before any swap - strip Parakeet "<unk>" tokens from fallback bubbles; nb/no mapping; settings-search entry; ptt_test_turn gains force_transcript test seam E2E-verified in omi-ptt-langs bundle: Gemini normal ru turn, Gemini + injected Italian transcript (bubble swapped to local ru transcript), OpenAI ru turn (whisper hint applied pre-commit), OpenAI en control. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 issue found across 10 files
Confidence score: 4/5
desktop/macos/Desktop/Sources/OnboardingLanguageStepView.swiftduplicates language display-name logic that also exists inOnboardingPagedIntroCoordinator.displayName(forLanguageCode:), which can show inconsistent language names across onboarding screens and confuse users; consolidate both call sites to a single resolver (preferably the coordinator/helper) before merging.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="desktop/macos/Desktop/Sources/OnboardingLanguageStepView.swift">
<violation number="1" location="desktop/macos/Desktop/Sources/OnboardingLanguageStepView.swift:120">
P2: There is duplicate language display-name resolution logic between the view and the coordinator that can produce inconsistent results. `OnboardingPagedIntroCoordinator.displayName(forLanguageCode:)` uses `Locale.current`, capitalizes the result, and falls back to `code.uppercased()`. The new `OnboardingLanguageStepView.displayName(_:)` prefers `AssistantSettings.supportedLanguages`, uses `Locale(identifier: "en")`, and falls back to raw `code`. If either side changes—e.g., a supported language gets a custom label or the localization behavior is adjusted—the onboarding chips and the persisted graph labels can diverge. Consider reusing the coordinator's resolver (or extracting a shared helper) so the UI and persisted labels stay in sync.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| } | ||
| } | ||
|
|
||
| private static func displayName(_ code: String) -> String { |
There was a problem hiding this comment.
P2: There is duplicate language display-name resolution logic between the view and the coordinator that can produce inconsistent results. OnboardingPagedIntroCoordinator.displayName(forLanguageCode:) uses Locale.current, capitalizes the result, and falls back to code.uppercased(). The new OnboardingLanguageStepView.displayName(_:) prefers AssistantSettings.supportedLanguages, uses Locale(identifier: "en"), and falls back to raw code. If either side changes—e.g., a supported language gets a custom label or the localization behavior is adjusted—the onboarding chips and the persisted graph labels can diverge. Consider reusing the coordinator's resolver (or extracting a shared helper) so the UI and persisted labels stay in sync.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At desktop/macos/Desktop/Sources/OnboardingLanguageStepView.swift, line 120:
<comment>There is duplicate language display-name resolution logic between the view and the coordinator that can produce inconsistent results. `OnboardingPagedIntroCoordinator.displayName(forLanguageCode:)` uses `Locale.current`, capitalizes the result, and falls back to `code.uppercased()`. The new `OnboardingLanguageStepView.displayName(_:)` prefers `AssistantSettings.supportedLanguages`, uses `Locale(identifier: "en")`, and falls back to raw `code`. If either side changes—e.g., a supported language gets a custom label or the localization behavior is adjusted—the onboarding chips and the persisted graph labels can diverge. Consider reusing the coordinator's resolver (or extracting a shared helper) so the UI and persisted labels stay in sync.</comment>
<file context>
@@ -78,12 +102,24 @@ struct OnboardingLanguageStepView: View {
}
}
+
+ private static func displayName(_ code: String) -> String {
+ AssistantSettings.supportedLanguages.first(where: { $0.code == code })?.name
+ ?? Locale(identifier: "en").localizedString(forLanguageCode: code)
</file context>
… keep continuity context + commit-result API from main, layer per-turn language ID on top) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
K3N4Y
added a commit
to K3N4Y/omi
that referenced
this pull request
Jul 3, 2026
…control plane main rearchitected floating pills (BasedHardware#8655: canonical sessions/runs via DesktopCoordinatorService, no per-pill ChatProvider) and added per-turn PTT language ID (BasedHardware#8948). This merge keeps main's architecture as the spine and re-ports the branch's features onto it: - Codex provider, strengths-based selection guidance, and setup_agent_provider survive as-is (enum-driven parsing now also covers handleRealtimeDelegationRequest and the voice tool path). - Provider startup fallback re-hooked: apply(inspection:) routes `failed` canonical runs through startupFallbackFailure (phase == "startup" allowlist) before committing terminal failure; startProviderAttempt now wraps the coordinator spawn + poll wiring so a retry re-enters it with the pill's updated harness override and cleared canonical ids. - Dropped old-architecture machinery the retry no longer needs (systemPromptSuffixByPill, boot-stagger, per-pill provider/stream teardown) and updated the source-contract tests to assert the live hooks instead. - setup_agent_provider voice ack now uses the canonical FloatingBarVoicePlaybackService.speakOneShot (main forbids raw speak(ack)). - Kept both sides where independent: voice rewarm deferral + late turn-done guard (branch) alongside the PTT headless test action (main).
kodjima33
added a commit
that referenced
this pull request
Jul 4, 2026
…re continuing (#8993) ## Problem Follow-up from #8948's review: `confirmLanguages()` fired the backend `updateUserLanguage` write as fire-and-forget (`Task { _ = try? await ... }`), so a failed save was silent and onboarding advanced anyway — losing the account's primary language with no error shown. This regressed the old save-before-continue contract the step used to have. ## Fix Await the call; on failure, surface an inline error and return before `onContinue()` fires. The local `voiceLanguages` selection is already saved either way, so retrying (tap Continue again) is safe and doesn't lose the user's picks. ## Verification Live-exercised in a named test bundle (`omi-ptt-langs`) with onboarding fully reset: 1. Selected **English** (primary) + **Russian** in the redesigned multi-select language step, tapped Continue. 2. Hit a **real** backend failure (`The data couldn't be read because it is missing` — test bundle's backend URL) — the fix correctly blocked advancement and displayed *"Couldn't save your language to your account — check your connection and tap Continue again."*, with both selections preserved. 3. Log confirms the exact path: `[error] Onboarding: saving primary language 'en' to backend failed: ...` 4. Retried Continue — same safe gate held (no crash, no silent bypass, no data loss). Screenshot of the live failure-blocked state attached to the PR conversation. Scope: one file, 11 lines (`OnboardingPagedIntroCoordinator.swift`) + a changelog fragment. Rebased clean on current `main`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/BasedHardware/omi/pull/8993?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
![https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source](https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source){kind=link}
![https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img](https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img){kind=link}
![https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a](https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a){kind=link}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Speaking Russian to the voice assistant could save the chat bubble in a completely different language — e.g. "Что за таски у меня на сегодня?" persisted as "Che domani sono in tasca come oggi." Gemini Live's input transcription auto-detects language per utterance, offers no language parameter, and short/colloquial utterances routinely misdetect.
Change
Users select every language they speak (multi-select in onboarding + a new Settings → Transcription → Voice Assistant Languages card; first pick = primary). Push-to-talk then identifies the spoken language per turn, fully on-device:
languagehint viasession.updatebefore each commit (verified per-turn ru→en flipping).Also fixes the onboarding normalization bug that saved the literal typed word (
language="russian") instead of the ISO code — which additionally pinned the ambient transcriber to a bogus single language. The language step no longer touches ambient transcription settings at all.Verification
Exercised end-to-end in a named bundle (
omi-ptt-langs) via a newptt_test_turnautomation action that drives the real controller turn path (begin → paced feed → commit → persist), plus real-UI runs:input transcription language → rupre-commit, clean transcripten(no stale hints)en,ru, hub re-warmed, KG updated)Adversarial multi-agent review confirmed 7 findings (2 critical: CoreAudio-thread data race in the turn buffer; a timeout that never bounded return time) — all fixed and re-verified. Agent-logic harness passed. Ambient/conversation transcription pipeline untouched by design.
🤖 Generated with Claude Code