feat: add Output Language setting for automatic translation#102
Conversation
Add a new Output Language picker in Settings > General that translates transcribed speech into a selected language during post-processing. When set (e.g. English), the system prompt is modified to translate instead of preserving the original language. Supports English, Chinese, Spanish, French, Japanese, Korean, German, and Portuguese. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds a persisted Changes
Sequence DiagramsequenceDiagram
actor User
participant SettingsView
participant AppState
participant SessionController
participant PostProcessingService
participant LLM
User->>SettingsView: choose output language
SettingsView->>AppState: set outputLanguage
AppState->>AppState: persist to UserDefaults
User->>SessionController: submit/stop transcript
SessionController->>AppState: read outputLanguage
AppState-->>SessionController: return outputLanguage
SessionController->>PostProcessingService: postProcess(transcript,..., outputLanguage)
PostProcessingService->>PostProcessingService: applyOutputLanguage(systemPrompt, language)
PostProcessingService->>LLM: send modified prompt + transcript
LLM-->>PostProcessingService: return processed transcript
PostProcessingService-->>SessionController: return final transcript
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@Sources/PostProcessingService.swift`:
- Around line 555-569: applyOutputLanguage currently assumes three exact bullet
strings and fails when customSystemPrompt changes; update applyOutputLanguage to
be robust by (1) searching for and removing or replacing any existing lines or
bullets that mention translation/preservation using pattern matching (e.g.,
regex matches for lines containing "translate", "preserve", "translation", or
"mixed-language") rather than exact literals, and then (2) appending a clear
fallback instruction like "Translate the final cleaned text into {language}.
Output ONLY in {language}." if nothing was replaced. Ensure you modify the
applyOutputLanguage(_ prompt: String, language: String) implementation so it
operates idempotently on prompts passed from process() (including user-supplied
customSystemPrompt) and always enforces the desired translation directive.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: b605e8c0-c843-4dcb-876c-67529ca89036
📒 Files selected for processing (3)
Sources/AppState.swiftSources/PostProcessingService.swiftSources/SettingsView.swift
When the user has a custom system prompt, the exact string replacements may not match. Now appends a translation directive as fallback when none of the default prompt lines were found. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Instead of fragile string replacements that break with custom prompts, just append the translation directive at the end. Simpler and works universally. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🧹 Nitpick comments (1)
Sources/PostProcessingService.swift (1)
472-479: Consider using append approach for consistency with dictation mode.Command mode uses string replacement while dictation mode (line 372) uses the append-based
applyOutputLanguage. Although the replacement targets a constant defined in this file and works correctly, a future edit tocommandModeSystemPromptcould silently break this without compile-time detection.For consistency and resilience, consider appending the language directive instead:
♻️ Suggested refactor
var systemPrompt = Self.commandModeSystemPrompt let trimmedOutputLanguage = outputLanguage.trimmingCharacters(in: .whitespacesAndNewlines) if !trimmedOutputLanguage.isEmpty { - systemPrompt = systemPrompt.replacingOccurrences( - of: "- Preserve the original language unless VOICE_COMMAND explicitly requests translation.", - with: "- Output the result in \(trimmedOutputLanguage)." - ) + systemPrompt += "\n\nIMPORTANT: Output the result in \(trimmedOutputLanguage), regardless of the original language of SELECTED_TEXT." }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@Sources/PostProcessingService.swift` around lines 472 - 479, The current code mutates systemPrompt using replacingOccurrences which can break silently if Self.commandModeSystemPrompt changes; instead, use the same append-based helper used in dictation mode: call the existing applyOutputLanguage helper (the same one referenced around line 372) to append the output language directive to systemPrompt when trimmedOutputLanguage is non-empty, referencing systemPrompt and Self.commandModeSystemPrompt so the logic matches dictation mode and avoids fragile string replacements.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@Sources/PostProcessingService.swift`:
- Around line 472-479: The current code mutates systemPrompt using
replacingOccurrences which can break silently if Self.commandModeSystemPrompt
changes; instead, use the same append-based helper used in dictation mode: call
the existing applyOutputLanguage helper (the same one referenced around line
372) to append the output language directive to systemPrompt when
trimmedOutputLanguage is non-empty, referencing systemPrompt and
Self.commandModeSystemPrompt so the logic matches dictation mode and avoids
fragile string replacements.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 9026557f-5347-4c2d-a84e-e966cd99bad6
📒 Files selected for processing (1)
Sources/PostProcessingService.swift
|
Hi, the idea of having translation built in is great. I would like to discuss the implementation in more detail: For edit mode, not sure this is needed as the speaker can just say "translate this to x language" and it seems to work well from my testing. Have you encountered any issues with that? I am hesitant to add the persistent settings toggle as I am not sure what the use case is for having all dictations translated. What do you think about adding a "trigger translation" phrase that if found in the transcription will add the "translate this to x language" to the prompt? I will test a couple of runs to see if having it in the user prompt is enough. In general I think it's better to not dynamically modify the system prompt between runs as that would bust the cache on Groq and cause extra usage. |
|
Thanks for taking the time to dig into this — these are fair concerns. I see this primarily as a UX improvement for non-native English speakers, a user persona underserved by the trigger-phrase alternative. Consider a non-native English speaker who uses Flow throughout their workday — Slack messages, emails, PR descriptions, Linear tickets, docs. Their native language might be Mandarin, Spanish, Portuguese, German, Hindi, whatever. Their output language, however, is English, essentially 100% of the time in a work context. For this user, translation isn't an occasional intent they opt into per-utterance; it's the stable default of their entire workflow. You're right that "translate this to X" works well when someone consciously wants a translation for a specific utterance — but asking this user to prepend it to every single dictation, or to remember a trigger phrase each time, turns what should be a fluent dictation experience into a constant cognitive tax, and worse, a constant reminder that the tool wasn't really built with them in mind. A trigger phrase solves the mechanics but not the ergonomics. This is actually one of the higher-leverage UX improvements Flow could make for non-English-speaking users, and a real lever for adoption beyond the native-English audience. The population of knowledge workers worldwide who think in one language and write professionally in another is enormous, and right now Flow implicitly assumes input language == output language. A persistent setting is the simplest, most honest way to say "your output language is a preference, not a per-utterance decision." On the Groq caching concern specifically: because this is a setting rather than a dynamic per-run value, the system prompt stays stable until the user explicitly changes it in settings. Cache invalidation would only happen on setting changes, which should be rare in practice. Happy to verify this empirically if helpful, but I don't think it meaningfully degrades cache hit rate for any given user. |
…translation # Conflicts: # Sources/AppState.swift
|
Makes sense. My only worry left is that the models that can realistically be used for post-processing are not that great at translation. Do you have experience using this feature in practice and have seen good results? I couldn't really find some good benchmarks around but would love seeing some. Re: groq caching should be OK with static setting Thanks! |
Adds a Transcription Language picker in Settings → Advanced Provider Settings (also visible in the initial Setup flow since the component is shared). The selection is passed as the `language` form field on non-realtime transcription requests and as the `language` field on the realtime streaming Configuration. The non-realtime TranscriptionService previously sent no language parameter at all. The realtime RealtimeTranscriptionService already had `language: String?` plumbing, but the call site in AppState hardcoded `language: nil`, so the feature was wired but inert. Users report wrong-script characters (for example CJK) leaking into English dictation — a known Whisper failure mode on short utterances when no language hint is supplied. Providing the ISO 639-1 code resolves this. Related: zachlatta#102 (marwei) takes a complementary approach by modifying the post-processing system prompt to translate output into a chosen language, operating at the LLM cleanup layer. This change operates at the transcription layer and is independent of post-processing. Note on provider behavior: empirically, Groq's Whisper endpoint treats this parameter closer to "target language" — audio spoken in one language is translated into the selected language. OpenAI documents it as a pure input-audio hint. Default is Auto-detect, which sends no field and preserves current behavior for existing users. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Thanks — ran an eval in both directions to check. 280 samples across 14 language pairs (20 each), using a script that mirrors Headline: 12 of 14 pairs are very strong, and the remaining two (English → CJK) are still solid on translation quality itself. The issues I saw on en-ja and en-zh weren't translation-quality problems — they were cases where Couldn't get a full run on Caveats: N=20/pair, synthesized inputs, single LLM judge. Script + raw 280-sample results: https://gist.github.com/marwei/3d7efb7126bea19831d22465d8bf936d |
|
Look good! Thank you for putting this together. Will solve the conflicts and merge shortly. |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
Sources/SettingsView.swift (1)
966-985: Consider replacing the empty-string sentinel list with explicit value/label options.Using
""inoutputLanguageOptionsplusdropFirst()works, but it’s brittle and ties storage values to display text. A small value/label model keeps persistence stable and simplifies rendering.♻️ Proposed refactor
- private static let outputLanguageOptions = [ - "", - "English", - "Chinese (Simplified)", - "Chinese (Traditional)", - "Spanish", - "French", - "Japanese", - "Korean", - "German", - "Portuguese", - ] + private struct OutputLanguageOption: Identifiable { + let id: String // persisted value + let label: String // UI label + } + + private static let outputLanguageOptions: [OutputLanguageOption] = [ + .init(id: "", label: "Same as spoken"), + .init(id: "English", label: "English"), + .init(id: "Chinese (Simplified)", label: "Chinese (Simplified)"), + .init(id: "Chinese (Traditional)", label: "Chinese (Traditional)"), + .init(id: "Spanish", label: "Spanish"), + .init(id: "French", label: "French"), + .init(id: "Japanese", label: "Japanese"), + .init(id: "Korean", label: "Korean"), + .init(id: "German", label: "German"), + .init(id: "Portuguese", label: "Portuguese"), + ] @@ - Picker("Language", selection: $appState.outputLanguage) { - Text("Same as spoken").tag("") - ForEach(Self.outputLanguageOptions.dropFirst(), id: \.self) { lang in - Text(lang).tag(lang) - } - } + Picker("Language", selection: $appState.outputLanguage) { + ForEach(Self.outputLanguageOptions) { option in + Text(option.label).tag(option.id) + } + }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@Sources/SettingsView.swift` around lines 966 - 985, Replace the empty-string sentinel in outputLanguageOptions with an explicit value/label model and update the Picker to use that model: create a small struct or tuple array (e.g., DisplayOption { let id/value: String; let label: String }) and replace Self.outputLanguageOptions with an array of those options, ensuring the first option represents "Same as spoken" with a stable value (not ""), then update outputLanguageSection's Picker to iterate over the new option collection and bind tags to option.id/value while showing option.label; keep appState.outputLanguage as the persisted value used for selection.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@Sources/SettingsView.swift`:
- Around line 989-991: Update the helper Text that currently reads "When set,
FreeFlow translates your speech into the selected language." to explicitly
mention that translation also applies during Edit Mode transforms; locate the
Text literal in SettingsView (the Text(...) line in Sources/SettingsView.swift)
and replace the string with wording like "When set, FreeFlow translates your
speech — and Edit Mode transforms — into the selected language." to make the
scope clear to users.
---
Nitpick comments:
In `@Sources/SettingsView.swift`:
- Around line 966-985: Replace the empty-string sentinel in
outputLanguageOptions with an explicit value/label model and update the Picker
to use that model: create a small struct or tuple array (e.g., DisplayOption {
let id/value: String; let label: String }) and replace
Self.outputLanguageOptions with an array of those options, ensuring the first
option represents "Same as spoken" with a stable value (not ""), then update
outputLanguageSection's Picker to iterate over the new option collection and
bind tags to option.id/value while showing option.label; keep
appState.outputLanguage as the persisted value used for selection.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 430406c4-c834-4092-8bbb-b23ea9236eb1
📒 Files selected for processing (2)
Sources/AppState.swiftSources/SettingsView.swift
🚧 Files skipped from review as they are similar to previous changes (1)
- Sources/AppState.swift
| Text("When set, FreeFlow translates your speech into the selected language.") | ||
| .font(.caption) | ||
| .foregroundStyle(.secondary) |
There was a problem hiding this comment.
Update the helper text to include Edit Mode behavior.
Line 989 currently says “translates your speech,” but translation also applies during Edit Mode transforms. This copy may mislead users about scope.
✏️ Suggested copy tweak
- Text("When set, FreeFlow translates your speech into the selected language.")
+ Text("When set, FreeFlow translates post-processed output into the selected language (including dictation and Edit Mode).")🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@Sources/SettingsView.swift` around lines 989 - 991, Update the helper Text
that currently reads "When set, FreeFlow translates your speech into the
selected language." to explicitly mention that translation also applies during
Edit Mode transforms; locate the Text literal in SettingsView (the Text(...)
line in Sources/SettingsView.swift) and replace the string with wording like
"When set, FreeFlow translates your speech — and Edit Mode transforms — into the
selected language." to make the scope clear to users.
2 participants
Summary
Test plan
🤖 Generated with Claude Code
Summary by CodeRabbit