Skip to content

feat: flux multilang #1034

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andrenatal merged 2 commits into from
Apr 29, 2026
+90 −41
Merged

feat: flux multilang #1034

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
943e7c1
Incorporating flux-general-multi
andrenatal Apr 28, 2026
765c665
Stating languages supported by multilang flux
andrenatal Apr 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion fern/customization/speech-configuration.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ This plan defines the parameters for when the assistant begins speaking after th

**Audio-text based providers:**

- **Deepgram Flux**: Deepgram's latest transcriber model with built-in conversational speech recognition. Flux combines high-quality speech-to-text with native turn detection, while delivering ultra-low latency and Nova-3 level accuracy.
- **Deepgram Flux**: Deepgram's latest transcriber model with built-in conversational speech recognition. Flux combines high-quality speech-to-text with native turn detection, while delivering ultra-low latency and Nova-3 level accuracy. Available in English (`flux-general-en`) and multilingual (`flux-general-multi`) variants. Supported languages for `flux-general-multi`: English (`en`), Spanish (`es`), French (`fr`), German (`de`), Hindi (`hi`), Russian (`ru`), Portuguese (`pt`), Japanese (`ja`), Italian (`it`), Dutch (`nl`).

- **Assembly**: Transcriber that also reports end-of-turn detection. To use Assembly, choose it as your transcriber without setting a separate smart endpointing plan. As transcripts arrive, we consider the `end_of_turn` flag that Assembly sends to mark the end-of-turn, stream to the LLM, and generate a response.

Expand Down
2 changes: 1 addition & 1 deletion fern/customization/transcriber-fallback-plan.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ Each transcriber provider supports different configuration options. Expand the a

<AccordionGroup>
<Accordion title="Deepgram">
- **model**: Model selection (`nova-3`, `nova-3-general`, `nova-3-medical`, `nova-2`, `flux-general-en`, etc.).
- **model**: Model selection (`nova-3`, `nova-3-general`, `nova-3-medical`, `nova-2`, `flux-general-en`, `flux-general-multi`, etc.). Use `flux-general-en` for English-only conversations and `flux-general-multi` for multilingual conversations. Supported languages for `flux-general-multi`: `en`, `es`, `fr`, `de`, `hi`, `ru`, `pt`, `ja`, `it`, `nl`.
- **language**: Language code for transcription.
- **keywords**: Keywords with optional boost values for improved recognition (e.g., `["companyname", "productname:2"]`).
- **keyterm**: Keyterm prompting for up to 90% keyword recall rate improvement.
Expand Down
127 changes: 88 additions & 39 deletions fern/customization/voice-pipeline-configuration.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ Uses AI models to analyze speech patterns, context, and audio cues to predict wh
- **krisp**: Audio-based model analyzing prosodic features (intonation, pitch, rhythm)

**Audio-text based providers:**
- **deepgram-flux**: Deepgram's latest transcriber model with built-in conversational speech recognition. (English only)
- **deepgram-flux**: Deepgram's latest transcriber model with built-in conversational speech recognition. Use `flux-general-en` for English-only conversations or `flux-general-multi` for multilingual conversations.
- **assembly**: Transcriber with built-in end-of-turn detection (English only)

<hr />
Expand All @@ -199,7 +199,7 @@ Uses AI models to analyze speech patterns, context, and audio cues to predict wh

**When to use smart endpointing:**

- **Deepgram Flux**: English conversations using Deepgram as a transcriber.
- **Deepgram Flux**: English and Multi-lingual conversations using Deepgram as a transcriber.
- **Assembly**: Best used when Assembly is already your transcriber provider for English conversations with integrated end-of-turn detection
- **LiveKit**: English conversations where Deepgram is not the transcriber of choice.
- **Vapi**: Non-English conversations with default stop speaking plan settings
Expand All @@ -221,19 +221,37 @@ Deepgram Flux's end-of-turn detection is configured at the transcriber level, al
- **4000-6000:** Standard timeout (default: 5000) - natural conversation flow
- **7000-10000:** Extended timeout for complex or thoughtful responses

**Configuration example:**
**Configuration examples:**

```json
{
"transcriber": {
"provider": "deepgram",
"model": "flux-general-en",
"language": "en",
"eotThreshold": 0.7,
"eotTimeoutMs": 5000
}
}
```
<Tabs>
<Tab title="English">
```json
{
"transcriber": {
"provider": "deepgram",
"model": "flux-general-en",
"language": "en",
"eotThreshold": 0.7,
"eotTimeoutMs": 5000
}
}
```
</Tab>
<Tab title="Multilingual">
```json
{
"transcriber": {
"provider": "deepgram",
"model": "flux-general-multi",
"eotThreshold": 0.7,
"eotTimeoutMs": 5000
}
}
```

Supported languages: English (`en`), Spanish (`es`), French (`fr`), German (`de`), Hindi (`hi`), Russian (`ru`), Portuguese (`pt`), Japanese (`ja`), Italian (`it`), Dutch (`nl`). Set the `language` field to one of these codes, or omit it to enable automatic language detection.
</Tab>
</Tabs>

### LiveKit's Wait function

Expand Down Expand Up @@ -669,32 +687,63 @@ User Interrupts → Assistant Audio Stopped → backoffSeconds Blocks All Output

### Audio-text based endpointing (Deepgram Flux example)

```json
{
"transcriber": {
"provider": "deepgram",
"model": "flux-general-en",
"language": "en",
"eotThreshold": 0.7,
"eotTimeoutMs": 5000
},
"stopSpeakingPlan": {
"numWords": 2,
"voiceSeconds": 0.2,
"backoffSeconds": 1.0,
"acknowledgementPhrases": [
"okay",
"right",
"uh-huh",
"yeah",
"mm-hmm",
"got it"
]
}
}
```
<Tabs>
<Tab title="English">
```json
{
"transcriber": {
"provider": "deepgram",
"model": "flux-general-en",
"language": "en",
"eotThreshold": 0.7,
"eotTimeoutMs": 5000
},
"stopSpeakingPlan": {
"numWords": 2,
"voiceSeconds": 0.2,
"backoffSeconds": 1.0,
"acknowledgementPhrases": [
"okay",
"right",
"uh-huh",
"yeah",
"mm-hmm",
"got it"
]
}
}
```

**Optimized for:** English conversations where Deepgram is set as transcriber.
</Tab>
<Tab title="Multilingual">
```json
{
"transcriber": {
"provider": "deepgram",
"model": "flux-general-multi",
"eotThreshold": 0.7,
"eotTimeoutMs": 5000
},
"stopSpeakingPlan": {
"numWords": 2,
"voiceSeconds": 0.2,
"backoffSeconds": 1.0,
"acknowledgementPhrases": [
"okay",
"right",
"uh-huh",
"yeah",
"mm-hmm",
"got it"
]
}
}
```

**Optimized for:** English conversations where Deepgram is set as transcriber.
**Optimized for:** Multilingual conversations where Deepgram is set as transcriber. Supported languages: English (`en`), Spanish (`es`), French (`fr`), German (`de`), Hindi (`hi`), Russian (`ru`), Portuguese (`pt`), Japanese (`ja`), Italian (`it`), Dutch (`nl`). Omit `language` to enable automatic language detection.
</Tab>
</Tabs>

### Education and training

Expand Down
Loading