[Support/Question]: Optimizing Streaming Accuracy for Spanish/Bilingual Voice Agents (Nova-3 vs Flux)

[ajay-ezb]

Environment
Integration: Real-time streaming via LiveKit

Current Model: Nova-3 (language=multi)

Use Case: Self-storage voice agent

Primary Demographics: Spanish-speaking and bilingual (English/Spanish code-switching) callers

Description
We are experiencing basic accuracy issues in our streaming integration and want to ensure our configuration is fully optimized before we consider committing to custom model training. We are seeing transcription failures on very standard, non-domain-specific Spanish vocabulary.

We have identified a few potential configuration adjustments based on the Deepgram documentation but need engineering clarification on best practices for our specific architecture.

Specific Inquiries

1. Nova-3 Base Accuracy (Monolingual vs. Multilingual)
   We are seeing failures on basic phrases like "cinco por diez," "ciento cincuenta," "planta baja sin elevador," and "una cama king."

Is there a known base accuracy gap between English and Spanish on Nova-3 specifically in real-time streaming mode?

Would explicitly setting language=es instead of language=multi yield a measurable improvement for short utterances? (The docs suggest multilingual detection adds complexity that can impact short utterance accuracy).

2. Evaluating Flux Multilingual (flux-general-multi)
   The recent technical deep dive for Flux Multilingual explicitly describes our architecture—bilingual callers, code-switching, real-time voice agent—as the target use case, utilizing language_hints=["en", "es"].

Given we are doing real-time streaming via LiveKit, is Flux Multilingual now the recommended default over Nova-3 for heavy code-switching?

Are there latency tradeoffs we need to account for when switching from Nova-3 to Flux in a streaming context?

3. Inference-Time Configuration Tools
   Instead of custom training, we want to maximize inference-time configurations to capture our specific storage vocabulary (unit dimensions, pricing, etc.).

Keyterm Prompting: The docs indicate Nova-3 supports up to 100 keyterms. Does this perform reliably in streaming mode, and what is the recommended weighting strategy when dealing with Spanish or multilingual streams?

Endpointing: The docs recommend endpointing=100 for code-switching scenarios (to account for language-switching pauses). When using LiveKit, should we pass this parameter to Deepgram, or rely entirely on LiveKit's native VAD?

Thank you in advance for any technical guidance you can provide!

[deepgram-community[bot]]

Thanks for asking your question. Please be sure to reply with as much detail as possible so the community can assist you efficiently.
_{Consider joining our Discord community for more opportunity to engage with your fellow Deepgram users. You can earn points which can be redeemed for cool stuff by being active in our communities!}

[deepgram-community[bot]]

Hey there! It looks like you haven't connected your GitHub account to your Deepgram account. You can do this at https://community.deepgram.com - being verified through this process will allow our team to help you in a much more streamlined fashion.

[deepgram-community[bot]]

It looks like we're missing some important information to help debug your issue. Would you mind providing us with the following details in a reply?

• The programming language you are working in (e.g. JavaScript, Python).
• A request ID that triggered your error or issue.

[ajay-ezb]

Programming Language: Python

Request ID: This issue isn't isolated to a single request; we are seeing these transcription failures across almost every Spanish conversation we process.

[albertocubeddu]

+1

[albertocubeddu]

Not even able to transcribe back the voice "nestor".

[mayowaosibodu]

Hey @ajay-ezb — we built a phonetic pre-filter that addresses exactly this. It verifies keyterms against the actual audio before they reach the decoder, so you don't get false matches on similar-sounding Spanish terms. 60% false positive reduction in testing with no valid keyterm loss. Here's how it works: https://aditu.tech/keyterm-filtering

Happy to help you test it on your self-storage use case — reach out at contact@aditu.tech if you want to try it.