Native Code-Switching Support for Mixed languages

[AzizDXT]

🌟 Feature Request: Native Code-Switching Support

Summary

Add native support for code-switching (mixing multiple languages within a single sentence) in Chatterbox TTS. Currently, the model requires specifying a single language_id, which limits its ability to handle the natural way multilingual speakers communicate.

---

Problem Statement

Chatterbox currently operates in a single-language mode, requiring users to specify one language_id per generation:

model.generate(text, language_id="ar")  # Only Arabic phonetics
model.generate(text, language_id="en")  # Only English phonetics
This creates issues when the input text contains multiple languages, which is extremely common in real-world multilingual communication.
Current Limitations:
Forced single-language processing: Words from secondary languages are processed using the primary language's phonetics
Unnatural pronunciation: Foreign words/phrases inherit the accent and phonetic rules of the specified language
Manual workaround required: Users must manually segment text by language, generate separately, and concatenate audio
Poor user experience: The manual approach produces inconsistent prosody, unnatural pauses, and voice characteristic mismatches
Current Behavior (Problem Examples)
When mixing languages in a single text input:
Scenario 1: Technical/Brand Names
# Input mixing Language A with Language B terms
text = "[Language A phrase] + [Language B brand/term] + [Language A phrase]"
model.generate(text, language_id="language_a")

# Result: Language B words pronounced with Language A phonetics ❌
# The foreign words sound unnatural and difficult to understand
Scenario 2: International Communication
# Professional or educational content with mixed terminology
text = "[Native language explanation] + [English technical term] + [Native language continuation]"

# Problem: Technical terms lose their proper pronunciation
# Users cannot understand which specific term is being referenced
Scenario 3: Cultural Content
# Content reviews, travel vlogs, multicultural discussions
text = "[Language A] + [Language B proper nouns/places] + [Language A]"

# Issue: Place names, brand names, titles are mispronounced
Desired Behavior
Enable automatic language detection and switching within a single generation call:
# Option 1: Fully automatic detection
model.generate(text, language_id="auto")

# Option 2: Specify primary language, auto-detect embedded languages  
model.generate(text, language_id="mixed", primary_language="ar")

# Option 3: Multi-language aware generation
model.generate(text, languages=["ar", "en"], auto_switch=True)
Expected output: The model automatically detects language boundaries and applies the correct phonetics, prosody, and accent for each segment while maintaining natural voice consistency.
Why This Matters
1. Real-World Communication Patterns
Multilingual speakers naturally code-switch: Mixing languages in speech is a normal linguistic phenomenon
Technical/professional contexts: International terminology, brand names, and technical terms are often borrowed from other languages
Global content creation: YouTubers, podcasters, educators frequently mix languages
Demographic reach: Billions of multilingual speakers worldwide
2. Current Market Gap
Commercial TTS solutions (ElevenLabs, Google Cloud) already support code-switching
Open-source gap: No major open-source TTS model handles this naturally
Competitive positioning: This feature would make Chatterbox unique in the open-source space
3. User Impact
Without this feature, users must:
Manually parse and segment text by language
Generate multiple audio files
Manually concatenate with proper timing
Deal with prosody/voice inconsistencies
Significantly increased development complexity
Use Cases & Examples
1. Technology & Education
Content creators explaining technical concepts in their native language while using English terminology:
Software tutorials with English programming terms
Academic lectures with international terminology
Tech reviews mentioning product/brand names
2. Product Reviews & E-commerce
Reviewing products with brand names in original language
Discussing features with manufacturer terminology
International shopping/unboxing content
3. Professional & Business
International business communications
Conference presentations mixing languages
Corporate training with global terminology
4. Entertainment & Media
Multilingual podcasts
Travel vlogs mentioning place names
Cultural content discussing international topics
5. Accessibility & Localization
Educational materials for language learners
Assistive technology for multilingual users
Localized content that preserves original terminology
Concrete Examples (Multiple Language Pairs)
Arabic-English (422M speakers):
"مرحباً، اليوم سنتحدث عن Machine Learning والذكاء الاصطناعي"
"اشتريت iPhone 15 Pro Max من Apple Store"
Spanish-English (500M+ speakers):
"Hoy vamos a hablar sobre Artificial Intelligence en detalle"
"Compré el nuevo MacBook Pro en la Apple Store"
Mandarin-English (1.3B speakers):
"今天我们讨论 Machine Learning 和人工智能"
"我买了 iPhone 15 Pro Max"
Japanese-English (125M speakers):
"今日は Machine Learning について話します"
"iPhone 15 Pro Max を買いました"
Hindi-English (600M+ speakers):
"आज हम Machine Learning के बारे में बात करेंगे"
"मैंने iPhone 15 Pro Max खरीदा"
French-English (275M speakers):
"Aujourd'hui nous parlons de Machine Learning en détail"
"J'ai acheté le iPhone 15 Pro Max"
Technical Implementation Suggestions
Several approaches could enable this feature:
1. Language Detection Layer
Add automatic per-word or per-phrase language detection
Use existing language identification models (fastText, langdetect)
Route detected segments to appropriate language-specific processing
2. Multi-Head Architecture
Implement separate phoneme encoders per language
Shared prosody/timing model for consistency
Language-aware attention mechanism
3. Training Data Approach
Collect code-switched speech datasets
Fine-tune on multilingual parallel data
Use synthetic code-switched data for augmentation
4. Hybrid Tokenization
Multi-script tokenizer supporting multiple writing systems
Language-aware embedding space
Cross-lingual phoneme mapping
5. Staged Implementation
Phase 1: Support 2-3 major language pairs (e.g., English + Spanish/Arabic/Mandarin)
Phase 2: Extend to all currently supported languages
Phase 3: Enable N-way code-switching (3+ languages)
Prior Art & Research
Code-switching TTS is an active research area with proven feasibility:
Mandarin-English TTS: Multiple successful implementations in research papers
Japanese-English TTS: Commercial systems already support this
Malay-English TTS: Research demonstrates viability for diverse language pairs
Cross-lingual voice cloning: Existing literature on maintaining voice identity across languages
Recent advances: Diffusion models and transformer architectures show promise for multilingual generation
Key insight: The technical foundations exist; implementation is primarily an engineering effort.
Proposed API Design
Simple API (Automatic Detection)
# Fully automatic - model detects and switches
audio = model.generate(
    text="[mixed language text]",
    language_id="auto"
)
Guided API (Primary Language + Auto-switch)
# Specify primary language, auto-detect secondary
audio = model.generate(
    text="[mixed language text]",
    language_id="mixed",
    primary_language="ar"  # or "en", "es", etc.
)
Explicit API (Advanced Users)
# Manual language boundary specification (fallback)
audio = model.generate_mixed(
    segments=[
        ("مرحباً اليوم سنتحدث عن", "ar"),
        ("Machine Learning", "en"),
        ("والذكاء الاصطناعي", "ar")
    ]
)
Backward Compatibility
# Existing single-language API remains unchanged
audio = model.generate(text="...", language_id="en")  # Still works
Benefits to Chatterbox Project
Market Differentiation: First major open-source TTS with native code-switching
User Base Growth: Appeal to billions of multilingual speakers globally
Research Community: Enable new research in multilingual speech synthesis
Real-World Utility: Solve actual problems users face daily
Competitive Parity: Match capabilities of commercial solutions
Community Engagement: Drive contributions and data collection efforts
Community Support & Contribution
I believe this feature would have significant community interest and am willing to contribute:
[ ] Data Collection: Help gather/curate code-switched speech datasets
[ ] Testing: Test beta implementations and provide detailed feedback
[ ] Documentation: Write user guides and examples
[ ] Code Contribution: Implement features with guidance from maintainers
[ ] Community Building: Help organize data collection efforts
[ ] Benchmark Creation: Develop evaluation metrics for code-switched TTS
Call to Community: If others are interested in this feature, please 👍 this issue and share your use cases!
Success Metrics
How to measure success of this feature:
Intelligibility: Native speakers can understand both languages clearly
Naturalness: Code-switching sounds like natural human speech
Voice Consistency: Same voice characteristics across language boundaries
Prosody: Natural rhythm and intonation across switches
Accuracy: Correct language detection rate (>95%)
Performance: Minimal latency increase (<10% overhead)
Related Issues / Discussions
[ ] Link to any existing related issues
[ ] Cross-reference multilingual support discussions
[ ] Connect to voice cloning across languages
Additional Context
Chatterbox already has excellent multi-language support (23 languages in v0.1.4)
The infrastructure for language-specific processing exists
This builds on existing strengths rather than requiring architectural overhaul
Strong alignment with project's multilingual mission
Timeline Expectations
Understanding this is a significant feature:
Research Phase: 1-2 months (feasibility, approach selection)
Data Collection: 2-4 months (community effort)
Implementation: 2-4 months (model training, integration)
Beta Testing: 1-2 months (community feedback)
Note: These are rough estimates. I understand complex features take time and am patient for quality implementation.
Thank you for considering this feature request! Chatterbox is already an exceptional project, and native code-switching would make it truly groundbreaking for the global multilingual community. 🌍
I look forward to discussion and collaboration on this feature.
Relevant Tags: #enhancement #multilingual #code-switching #feature-request #community

[thdemeola]

Wanted to second this request. I've been trying to create a synthetic dataset to mix into ASR training, specifically those that mix EN and ES phonotactics, e.g. Spanish phonotactics applied to certain US State names. Georgia is the one that is giving me the most trouble – most of the time, the speech produced sounds like /xeˈoɾxja/ rather than something closer to the English /d͡ʒˈɔɹd͡ʒə/

I was looking through the tokenizer's vocabulary, and found that there are IPA characters there, but the speech produced when using IPA is definitely not supported. Would be great to have the option to use IPA for generating specific pronunciations, dialect variants, etc.

I can sometimes get a viable pronunciation by using a reference audio of Spanish speech using the English language id with Spanish input text, but you don't get very reliable and repeatable results, along with an increase in strange audio artifacts and hallucinated speech appended to the end of the generated audio.

[AzizDXT]

Really, dude, I'm going to try as much as I can to train an AI or Fine-tuning for this current model.

Training really needs accuracy in adjusting the parameters so I try a lot in training attempts🥲

[thdemeola]

I think fine-tuning would be the way to go, barring some sort of architectural or model change. If you have enough code-switching data transcribed correctly, you may get some improvements, but you'd probably need a lot of clean data with many different speakers (sounds very expensive to me lol)