Urdu Text-to-Speech with Voice Cloning using SpeechT5

A fine-tuned SpeechT5 model for high-quality Urdu text-to-speech generation with voice cloning capabilities. This model supports both Urdu and Roman Urdu scripts and allows speaker selection for personalized speech synthesis.

Features

• 🗣️ Urdu TTS: High-quality text-to-speech synthesis for Urdu language
• 🔊 Voice Cloning: Generate speech in the style of specific speakers (including Zia Mohiuddin's voice)
• 🌐 Dual Script Support: Works with both Urdu (نثر) and Roman Urdu (Urdu written in Latin script)
• 🎛️ Speaker Selection: Choose between different voice profiles
• 🚀 FastAPI Demo: Interactive web interface for testing the model

Model Details

This implementation is based on SpeechT5, a state-of-the-art model for speech synthesis tasks. Key modifications include:

• Tokenization: Character-level tokenization specifically adapted for Urdu script
• Preprocessing: Updated tokenizer and processor to handle Urdu phonetics and pronunciation
• Architecture: Fine-tuned SpeechT5 architecture with multilingual capabilities

Dataset

The model was trained on a merged dataset comprising:

1. xcollab tts 15k dataset: A comprehensive Urdu speech dataset with 15,000+ recordings
2. Zia Mohiuddin Dataset: 350 high-quality recordings of the renowned Pakistani broadcaster

This combination enables both general Urdu TTS and voice cloning capabilities for specific speakers.

Training

• Epochs: 50 (significant improvement observed after 40 epochs)
• Batch Size: 6-8 (smaller batch sizes yielded better results)
• Hardware: GPU-accelerated training
• Performance: Mid-level quality with potential for improvement through:
  • Longer training (100+ epochs recommended)
  • Larger model variants
  • Additional high-quality data

OpenCut API

This repository is the Python service pinned by OpenCut. Transliteration is available with the default lightweight dependency set. SpeechT5 is optional because its fine-tuned processor and model checkpoint are not stored in Git.

Prerequisites

• Python 3.10 through 3.13
• uv 0.11 or newer

Local development

uv sync --frozen
uv run uvicorn main:app --host 127.0.0.1 --port 8000 --reload

Open http://127.0.0.1:8000/health to inspect process, transliteration, and TTS model readiness. Run the contract tests with:

uv run pytest

To install the optional TTS Python libraries and the pinned, checksum-verified model files, run:

uv sync --frozen --group tts
./scripts/install_models.sh

The compatible Urdu SpeechT5 checkpoint is downloaded from pocketmonkey/speecht5_tts_urdu at a fixed revision. It replaces the original prototype's unpublished fine-tuned checkpoint. Model directories are ignored by Git.

Demo Features:

• Text input in Urdu or Roman Urdu
• Speaker selection dropdown
• Real-time audio generation
• Responsive web interface

Installation

git clone https://github.com/your-username/urdu-tts-voice-cloning.git
cd urdu-tts-voice-cloning
pip install -r requirements.txt

Usage

Python API

from model import UrduTTS

tts = UrduTTS()
audio = tts.generate_text_to_speech(
    text="یہ ایک مثال ہے",  # Urdu text
    speaker="zia_mohiuddin"  # Optional speaker selection
)
audio.save("output.wav")

Roman Urdu Support

audio = tts.generate_text_to_speech(
    text="Ye aik misaal hai",  # Roman Urdu
    speaker="default"
)

Performance Notes

• Current model achieves mid-level quality with natural-sounding output
• Best results obtained with:
  • 40+ training epochs
  • Batch sizes of 6-8
  • Adequate GPU memory (recommended: 16GB+)
• Voice cloning works best with clear reference recordings

Future Improvements

• Increase training epochs to 100+
• Experiment with larger SpeechT5 variants
• Expand dataset with more diverse speakers
• Implement Roman Urdu normalization
• Add prosody control features
• Optimize for real-time applications

Tech Stack

• Model: SpeechT5 (fine-tuned)
• Backend: FastAPI
• Frontend: HTML/CSS/JavaScript
• Audio Processing: Librosa, SoundFile
• ML Framework: PyTorch, Transformers

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Acknowledgments

• Original SpeechT5 model by Microsoft Research
• xcollab tts dataset contributors
• Zia Mohiuddin dataset providers