Supertonic-PyTorch

A PyTorch implementation of Supertonic TTS v2, reconstructed from ONNX models.

Overview

This project provides a pure PyTorch implementation of the Supertonic text-to-speech system, enabling:

• GPU acceleration with PyTorch
• Model fine-tuning and transfer learning
• Integration with PyTorch-based pipelines
• Research and experimentation

Credits

Original Model: Supertone - The Supertonic TTS model was developed by Supertone Inc.

• Original ONNX models: Supertone/supertonic-2
• License: OpenRAIL

This repository is an unofficial PyTorch reconstruction for research and educational purposes.

Features

• 5 Female voices: F1, F2, F3, F4, F5
• 5 Male voices: M1, M2, M3, M4, M5
• Multi-language support: 5 languages
• High-quality 44.1kHz audio output
• Flow matching for audio generation (8-step Euler integration)

Architecture

Component	Description
Duration Predictor	Predicts speech duration from text
Text Encoder	Encodes text with style conditioning
Vector Estimator	Flow matching model with RoPE attention
Vocoder	Converts latent representations to waveform

Installation

# Clone the repository
git clone https://github.com/chantysothy/supertonic-pytorch.git
cd supertonic-pytorch

# Install dependencies
pip install torch numpy onnxruntime scipy

# Download ONNX models from HuggingFace (required)
cd onnx_models/onnx
curl -LO https://huggingface.co/Supertone/supertonic-2/resolve/main/onnx/duration_predictor.onnx
curl -LO https://huggingface.co/Supertone/supertonic-2/resolve/main/onnx/text_encoder.onnx
curl -LO https://huggingface.co/Supertone/supertonic-2/resolve/main/onnx/vector_estimator.onnx
curl -LO https://huggingface.co/Supertone/supertonic-2/resolve/main/onnx/vocoder.onnx
cd ../..

Usage

Generate Audio with PyTorch

import torch
from supertonic_v2_pytorch import load_all_models, UnicodeProcessor, load_voice_style

# Load models
models = load_all_models()

# Setup text processing
processor = UnicodeProcessor("assets/unicode_indexer.json")
style = load_voice_style("assets/voice_styles/F1.json")  # or M1, F2, etc.

# Process text
text = "Hello, this is a test of the text to speech system."
text_ids, text_mask = processor([text], ["en"])
text_ids = torch.from_numpy(text_ids)
text_mask = torch.from_numpy(text_mask)
style_ttl = torch.from_numpy(style.ttl)
style_dp = torch.from_numpy(style.dp)

with torch.no_grad():
    # Predict duration
    duration = models['duration_predictor'](text_ids, style_dp, text_mask)

    # Encode text
    text_emb = models['text_encoder'](text_ids, style_ttl, text_mask)

    # Generate latent with flow matching
    latent_length = int(duration.item() * 44100 / 512 / 6)
    latent_mask = torch.ones(1, 1, latent_length)

    # Start from noise
    xt = torch.randn(1, 144, latent_length)

    # 8-step Euler integration
    for step in range(8):
        v = models['vector_estimator'](
            xt, text_emb, style_ttl, latent_mask, text_mask,
            torch.tensor([step], dtype=torch.float32),
            torch.tensor([8], dtype=torch.float32)
        )
        xt = xt + v * (1.0 / 8)

    # Decode to audio
    audio = models['vocoder'](xt * latent_mask)

# Save audio (44.1kHz)
import scipy.io.wavfile as wavfile
audio_np = audio.numpy().flatten()
audio_int16 = (audio_np * 32767).astype('int16')
wavfile.write("output.wav", 44100, audio_int16)

Compare ONNX vs PyTorch

python compare_stages.py

Project Structure

supertonic-pytorch/
├── assets/
│   ├── unicode_indexer.json
│   └── voice_styles/
│       ├── F1.json - F5.json    # Female voices
│       └── M1.json - M5.json    # Male voices
├── onnx_models/
│   └── onnx/
│       ├── duration_predictor.onnx
│       ├── text_encoder.onnx
│       ├── vector_estimator.onnx
│       ├── vocoder.onnx
│       └── tts.json
├── training/                     # Training code
│   ├── config.py                # Training configurations
│   ├── dataset.py               # Data loading
│   ├── latent_encoder.py        # VAE encoder for audio→latent
│   ├── speaker_encoder.py       # Style/speaker embeddings
│   ├── losses.py                # Loss functions
│   ├── train_encoder.py         # Phase 1: Latent encoder
│   ├── train_duration.py        # Phase 2: Duration predictor
│   └── train_flow.py            # Phase 3: Flow matching
├── supertonic_v2_pytorch.py     # PyTorch implementation
├── compare_stages.py            # ONNX vs PyTorch comparison
├── generate_pytorch_audio.py    # Audio generation script
└── README.md

Training

Training is done in 3 phases. See training/README.md for details.

Phase 1: Latent Encoder

Train a VAE encoder to convert audio waveforms to the 144-dim latent space.

python -m training.train_encoder --data_dir /path/to/data --batch_size 8

Phase 2: Duration Predictor

python -m training.train_duration --data_dir /path/to/data --batch_size 32

Phase 3: Flow Matching (Text Encoder + Vector Estimator)

python -m training.train_flow \
    --data_dir /path/to/data \
    --encoder_checkpoint checkpoints/encoder/encoder_final.pt

Data Format

Prepare data as metadata.json with audio paths, text, language, and speaker IDs. See training/README.md for the full format.

Validation

The PyTorch implementation has been validated against the original ONNX models:

Stage	Max Difference
Duration Predictor	0.000000
Text Encoder	0.000005
Vector Estimator	0.000032
Final Latent	0.000050
Audio	~0.17 (numerical precision)

License

This project follows the OpenRAIL license from the original Supertonic model.

Disclaimer

This is an unofficial implementation created for research and educational purposes. For official support and commercial use, please contact Supertone.

---

Built with Claude Code

This entire project was built using Claude Code, Anthropic's agentic coding tool.

Metric	Value
Model	Claude Opus 4.5 (claude-opus-4-5-20251101)
Development Time	~7.5 hours
Estimated Tokens	~800K input, ~150K output
Estimated Cost	~$23 USD

The process included:

• Analyzing ONNX model graphs to understand architecture
• Reconstructing PyTorch modules from ONNX weights
• Debugging RoPE attention, CrossAttention, and ConvNeXt blocks
• Validating output matching between ONNX and PyTorch
• Writing documentation and pushing to GitHub

All code was generated and debugged interactively through Claude Code.