🦜 VieNeu-TTS

VieNeu-TTS is the next generation of on-device Vietnamese TTS: 10,000+ hours of bilingual training, instant voice cloning, a dedicated Podcast/Conversation mode, and extremely fast — real-time streaming with first audio in ~115 ms, RTF ≈ 0.01–0.02 batched on a consumer GPU (RTX 3060), ~2× real-time on a plain CPU (benchmarks).

1. 🦜 Installation & Web UI
2. 📦 Using the Python SDK
3. 🐳 API Server & Docker — OpenAI-compatible streaming API (v3 Turbo) · legacy v2 server
4. 📊 Benchmarks — every speed / latency number in one place (CPU vs GPU, batch, streaming, Nano)
5. 🎓 Fine-tuning (LoRA)
6. 🔬 Model Overview
7. 🚀 Roadmap
8. 🤝 Support & Contact
9. 📑 Citation

uv is the fastest way to manage dependencies.

# Windows:
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

# Linux/macOS:
curl -LsSf https://astral.sh/uv/install.sh | sh

1. Clone the Repo:

   git clone https://github.com/pnnbao97/VieNeu-TTS.git
   cd VieNeu-TTS

2. Install Dependencies:

   📊 Which one? CPU ≈ RTF 0.5 (2× real-time, one stream) · GPU ≈ RTF 0.02 batched (~50× real-time) and 16 real-time streams — all measurements in §4 Benchmarks.

   • Option 1: CPU & macOS (minimal, torch-free) — RTF ≈ 0.5, no GPU needed

     uv sync

   • Option 2: GPU — v3 Turbo on GPU (PyTorch) — RTF ≈ 0.02 batched, 16 real-time streams

     uv sync --extra cuda

3. Start the Web UI:

   uv run vieneu-web

   Access the UI at http://127.0.0.1:7860.

The vieneu SDK defaults to VieNeu-TTS v3 Turbo (48 kHz). The minimal install is torch-free: on CPU everything runs on ONNX Runtime (PyTorch is never imported), and on a CUDA machine it auto-switches to the PyTorch engine — where inference is batched automatically (same API, no code change).

CPU (default) — torch-free, runs v3 Turbo via ONNX Runtime. Most users want this — RTF ≈ 0.5 on a 12th-gen Core i5 (numbers in §4 Benchmarks):

⚡On CPU the backbone runs fp32 by default (maximum fidelity). Need more speed? Pass Vieneu(precision="int8") — ~1.6× faster (RTF ≈ 0.35) and ~4× smaller, but it requires a CPU with VNNI (AVX-512 VNNI / AVX-VNNI); on older CPUs int8 can produce garbled audio. precision only affects the CPU/ONNX path; on GPU it's ignored (PyTorch).

🪶 Still too slow, or deploying on a phone / ARM board? Try VieNeu-TTS v3 Nano (preview) — Vieneu(mode="v3nano"), ~3× faster than Turbo fp32 on CPU (RTF 0.11–0.22 on a desktop CPU), but noticeably lower quality (especially English / bilingual), 24 kHz, 11 preset voices + voice cloning. Details and caveats in the v3 Nano section below.

pip install vieneu

GPU (CUDA) — only if you have an NVIDIA GPU; ~25× faster than the CPU path (RTF ≈ 0.02 batched, 16 real-time streams — §4 Benchmarks). On Linux pip install "vieneu[cuda]" is enough (PyPI torch ships CUDA there); on Windows install the CUDA torch first as below.

⚡ Every audio frame on GPU is one CUDA graph (acoustic decoder + sampling + repetition penalty + backbone step in a single replay — no torch.compile, no C++ toolchain needed). The first call for each batch size pays ~0.5 s to capture the graph (kept afterwards; servers can call warm_fused() at start-up). VIENEU_FUSED_FRAME=0 restores the plain loop.

pip install torch==2.8.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128
pip install "transformers==4.57.6"   # pinned — most stable transformers for the GPU SDK
pip install vieneu

import time
from vieneu import Vieneu

# Default = v3 Turbo (48 kHz). GPU → PyTorch (auto-detected).
vieneu = Vieneu() # On a GPU machine you can still switch to ONNX/CPU if you prefer: Vieneu(backend="onnx")

# 1. Built-in voice by name — no reference clip needed
print("🔊 Generating speech...")

start_time = time.time()
audio = vieneu.infer("[cười] Trời ơi, cái giọng nó tự nhiên mà nó mượt mà dã man, nghe không khác gì người thật luôn. Giờ thì tha hồ mà quẩy content với cả kho giọng nói đa dạng, đủ mọi sắc thái biểu cảm. Mọi người bật loa lên rồi cùng trải nghiệm thử với mình nhé!", voice="Phạm Tuyên")
elapsed_time = time.time() - start_time

vieneu.save(audio, "output.wav")
print("✅ Saved to output.wav")

# Tính RTF (Real-Time Factor)
sample_rate = 48000
audio_duration = len(audio) / sample_rate
rtf = elapsed_time / audio_duration

print(f"\n⏱️  Thời gian xử lý: {elapsed_time:.3f}s")
print(f"🎵 Thời lượng audio: {audio_duration:.3f}s")
print(f"📊 RTF: {rtf:.4f}  ({'nhanh hơn' if rtf < 1 else 'chậm hơn'} real-time {1/rtf:.2f}x)" if rtf > 0 else "")

# List the built-in voices
voices = vieneu.list_preset_voices()
print(f"\n🎙️  {len(voices)} built-in voices available:")
for label, voice_id in voices:
    print(f"  - {label} ({voice_id})")

# 2. ⚡ Batch on GPU: infer_batch() runs many texts in ONE batched forward — same API.
#    On a CUDA GPU the chunks from every text share each forward step (big throughput
#    win). On CPU it still WORKS (no error) — just sequentially, so there's no batch
#    gain. Batch caps at max_batch_size (default 32; tune via Vieneu(max_batch_size=64)
#    or infer_batch(..., batch_size=64), or batch_size=1 to disable). A single long
#    infer() also auto-batches its own chunks. For real-time use, infer_stream() is the
#    streaming twin (GPU: 16 concurrent streams — see "Streaming" below). Uncomment to
#    try (GPU recommended):
#
# import time
# texts = [
#     "Chào cả nhà, hôm nay mình sẽ hướng dẫn các bạn cách cài đặt và sử dụng bộ giọng đọc mới.",
#     "Giọng nghe cực kỳ tự nhiên và truyền cảm, lại có thể chuyển đổi biểu cảm một cách linh hoạt.",
#     "Nếu thấy hữu ích, các bạn nhớ để lại một lượt thích và chia sẻ video này cho mọi người nhé!",
# ] * 10   # 30 texts — enough to fill the batch and really show the GPU throughput win
# t0 = time.time()
# audios = vieneu.infer_batch(texts, voice="Minh Quân Pro")
# elapsed = time.time() - t0
# total_audio = sum(len(a) for a in audios) / 48_000
# print(f"⚡ {len(texts)} texts | audio {total_audio:.1f}s | wall {elapsed:.1f}s | RTF {elapsed/total_audio:.3f}")
# for i, a in enumerate(audios):
#     vieneu.save(a, f"batch_{i}.wav")

v3 Turbo streams frame by frame on both backends. GPU (PyTorch): first audio in ~115 ms and 16 concurrent streams on one RTX 3060 (continuous batching — one CUDA graph serves every infer_stream call, each keeping RTF ≈ 0.5–0.6). CPU (ONNX): first audio in ~140 ms (int8) / ~300 ms (fp32), one stream (two with int8). Just iterate infer_stream:

from vieneu import Vieneu
vieneu = Vieneu()                                  # GPU → PyTorch + stream scheduler; no GPU → ONNX/CPU
for chunk in vieneu.infer_stream("Xin chào các bạn!", voice="Mai Anh"):
    play(chunk)                                   # np.float32 @ 48 kHz — play/write as it arrives

Calling infer_stream from many threads at once is the intended way to serve many listeners on a GPU (Vieneu(max_streams=16) sets the ceiling).

An OpenAI-compatible streaming API (POST /v1/audio/speech, pcm/wav, chunked or SSE — works with the OpenAI SDK, Pipecat, LiveKit, …) is in apps/openai_speech.py:

# Pick ONE of these — all serve http://localhost:8000/v1/audio/speech
uv run python -m apps.openai_speech                                  # from the repo (auto-detects GPU/CPU)
docker compose -f docker/docker-compose.yml --profile api-gpu up     # or: Docker, GPU
docker compose -f docker/docker-compose.yml --profile api-cpu up     # or: Docker, CPU only

📊 docs/streaming.md — every measurement on an RTX 3060 (TTFA / RTF / streams vs max_streams), estimates for smaller GPUs, and the CPU numbers. The older browser demo is still at apps/web_stream.py.

The v3 Turbo engine includes 25 preset voices covering 3 regions (North, Central, South) with diverse genders and speaking characters. list_preset_voices() (and the Web UI / API voice lists) show them in this order:

• ⭐ Editors' picks — the 10 we recommend starting with, hand-selected for naturalness and stability: Adam bựa, Trúc Ly, Anh Khôi, Mai Anh, Minh Quân Pro (default; "Minh Quân" still works as an alias), Thùy Dung, Thiền Tâm Đức, Ngọc Huyền, Quang Sơn, Ngọc Trân
• Northern (Bắc): Minh Đức, Phạm Tuyên, Xuân Vĩnh, Thanh Bình, Ngọc Linh, Đoan Trang, Quỳnh Anh, Mạnh Dũng (+ picks above)
• Central (Trung): Quang Sơn, Ngọc Trân
• Southern (Nam): Adam, Thái Sơn, Thục Đoan, Minh Triết, Mỹ Duyên, Đức Trí, Kim Thanh (+ Thùy Dung)

# Old code — still runs, but `style` is ignored
audio = vieneu.infer("Bản tin sáng nay.", voice="Minh Quân Pro", style="tin_tuc")

# New code — pick the reading character through the voice / reference clip instead
audio = vieneu.infer("Bản tin sáng nay.", voice="Minh Quân Pro")

Inline tags are supported anywhere in the text: [cười] (chuckle), [thở dài] (sigh), [hắng giọng] (clear throat).

audio = vieneu.infer("Nghe hay quá đi [cười]. Để mình nói tiếp [hắng giọng].", voice="Minh Quân Pro")

Clone any voice from a short reference clip. The clip is cleaned up automatically (background noise removed, and trimmed to ≤ 8 seconds) before cloning — keep denoise=True unless your clip is already clean.

audio = vieneu.infer(
    "Đây là giọng được nhân bản tức thì.",
    ref_audio="my_voice.wav",   # a 3–8s reference clip
    denoise=True,               # default; set False if the clip is already clean
)
vieneu.save(audio, "cloned.wav")

Register a reference once with add_voice, then use it by name like a built-in voice.

# Enroll a voice (denoises + extracts the speaker profile once)
vieneu.add_voice("Giọng của tôi", "my_voice.wav")

# Now reuse it anywhere, including the conversation mode
audio = vieneu.infer("Câu này dùng giọng đã lưu.", voice="Giọng của tôi")

# Persist your voices so they load next session
vieneu.save_voices()                 # writes to the default voices file
# vieneu.remove_voice("Giọng của tôi")

# Add a voice you already cleaned yourself → skip denoising
vieneu.add_voice("Giọng sạch", "already_clean.wav", denoise=False)

Get the denoised audio without synthesizing anything (e.g. to inspect or store it):

wav, sr = vieneu.denoise("noisy.wav", out_path="clean.wav")   # 44.1 kHz mono

Note: denoise, add_voice, and voice cloning work on every backend — the torch-free CPU/ONNX install included (the whole cloning pipeline runs on onnxruntime + soxr + kaldi-native-fbank). v3 Nano below clones the same way (its cloning graphs are fetched on first use).

On the same 12th-gen Core i5, Nano runs at RTF 0.22 (16 steps) or 0.11 (8 steps) against Turbo's 0.62 fp32 / 0.37 int8 — about 3× faster than Turbo fp32, with a 282 MB download and ~3 s load time. Full table in §4 Benchmarks.

from vieneu import Vieneu

tts = Vieneu(mode="v3nano")                      # ONNX, CPU, torch-free
audio = tts.infer("Xin chào, mình là giọng đọc của VieNeu Nano.", voice="Minh Quân")
tts.save(audio, "nano.wav")                      # 24 kHz

tts.list_preset_voices()                         # Adam, Ái Hân, Mỹ Duyên, Đức Trí, Hữu Quân, Xuân Tiên, Mai Anh, Trúc Ly, Anh Khôi, Minh Quân, Mạnh Dũng
audio = tts.infer("Bản nhanh cho máy rất yếu.", voice="Ái Hân", steps=8, sway=-1)   # ~2× faster

Knobs: steps (Euler steps, 16 default; 8 ≈ 2× faster, slightly rougher — pair with sway=-1), cfg (classifier-free guidance, 3.0 default; cfg=0 halves compute but hurts intelligibility), speed, seed, threads. Emotion cues [cười] [thở dài] [hắng giọng] work as on Turbo.

apps/openai_speech.py serves POST /v1/audio/speech exactly like OpenAI's TTS endpoint (pcm/wav, chunked body or SSE), so the OpenAI SDK, Pipecat, LiveKit Agents, Vercel AI SDK, … work by changing base_url. Audio streams as it is generated: first chunk in ~115 ms with 16 concurrent streams on an RTX 3060 (continuous batching), ~140–300 ms and 1–2 streams on a CPU.

# Start the server — pick ONE of these three (all listen on http://localhost:8000):
uv run python -m apps.openai_speech                                  # from the repo (auto-detects GPU/CPU)
docker compose -f docker/docker-compose.yml --profile api-gpu up     # or: Docker, GPU container
docker compose -f docker/docker-compose.yml --profile api-cpu up     # or: Docker, CPU container (torch-free)

# Then, in another terminal: measure TTFA / RTF on your machine
uv run python examples/openai_speech_client.py --bench 8

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")
with client.audio.speech.with_streaming_response.create(
    model="vieneu-v3-turbo", voice="Mai Anh", input="Xin chào! Đây là chế độ streaming của VieNeu, phát tới đâu nghe tới đó.", response_format="pcm",
) as r:
    for chunk in r.iter_bytes(4096):   # s16le 48 kHz mono, as it is generated
        play(chunk)

Endpoints: POST /v1/audio/speech, GET /v1/models, GET /v1/voices, POST /v1/voices (clone from an uploaded clip), GET /health. Concurrency is capped per backend (VIENEU_MAX_STREAMS, default 16 on GPU / 1 on CPU) with a small queue and 429 beyond it.

📊 docs/streaming.md — every measurement on an RTX 3060 (TTFA / RTF / streams vs max_streams), estimates for smaller GPUs, the CPU numbers, and tuning notes (e.g. the first request after the GPU idles pays +100–300 ms until it clocks up).

docker compose -f docker/docker-compose.yml --profile gpu up   # or --profile cpu → http://localhost:7860

See docs/Deploy.md for production builds and images.

docker run --gpus all -p 23333:23333 -v huggingface_cache:/root/.cache/huggingface pnnbao/vieneu-tts:latest --tunnel

pip install "vieneu[legacy]"

from vieneu import Vieneu
import os

# Configuration
REMOTE_API_BASE = 'http://your-server-ip:23333/v1'  # Or bore tunnel URL
REMOTE_MODEL_ID = "pnnbao-ump/VieNeu-TTS-v2"

# Initialization (LIGHTWEIGHT - only loads small codec locally)
# Default emotion is "natural" (conversational) - set emotion="storytelling" for storytelling mode
vieneu = Vieneu(mode='remote', api_base=REMOTE_API_BASE, model_name=REMOTE_MODEL_ID, emotion="natural")
os.makedirs("outputs", exist_ok=True)

# List remote voices
available_voices = vieneu.list_preset_voices()
for desc, name in available_voices:
    print(f"   - {desc} (ID: {name})")

# Use specific voice (dynamically select second voice)
if available_voices:
    _, my_voice_id = available_voices[1]
    voice_data = vieneu.get_preset_voice(my_voice_id)
    audio_spec = vieneu.infer(text="Chào bạn, tôi đang nói bằng giọng của bác sĩ Tuyên.", voice=voice_data)
    vieneu.save(audio_spec, f"outputs/remote_{my_voice_id}.wav")
    print(f"💾 Saved synthesis to: outputs/remote_{my_voice_id}.wav")

# Standard synthesis (uses default voice)
text_input = "Chế độ remote giúp tích hợp VieNeu vào ứng dụng Web hoặc App cực nhanh mà không cần GPU tại máy khách."
audio = vieneu.infer(text=text_input)
vieneu.save(audio, "outputs/remote_output.wav")
print("💾 Saved remote synthesis to: outputs/remote_output.wav")

# Zero-shot voice cloning (encodes audio locally, sends codes to server)
if os.path.exists("examples/audio_ref/example_ngoc_huyen.wav"):
    cloned_audio = vieneu.infer(
        text="Đây là giọng nói được clone và xử lý thông qua VieNeu Server.",
        ref_audio="examples/audio_ref/example_ngoc_huyen.wav",
        ref_text="Tác phẩm dự thi bảo đảm tính khoa học, tính đảng, tính chiến đấu, tính định hướng."
    )
    vieneu.save(cloned_audio, "outputs/remote_cloned_output.wav")
    print("💾 Saved remote cloned voice to: outputs/remote_cloned_output.wav")

docker run --gpus all pnnbao/vieneu-tts:serve --model pnnbao-ump/VieNeu-TTS-0.3B --tunnel

tts = Vieneu(mode="v3turbo", backbone_repo="finetune/output/my_voice/merged")

Every speed and latency number quoted in this README, measured on one machine so they are comparable: RTX 3060 12 GB · Intel Core i5 12th gen (6 P-cores, 12 threads) · Windows 11 · torch 2.8 + cu128 (bf16) · ONNX Runtime 1.24 (fp32/int8, 6 threads) · vieneu 3.8.x · September 2026. Reproduce with the snippets in §2 and examples/openai_speech_client.py --bench N.

RTF = generation time ÷ audio duration — lower is faster, < 1 is faster than real-time (0.02 = 50× real-time). TTFA = time to the first audio chunk when streaming.

The GPU is ~25–50× faster than the CPU on bulk work; on a single short sentence the gap shrinks to ~5× because the work is too small to fill the GPU.

Streams ≠ users: a stream lives only while a reply plays, so 16 streams ≈ 45–80 active voice-chat users. Set max_streams to the real load — every reserved slot adds ~2.5 ms to each codec call. After a few idle seconds the GPU parks at low clocks and the first request pays +100–300 ms (lock clocks with nvidia-smi -lgc or "prefer maximum performance"). VRAM: 1.1 GB peak at 16 streams. Estimates for other GPUs and the full method: docs/streaming.md.

v3 Turbo already clones a voice from a clip of a few seconds. Fine-tune with LoRA when you need a tighter match than cloning, a specific reading style (storytelling, news, narration…), or better reading on your own domain. A LoRA teaches one voice: about 10–30 minutes of clean audio from one speaker (one LoRA per voice if you need several). Only a few million parameters are trained, so a ~6 GB GPU is enough.

uv sync --extra finetune
uv run python finetune/prepare_dataset.py --dataset-dir finetune/dataset   # CPU, torch-free; all clips = one speaker
uv run python finetune/train_lora.py --data finetune/dataset/train.parquet --run my_voice --merge
uv run python finetune/make_voice.py --audio ref.wav --name "My voice" --out finetune/output/my_voice/merged

tts = Vieneu(mode="v3turbo", backbone_repo="finetune/output/my_voice/merged")   # or your Hub repo
audio = tts.infer("Xin chào!", voice="My voice")        # packed voice — no reference audio needed

The merged model speaks that one voice from its packed speaker embedding (no reference audio) and keeps the v3 Turbo API (presets, batching, streaming) on the PyTorch/GPU backend; use the base model to clone other voices. Data layout, options and tips: finetune/README.md.

⛔ Deprecated models are no longer updated and are kept only for existing deployments; new projects should use v3 Turbo (and v3 once released).

• VieNeu-TTS v3 Turbo (on-device, personal use): from-scratch 48 kHz architecture — preset voices, instant voice cloning, emotion cues, batched generation, multi-speaker conversation, frame-level streaming; torch-free on CPU.
• VieNeu-TTS v3 Nano (preview): 48M flow-matching model for weak CPUs / edge devices — 11 preset voices + cloning, torch-free.
• LoRA fine-tuning for v3 Turbo — train your own voice or reading style on one consumer GPU.
• VieNeu-TTS v3 Turbo GPU streaming server: OpenAI-compatible /v1/audio/speech with continuous batching on one CUDA graph — first audio ~115 ms, 16 concurrent streams on an RTX 3060, Docker profiles api-gpu / api-cpu (docs/streaming.md).
• VieNeu-TTS v3 (GPU, server release): the full v3 model for API / server deployment — finalized quality, stable emotion control, more voices.
• Mobile SDK: official Android / iOS deployment.

• Hugging Face: pnnbao-ump
• Discord: Join our community
• Facebook: Pham Nguyen Ngoc Bao
• License: Apache 2.0 (Free to use).

@misc{vieneutts2026,
  title        = {VieNeu-TTS: Advanced Vietnamese Text-to-Speech with Instant Voice Cloning},
  author       = {Pham Nguyen Ngoc Bao},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/pnnbao-ump/VieNeu-TTS}}
}

Thanks to all the amazing people who have contributed to this project!

This project uses neucodec (v1/v2) and MOSS-Audio-Tokenizer-Nano (v3 Turbo) for audio coding, and sea-g2p for text normalization and phonemization.

Made with ❤️ for the Vietnamese TTS community