Add voice library folder (voice_dir) support for name-based voice cloning for OpenAI API (audiocpp_server)

[jasonchen31]

Status: Ready for review
Branch base: main
Files changed: 6 (5 modified, 1 added)

Summary

The WebUI's TTS/Voice Cloning tab lets a user pick a built-in voice from webui/voice/
(a .wav file plus its reference transcription in webui/voice/prompt_text) and clone
that voice. Today this works only because the Python client resolves the dropdown
name into a server-side file path (voice_ref) plus a transcription (reference_text)
and sends both to POST /v1/audio/speech.

A third-party OpenAI-compatible client cannot clone by name: sending the standard
OpenAI field "voice": "jenny" does not clone webui/voice/jenny.wav, because the
C++ server has no knowledge of the webui voice library.

This PR teaches audiocpp_server to resolve a voice name against a configured voice
library directory (voice_dir) containing .wav files and a prompt_text mapping file,
so {"voice": "jenny"} performs the same voice cloning as the WebUI tab — with no
voice_ref / reference_text required from the caller.

Changes

app/server/config.h — new config field

// Voice library shared across all TTS models: *.wav files plus a `prompt_text`
// mapping file (<basename>|<transcript>). A request `voice` name that is not a
// model preset resolves to <voice_dir>/<name>.wav as the cloning reference.
std::optional<std::filesystem::path> voice_dir;

app/server/config.cpp — parse and resolve voice_dir

Parses the server-level voice_dir key (must be a string). Relative paths resolve
against the config file's directory, same as model_spec_override / model paths.

app/server/runtime.cpp — voice-library resolution

• build_speech_request(): when voice is not a configured model preset, the
  server checks <voice_dir>/<name>.wav. If the file exists, it is loaded as the
  cloning reference (voice.speaker->audio) and the matching transcript from
  prompt_text is injected as reference_text — but only if the request does not
  already set reference_text.
• load_voice_library_text(): file-local helper parsing prompt_text (one
  <basename>|<transcript> line per voice, split on first |, name trimmed).
• handle_voices(): GET /v1/audio/voices now also lists voice_dir/*.wav
  basenames (deduplicated against preset names by the existing sort+unique).

webui/webui.py — wire the library into the managed server

• _write_temp_config(): the temp server config now emits
  "voice_dir": VOICE_DIR (absolute path), so a webui-launched server exposes the
  same voices the tab offers to third-party OpenAI-compatible clients.
• API help text (EN + zh): documents cloning by name, e.g. "voice": "jenny".

app/server/README.md — documentation

New "Voice library (voice_dir)" section documenting the config, the prompt_text
format, and the voice resolution precedence.

examples/voice_dir_example.json — runnable example config (new)

Minimal config demonstrating voice_dir with a TTS model.

Voice resolution precedence (unchanged semantics + new step)

1. voice_ref present → load that file. Always wins.
2. voice matches a configured model voice_presets preset → preset wins.
3. voice matches <voice_dir>/<name>.wav → voice-library clone (new).
4. Otherwise → voice used as model-native cached voice id (previous behavior).

Implementation note: the library lookup is merged into the existing voice →
cached_voice_id block (gating the fallback on !voice_library_resolved) rather
than inserted as a separate preceding block. As originally sketched, the untouched
cached_voice_id block would still run after a successful library hit and set
cached_voice_id alongside audio — pocket_tts hard-errors when both a preset
name and clone audio are set. The merge preserves the intended precedence exactly:
a name that resolves to a wav clones; anything else falls through unchanged.

Usage

1. Configure the voice library

Add voice_dir to your server config:

{
  "host": "127.0.0.1",
  "port": 8020,
  "backend": "cuda",
  "device": 0,
  "threads": 1,
  "lazy_load": true,
  "voice_dir": "/absolute/path/to/webui/voice",
  "models": [
    {
      "id": "qwen3-tts",
      "family": "qwen3_tts",
      "path": "/absolute/path/to/models/Qwen3-TTS",
      "task": "tts",
      "mode": "offline"
    }
  ]
}

The directory holds one .wav per built-in voice and a prompt_text file with one
<basename-without-extension>|<transcript> line per voice:

jenny|This is the longer voice from Jenny Neutral.
原神-甘雨|但只要最后落在具体的「人」身上，那，我可以想办法。

See examples/voice_dir_example.json for a complete runnable example (paths relative
to examples/, so ../../webui/voice resolves to the repo's voice folder).

2. List available voices

curl 'http://127.0.0.1:8020/v1/audio/voices?model=qwen3-tts'
# {"voices": ["ana", "andrew", "aria", "demo_01_man", "jenny", ..., "原神-甘雨", ...]}

3. Clone by name (the core fix)

curl http://127.0.0.1:8020/v1/audio/speech -H "Content-Type: application/json" -o out.wav \
  -d '{"model": "qwen3-tts", "input": "Hello from audio.cpp.", "voice": "jenny"}'

No voice_ref / reference_text needed — the transcript is injected from
prompt_text automatically. Non-ASCII names work too:

curl http://127.0.0.1:8020/v1/audio/speech -H "Content-Type: application/json" -o out.wav \
  -d '{"model": "qwen3-tts", "input": "你好，audio.cpp。", "voice": "原神-甘雨"}'

Verification (performed)

Built with cmake -S . -B build -DENGINE_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=61
and cmake --build build --config Release -j — clean, audiocpp_server links.

Check	Result
GET /v1/audio/voices?model=qwen3-tts lists library voices	25 voices incl. jenny, andrew, 原神-甘雨
voice + voice_ref both present	voice_ref wins (unchanged)
Explicit reference_text in request	overrides the library transcript
Unknown voice name (no wav)	falls through to cached voice id (unchanged)
voice_dir unset	behavior exactly as before
WebUI regression (sends voice_ref+reference_text)	unchanged path

[0xShug0]

@jasonchen31 Thank you for adding this great feature! Could you update your code based on the dev branch? We’re currently testing the official lightweight UI (#185), and the Python Web UI will be removed eventually. Instead of adding voice library support to the Python Web UI, it would be better to add it to the official UI.

[mirek190]

Yo jasonchen31 ;)

Good work.

WebUi has already support for saving voices + text.
But that implementation is good to a clean server.

[0xShug0]

Thank you @jasonchen31. Merged!

[jasonchen31]

@0xShug0 Thanks for all your great work. This is a nice project!