Add support for inflect_v2 TTS model

[JanWerder]

Summary

This PR adds native GGML support for the Apache-2.0 licensed Inflect Micro v2 and Inflect Nano v2 models as the new inflect_v2 community TTS family.

Both variants use the same configurable runtime and provide:

• offline English TTS;
• one fixed synthetic voice;
• 24 kHz mono output;
• deterministic seed-based generation;
• speaking-rate and latent-variation controls;
• punctuation-aware long-form synthesis;
• native CPU and CUDA GGML execution;
• FP32 SafeTensors packages and standalone GGUF support.

ONNX Runtime and PyTorch are not used during synthesis. The official FP32 ONNX exports are used only as pinned conversion sources and numerical parity references.

Implementation

The new community model is implemented under src/community_models/inflect_v2/ and includes:

• a model-specific English normalization and phoneme frontend;
• dynamic eSpeak-ng loading;
• the 178-symbol VITS inventory with blank insertion;
• relative-window text attention;
• deterministic duration prediction and monotonic expansion;
• seeded Gaussian latent sampling controlled by variation;
• four reverse residual-coupling flows;
• the complete VITS/HiFiGAN-style waveform decoder;
• punctuation-aware chunking, pauses, and 5 ms edge fades;
• bounded duration- and decoder-graph caches;
• strict model-configuration and 302-tensor inventory validation.

Micro and Nano select different validated model dimensions while sharing the same runtime implementation. Unexpected variants, tensor names, shapes, or non-FP32 tensors are rejected while loading.

The loader is registered as:

• family: inflect_v2
• task: tts
• mode: offline
• language: en
• capability: long_form

Both the legacy model specification and the current model_specs_v1 catalog are included.

Request options

Option	Range	Default	Behavior
speaking_rate	0.5–2.0	1.0	Maps directly to Inflect's speed control
variation	0.0–1.0	0.667	Scales the seeded Gaussian latent variation
seed	non-negative integer	0	Long-form chunks use seed + chunk_index
text_chunk_mode	word_budget	word_budget	Punctuation-aware long-form mode
text_chunk_size	positive integer	280	Maximum Unicode codepoints per chunk

The model intentionally exposes no language selection, speaker selection, voice cloning, or streaming controls.

eSpeak-ng frontend

Inflect requires the English eSpeak-ng frontend before neural inference.

The runtime dynamically opens the user-provided or system eSpeak-ng shared library:

• Linux: libespeak-ng.so.1 or libespeak-ng.so
• macOS: libespeak-ng.dylib or libespeak-ng.1.dylib
• Windows: espeak-ng.dll or libespeak-ng.dll

It initializes eSpeak-ng once per process, selects en-us, requests IPA with stress and preserved punctuation, and serializes access through a shared mutex because the eSpeak C API uses global process state.

Optional explicit paths are available through:

• inflect_v2.espeak_library_path
• inflect_v2.espeak_data_path

A missing library, invalid data directory, or missing en-us voice fails session creation with an actionable error.

No eSpeak binary, source, voice data, or Python package is vendored, downloaded by the model manager, embedded in GGUF, or distributed by this PR.

On Windows, users without a system installation can use espeakng-loader==0.2.4 as an external source for the DLL and data directory:

$espeakPaths = uv run --with espeakng-loader==0.2.4 python -c `
  "import espeakng_loader; print(espeakng_loader.get_library_path()); print(espeakng_loader.get_data_path())"
$espeakLibrary = $espeakPaths[0]
$espeakData = $espeakPaths[1]

Python is only used here to locate the external files. Synthesis remains inside the native audio.cpp process.

Model installation and provenance

The model manager downloads pinned official ONNX snapshots:

Package	Repository	Revision
inflect_micro_v2	owensong/Inflect-Micro-v2-ONNX	91b1ab6432323064ec0e8e9704d92fcecd24855f
inflect_nano_v2	owensong/Inflect-Nano-v2-ONNX	df3627e44c26192714c9dcba6b8e9b47e7a6e3d9

The converter validates both ONNX initializer inventories and dimensions, then writes one FP32 model.safetensors plus the native configuration.

uv run --with onnx --with safetensors python \
  tools/model_manager.py install inflect_micro_v2 --models-root models

uv run --with onnx --with safetensors python \
  tools/model_manager.py install inflect_nano_v2 --models-root models

Generated weight files:

Variant	Parameters	SafeTensors size
Micro v2	9,356,513	37,411,460 bytes
Nano v2	3,966,721	15,867,292 bytes

PyTorch is not required for installation or inference.

Standalone FP32 GGUF packages embed the model specification and config.json. eSpeak-ng remains an external runtime dependency.

Licensing

Component	License	Distribution in this PR
audio.cpp and this integration	Apache-2.0	Source changes in this repository
Inflect v2 code and weights	Apache-2.0	Weights downloaded from the official pinned releases
Official Inflect ONNX exports	Apache-2.0	Downloaded and converted locally
eSpeak-ng	GPL-3.0	External runtime dependency; not vendored or redistributed

The model releases explicitly identify the original Inflect code and weights as Apache-2.0. eSpeak-ng remains under its own GPL-3.0 license.

Dynamic loading is the technical packaging boundary used by this implementation; this PR does not claim that every possible downstream combined distribution is automatically license-compliant. Distributors remain responsible for satisfying the licenses of the components they package.

Numerical parity

Frontend goldens use phonemizer==3.3.0 and espeakng-loader==0.2.4.

Fixed-latent CPU output was compared with the official FP32 ONNX graphs:

Variant	Mean absolute error	Maximum error	Correlation
Micro v2	1.54e-5	9.16e-5	0.999999976
Nano v2	1.53e-5	9.16e-5	0.999999964

Fixed-input CUDA comparisons produced:

Variant	Mean absolute error	Correlation
Micro v2	2.81e-4	0.999892
Nano v2	3.39e-4	0.999832

The component validation also covered deterministic seeds, variation and rate controls, long-form output without NaN/Inf, the 4000-frame rejection, and standalone FP32 GGUF loading.

Real-time performance

All tested paths ran faster than real time (RTF < 1.0). For the 48.544-second Micro v2 long-form output, Windows CPU reached 0.246 / 0.237 RTF for cold/repeated requests, CUDA reached 0.0138 / 0.0116, and WSL2 CPU reached 0.257 / 0.272. Nano v2 produced 48.512 seconds of audio under WSL2 with observed RTFs ranging from 0.139 to 0.475; the wider range reflects WSL host-scheduling variance. The slowest measured run was still approximately 2.1× faster than real time.

Platform coverage

Platform/backend	Coverage
Windows CPU	Release build, frontend tests, Micro runtime and long-form path
Windows CUDA 13.3	Release build, fixed-input parity, Micro runtime and long-form path
WSL2 Debian 13 CPU	GCC 14.2 Release build, system eSpeak-ng 1.52.0, Micro and Nano runtime and long-form paths
Native Linux CPU	Not separately tested outside WSL2
Native Linux/WSL CUDA	Not runtime-tested
Vulkan	Not practically runtime-validated
Metal	Not practically runtime-validated

CPU and CUDA are deterministic within the same backend and configuration.
Bit-identical output across different backends is not claimed.

CUDA deliberately keeps duration alignment on CPU so small TF32 differences cannot alter monotonic expansion. The flows and waveform decoder execute on CUDA.

Reproduction

Windows CPU and CUDA builds:

cmake -S . -B build-inflect -G Ninja `
  -DCMAKE_BUILD_TYPE=Release -DENGINE_BUILD_TESTS=ON
cmake --build build-inflect --parallel 8 `
  --target audiocpp_cli inflect_v2_frontend_test

cmake -S . -B build-inflect-cuda -G Ninja `
  -DCMAKE_BUILD_TYPE=Release -DENGINE_BUILD_TESTS=OFF -DGGML_CUDA=ON
cmake --build build-inflect-cuda --parallel 8 --target audiocpp_cli

Linux/WSL2 CPU build:

sudo apt install build-essential cmake python3 espeak-ng libespeak-ng1

cmake -S . -B build-inflect-wsl \
  -DCMAKE_BUILD_TYPE=Release -DENGINE_BUILD_TESTS=ON

cmake --build build-inflect-wsl --parallel "$(nproc)" \
  --target audiocpp_cli inflect_v2_frontend_test

Representative system-eSpeak run:

build-inflect-wsl/bin/audiocpp_cli \
  --task tts --family inflect_v2 \
  --model build-inflect/models/Inflect-Micro-v2 \
  --backend cpu \
  --text "Hello from Inflect Micro version two." \
  --seed 0 --out inflect.wav

Long-form path test:

python3 tools/audiocpp_cli/run_audiocpp_cli_path_tests.py \
  --cases tools/audiocpp_cli/audiocpp_cli_longform_tts_clone_cases.json \
  --only inflect_v2_tts_longform \
  --audiocpp-cli-bin build-inflect-wsl/bin/audiocpp_cli \
  --model-path build-inflect/models/Inflect-Micro-v2 \
  --backend cpu --threads 8 \
  --out-root build-inflect-wsl/validation/inflect-v2-wsl --log

Replace the model path with Inflect-Nano-v2 to exercise Nano.

Tests

The following checks pass:

inflect_v2_frontend_test        passed
model_spec_system_test          passed
relevant CTest result           2/2 passed
loader catalog self-test        2/2 passed
loader/catalog synchronization  passed
Micro WSL2 long-form path       passed
Nano WSL2 long-form path        passed
git diff --check                passed

The catalog check reports the existing unrelated warning that bs_roformer has no installable standalone ModelPackage; installable catalog families still match all registered loaders.

Known limitations

• English-only offline TTS with one fixed voice.
• No speaker selection, voice cloning, or streaming.
• FP32 is the only supported weight format.
• eSpeak-ng and its English data are required externally at runtime.
• Long input is synthesized as punctuation-aware chunks rather than one unlimited model pass.
• The maximum expanded latent length is 4000 frames.
• Native Linux CUDA, Vulkan, and Metal have not been practically validated.
• Windows WDDM did not provide a reliable per-process CUDA VRAM measurement.
• CPU and CUDA determinism is scoped to the same backend and configuration.

[JanWerder]

@0xShug0 The eSpeak part is a bit different from the other models, please let me know if you are okay with that.

[0xShug0]

@JanWerder Thanks! I think runtime-loading eSpeak-ng is a reasonable workaround here. I had considered writing a minimal built-in eSpeak-compatible frontend, but that would be a much larger effort and probably not worth it for this integration. This approach also gives us a path to integrate Kokoro and KittenTTS. I had previously set those (and inflect micro v2) aside because of the frontend problem (the code is ready if you’re interested).

Could you take a look at my earlier Inflect v2 attempt on preview/inflect-micro-v2 and compare performance?https://github.com/0xShug0/audio.cpp/tree/preview/inflect-micro-v2. Some of the optimizations there may be useful for your implementation.

[0xShug0]

BTW, move assets/resources/inflect_v2/misaki_en/ into the model weights directory before running the model.

Another minor issue is if eSpeak is missing, a bad task or bad option may fail with "missing eSpeak" first instead of the real validation error. This is because frontend_ is a direct member and is constructed before the constructor body validates task/options.

[0xShug0]

@JanWerder Please check #128. The example is Confucius4-TTS.

[JanWerder]

Thanks for the feedback. I’ve addressed all three points:

• Benchmarked against preview/inflect-micro-v2 after placing the Misaki assets correctly and adopted the relevant runtime optimizations. On the same 6,026-character CPU workload, RTF improved from 0.3095 to 0.2888 (~6.7% lower). Which is faster, but inflect is very fast anyways 😄
• Task and session-option validation now runs before eSpeak initialization, with regression tests added.
• Migrated Inflect to the schema-v1 conventions from Using Model Spec v1 for New Models #128: one canonical model spec, the generic spec-backed loader, and an embedded-spec standalone GGUF. The duplicate spec and custom loader files were removed.

Micro and Nano were revalidated on Windows CPU/CUDA and Debian under WSL2.

[0xShug0]

@JanWerder Merged! Thank you for contributing the new model!