audio.cpp is a high-performance C++ audio inference framework built on top of ggml, designed to make modern local audio models practical, portable, and fast.

It is built for real end-to-end execution rather than one-off model demos: the same runtime powers TTS, voice cloning, voice conversion, ASR, diarization, VAD, source separation, alignment, codec-style models, and higher-level workflows through a common framework surface.

Highlights:

• Parity. Strong parity tooling against Python reference paths.
• Performance. Performance-focused execution, reusable sessions, and batch-style offline inference. Optimized for CUDA.
• Portability. A portable native stack centered on ggml, with CLI and server entry points instead of Python-only deployment paths.
• Pipelines. Experimental JSON pipeline support for higher-level multi-step workflows.
• Audio Utilities. Built-in denoise, enhancement, resampling, and STFT/ISTFT utilities for real production-style task paths.

The goal of the framework is to provide highly optimized, reusable building blocks for audio-related models, so new model integrations can be brought up faster, shared components can be improved once and benefit many families, and real end-to-end inference paths can stay efficient, maintainable, and portable.

Current model status in the framework:

• released: The model is fully wired into the broader framework surface and ready for normal use.
• integration: The model is end-to-end working and optimized, but not yet fully wired into the broader framework surface. Those models are expected to be added to the broader framework surface gradually over time.
• optimization: The model is end-to-end working, but still needs more optimization work before it should be treated like a released or integration-level path.

PocketTTS language selection is a model-load option. When the model path points at the PocketTTS root, the loader uses english unless you pass --load-option language=<name>.

On Linux, use a normal CMake build directory such as build/.

For single-config generators, the default build type is RelWithDebInfo.

That default configure is a CPU build unless you enable an accelerator backend explicitly.

Common Linux configure examples:

CPU-only:

cmake -S . -B build

CUDA:

cmake -S . -B build -DENGINE_ENABLE_CUDA=ON

Vulkan:

cmake -S . -B build -DENGINE_ENABLE_VULKAN=ON

Build the CLI and server from the configured tree:

cmake --build build --parallel --target audiocpp_cli --target audiocpp_server

If you use an environment manager or custom toolchain, activate it before running the commands above.

The optional Linux helper script wraps the same CMake flow and uses aligned build directory names:

• build/linux-cuda-release
• build/linux-vulkan-release
• build/linux-cpu-release

Examples:

scripts/build_linux.sh --backend cuda --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend vulkan --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend cpu --target audiocpp_cli --target audiocpp_server

Use --build-dir <dir> only when you intentionally want a custom output directory.

The recommended native Windows build is command-line only:

• Visual Studio Build Tools 2022 or newer with the C++ desktop workload
• MSVC x64 compiler, Windows SDK, CMake, Ninja, and MSVC OpenMP components
• Official NVIDIA CUDA Toolkit installed under C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\... for CUDA builds

Use MSVC cl.exe as the compiler. For CUDA builds, cl.exe is also used as the CUDA host compiler. Native Windows nvcc does not support clang-cl as its host compiler, and the Visual Studio IDE is not required.

From PowerShell:

powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\build_windows.ps1

CPU-only:

.\scripts\build_windows.ps1 -Preset windows-cpu-release -Target audiocpp_cli

From cmd.exe:

scripts\build_windows.cmd

If GNU Make is available on Windows:

make -f Makefile.windows cpu JOBS=16
make -f Makefile.windows cuda JOBS=16

The Windows script configures build/windows-cuda-release by default and builds audiocpp_cli. CUDA presets enable CUDA, CUDA graphs, OpenMP, Ninja, /utf-8, /EHsc, MSVC OpenMP SIMD support with /openmp:experimental, and the same portable CPU optimization baseline used for the Windows CUDA path. The CPU preset uses the same MSVC/Ninja/OpenMP setup without requiring CUDA. CUDA presets auto-detect the local GPU CUDA architecture when nvidia-smi is available.

Useful variants:

.\scripts\build_windows.ps1 -Target audiocpp_server -Jobs 16
.\scripts\build_windows.ps1 -Preset windows-cpu-release -Target audiocpp_cli
.\scripts\build_windows.ps1 -Preset windows-cuda-debug -Target audiocpp_cli
.\scripts\build_windows.ps1 -ConfigureOnly
.\scripts\build_windows.ps1 -CudaArchitectures 120a-real

If multiple Build Tools installations are present, pass the one you want explicitly:

.\scripts\build_windows.ps1 -VsInstall "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools"

The built CLI is written to the selected preset directory:

build\windows-cpu-release\bin\audiocpp_cli.exe
build\windows-cuda-release\bin\audiocpp_cli.exe

Build options:

• For single-config generators, the recommended config is RelWithDebInfo
• For multi-config generators, choose the configuration at build time
• Backend and feature options are independent from build type

The main CLI binary is:

build/bin/audiocpp_cli

High-level command shape:

audiocpp_cli --task <task> --family <family> --model <path> --backend <backend> --mode <mode> [options]

Core selectors:

• --task vad|asr|diar|sep|tts|clon|vc|s2s|align|vdes|spk|svc
• --family <name>
• --model <path>
• --backend cpu|cuda|vulkan|metal|best
• --mode offline|streaming

Examples:

Text-to-speech:

build/bin/audiocpp_cli \
  --task tts \
  --family pocket_tts \
  --model /path/to/model \
  --backend cuda \
  --text "audio.cpp is running PocketTTS locally." \
  --voice-ref assets/resources/sample.wav \
  --out build/out/pocket_tts.wav

PocketTTS with another language and a built-in voice:

build/bin/audiocpp_cli \
  --task tts \
  --family pocket_tts \
  --model /path/to/models/pocket-tts \
  --backend cuda \
  --load-option language=spanish \
  --text "Hola, esta es una prueba corta de Pocket TTS." \
  --voice-id alba \
  --out build/out/pocket_tts_spanish.wav

ASR:

build/bin/audiocpp_cli \
  --task asr \
  --family qwen3_asr \
  --model /path/to/model \
  --backend cuda \
  --audio assets/resources/sample_16k.wav

Voice conversion:

build/bin/audiocpp_cli \
  --task vc \
  --family seed_vc \
  --model /path/to/model \
  --backend cuda \
  --audio assets/resources/a.wav \
  --voice-ref assets/resources/b.wav \
  --out build/out/seed_vc.wav

Useful CLI features:

• --help with --task shows task-oriented help
• --help with --model shows model-owned options
• --inspect prints discovered configs, weights, and capabilities
• --batch-text-file <txt> runs one offline request per non-empty line
• --batch-audio-dir <dir> runs one offline request per .wav
• --request-sequence <json> runs a multi-request offline session
• --pipeline <json> runs a workflow instead of a raw task
• --log streams framework logs to stdout
• --log-file <path> streams framework logs to a file in real time

Pipelines are an experimental JSON workflow feature for chaining multiple model and audio-processing steps behind one CLI command. A pipeline can define default inputs, let users override them with --workflow-input key=value, split long media into model-sized chunks, merge text or audio outputs back together, write intermediate artifacts under --out-dir, and copy the declared final_audio to --out.

This is the higher-level layer for production-style audio jobs: redubbing, batch cleanup, long-form narration, voice conversion, source-separation workflows, transcription-plus-alignment, and future workflows that combine translation, diarization, denoise, enhancement, or review steps as those model surfaces are wired into the framework.

The included same-language speech redub pipeline transcribes long speech in chunks with Qwen3 ASR, merges the transcript, then regenerates the speech in a target reference voice with Qwen3 TTS. The default test input assets/resources/speech.wav is about 418 seconds long and was generated from an 8,091-character speech text, so it exercises long-audio split and merge behavior rather than a short one-shot request:

build/bin/audiocpp_cli \
  --pipeline assets/pipeline/speech_redub.json \
  --backend cuda \
  --out-dir build/out/speech_redub_pipeline \
  --out build/out/speech_redub_pipeline.wav

Override the source speech or target voice without editing the JSON:

build/bin/audiocpp_cli \
  --pipeline assets/pipeline/speech_redub.json \
  --backend cuda \
  --workflow-input source_audio=/path/to/speech.wav \
  --workflow-input target_voice=/path/to/voice.wav \
  --workflow-input language=English \
  --out-dir build/out/speech_redub_pipeline \
  --out build/out/speech_redub_pipeline.wav

The repository also ships a model manager at tools/model_manager.py for downloading supported model packages into the framework expected models/ layout.

Dependencies:

• Python 3
• torch
• safetensors
• PyYAML
• network access to the upstream model sources

The tool supports three main commands:

• list shows the available package ids
• info shows the target layout, required files, and install source for one package
• install downloads or converts one package into a models root

Recommended top-level install packages:

Yes means Hugging Face has a ready-to-use repo that the framework can download as-is. No means the tool must assemble, convert, or post-process files before the framework can use them.

Examples:

List packages:

python3 tools/model_manager.py list

Show one package:

python3 tools/model_manager.py info qwen3_tts_1_7b_base

Install one package into the default models/ directory:

python3 tools/model_manager.py install qwen3_tts_1_7b_base

Install into a custom models root:

python3 tools/model_manager.py install vevo2 --models-root /path/to/models

Replace an existing installed package:

python3 tools/model_manager.py install pocket_tts --overwrite

Some packages are direct snapshots, while others are composite installs or local-file utilities. Use info first when you want to inspect the expected target directory, required files, or whether a package needs extra local source inputs such as --source-file or --source-dir.

Run a local-file utility:

python3 tools/model_manager.py info voxcpm2_audiovae
python3 tools/model_manager.py install voxcpm2_audiovae --source-file models/VoxCPM2/audiovae.pth --models-root models --overwrite

The server binary is:

build/bin/audiocpp_server

Build:

cmake --build build --parallel --target audiocpp_server

Create a config file with your own model paths:

cat > server.json <<'JSON'
{
  "host": "127.0.0.1",
  "port": 8080,
  "device": 0,
  "threads": 1,
  "models": [
    {
      "id": "pocket-tts",
      "family": "pocket_tts",
      "path": "/path/to/models/pocket-tts",
      "task": "tts",
      "mode": "offline",
      "load_options": {
        "language": "english"
      },
      "session_options": {
        "language": "english"
      }
    },
    {
      "id": "qwen3-asr",
      "family": "qwen3_asr",
      "path": "/path/to/models/Qwen3-ASR-0.6B",
      "task": "asr",
      "mode": "offline"
    }
  ]
}
JSON

Start:

build/bin/audiocpp_server --config server.json

The server exposes:

• GET /health
• GET /v1/models
• POST /v1/audio/speech
• POST /v1/audio/transcriptions
• POST /v1/tasks/run

More server examples are in app/server/README.md.

The repository includes both framework-level parity validation and app-level end-to-end path checks. At a high level, the flow is:

The main harness under tests/ is tests/warmbench.py. It is used for long-lived multi-request validation, parity checks against Python references, and performance-oriented session reuse scenarios. The tests/ tree also contains model-specific C++ and Python warmbench entrypoints that warmbench.py coordinates.

The main app-facing test tooling under tools/ is tools/audiocpp_cli/run_audiocpp_cli_path_tests.py. It drives audiocpp_cli through cataloged offline and streaming-shaped cases, verifies expected outputs such as audio or JSON artifacts, and is useful for checking real user-facing request paths rather than just lower-level model components. The streaming-shaped coverage here refers to the CLI/request path surface; it should not be read as a claim that streaming inference is broadly supported across the framework today.

The Python-reference side of these tests usually requires more time-consuming setup than the C++ path because different models rely on different Python reference repos and dependency stacks. In practice, the framework-side tooling is fast to iterate on once models are installed, while Python parity runs often need extra environment preparation before they are ready.

All performance metrics in this section were measured on Ubuntu with the CUDA backend on an NVIDIA GeForce RTX 5090. The Python-relative one-shot and long-lived-session comparisons come from direct framework/runtime API benchmark calls, not from audiocpp_cli; CLI path tests are separate and include app-layer request parsing, output writing, and other user-facing overhead.

Absolute RTF depends on the GPU and system setup, but the Python-relative speedups are real because audio.cpp and the matching Python reference paths were measured on the same CUDA setup.

audio.cpp already shows some genuinely exciting wins against the matching Python reference paths, especially on the TTS side, even when using the original model weights without quantization. The headline win is wall time: several TTS paths run 1.8x-5.0x faster than Python while cutting end-to-end latency by 45%-80%.

• In one-shot runs, several TTS-family models already land far ahead of Python: pocket tts is 3.68x faster with 72.80% less wall time, miotts is 2.73x faster with 63.39% less wall time, moss tts is 2.33x faster with 57.07% less wall time, qwen3 tts is 1.83x faster with 45.34% less wall time, and vevo2 is 5.03x faster with 80.11% less wall time.
• In long-lived-session runs, where the same loaded session serves multiple requests in sequence, the gains stay strong: pocket tts is 3.22x faster with 68.91% less wall time, qwen3 tts is 2.74x faster with 63.47% less wall time, moss tts is 2.66x faster with 62.35% less wall time, miotts is 2.28x faster with 56.22% less wall time, and vevo2 is 1.75x faster with 42.72% less wall time.
• In long-form runs on the shared 6,026-character, 1,028-word passage, the strongest Python-relative wins still show up clearly: pocket tts is 3.15x faster with 68.23% less wall time, qwen3 tts is 3.06x faster with 67.33% less wall time, vevo2 is 1.77x faster with 43.51% less wall time, and chatterbox is 1.58x faster with 36.83% less wall time.
• These long-lived-session numbers are especially important for real applications, because they reflect the common case where model load, cached state, and reusable runtime setup are amortized across many requests.
• Bars below the 1.0x line are useful too: they spotlight exactly where more optimization work is still worth doing.

The figures report Python wall time / audio.cpp wall time. The 1.0x line means equal wall time; bars above 1.0x mean audio.cpp is faster than Python, and bars below 1.0x mean it is slower.

For TTS-family models, the measured one-shot RTF is:

For long-form TTS tests, each run uses the same 6,026-character, 1,028-word input text. The measured RTF is:

Many model sessions expose quantization through --session-option <family>.weight_type=<mode>, and some families also expose more specific knobs such as ...conv_weight_type, ...talker_weight_type, or ...speech_decoder_weight_type. The exact supported modes are model-specific rather than global.

Example:

build/bin/audiocpp_cli --task tts --family qwen3_tts --model /path/to/model --session-option qwen3_tts.weight_type=q8_0

In practice, lower precision and quantized modes should be treated as model- and route-specific optimizations rather than universally safe defaults.

• Safety. Quantization may not be safe on every path even when a model parser accepts the option. For example, in our ACE-Step 1.5 checks, lower-precision runs could fail at runtime with ACE-Step planner masked decode found no valid token while higher-precision settings completed normally.

• Quality Drop. Output quality can drop a lot. In our VeVo2 checks, non-fp32 outputs showed noticeably weaker similarity to the fp32 reference under the repo's existing waveform and log-mel comparison metrics, and even output length could shift.

• Performance Gain. The performance gain may be minor relative to that quality risk. For example, q8_0 was faster than the default setting by only around 3.8% on Qwen3-TTS and around 3.6% on VeVo2. Other models may benefit more, but the tradeoff should be validated per model and per route rather than assumed.

• Memory Benefit. Lower precision and quantized weights can still be useful for reducing weight memory footprint and making larger models easier to fit within device limits. For example, in our Qwen3-TTS checks, switching from the default setting to q8_0 reduced peak RAM by about 3.7% and peak VRAM by about 25.0%. That benefit is real, but it should be evaluated together with runtime stability, output quality, and end-to-end speed rather than assumed from precision alone.

• The repo supports multiple backends, but backend and model coverage are model-dependent.
• GGUF model loading is planned, but not supported yet.
• Build_xcframework.sh is outdated; Metal and Apple XCFramework packaging still need to be retested after the framework refactor.