Skip to content

Releases

v1.0.1

Choose a tag to compare

@sansamour sansamour released this 17 Aug 10:48

The speech generator runs on the GPU now. In v1.0.0 it was the only part of the
loop still pinned to the CPU, and once the llm was offloaded it accounted for
almost all of a turn.

What changed

voicechat-tts.cpp builds its graphs into a ggml_backend_sched over
[device, cpu] instead of computing them on a CPU scratch arena. Weights and the
backbone kv cache live in a backend buffer, and the scheduler keeps a CPU
backend behind the chosen device so an op the device does not implement falls
back instead of aborting the turn.

New flag --tts-device <dev> (or VC_TTS_DEVICE) names that backend. It
defaults to whatever --device put the llm on, then to the first GPU there is,
then to the CPU - so an existing command line moves to the GPU with no change.
--tts-device CPU puts it back.

Measured

RTX 5060 Ti, one 17.4 s question, a session carrying the full tool block, llm
and speech generator on the same card:

v1.0.0 v1.0.1
open a session (load + 329 system frames) 39 s 22 s
one turn, 843 frames 205 s 36-47 s
per frame, whole loop 0.24 s 0.05 s
peak VRAM 7943 MiB 9977 MiB

About 4.3x. The speech generator costs 719 MiB of weights plus 1210 MiB of kv
cache for a 180 s timeline.

What is left on the CPU is not reachable by a flag: the 31 stage residual RVQ
search (1024 candidates per stage per frame), the nucleus filter and gumbel
pick over the 1024 mixture components, and the 16 point ISTFT. Those are now
the floor.

With --tts-device CPU a --say clip is bit identical to v1.0.0, so nothing
about the speech path changed on that route.

Downloads

file what
llama-voicechat-bin-win-cpu-x64.zip Windows x64, CPU. Needs an AVX2 CPU (Haswell 2013 or newer).
llama-voicechat-bin-win-cuda-12.8-x64.zip Windows x64, CUDA 12.8. Self-contained.
cudart-llama-bin-win-cuda-12.8-x64.zip The CUDA runtime dlls. Unzip into the same folder as the CUDA build.

The CUDA build carries device code for sm_86, sm_89 and sm_120 (RTX 30xx, 40xx,
50xx) and PTX from sm_50 up, so anything Maxwell or newer runs.

Model

Unchanged from v1.0.0. The four GGUF files are in the llamacpp/ folder of
hoidhxd/NVIDIA-NemotronLabs-VoiceChat-11B-GGUF:

hf download hoidhxd/NVIDIA-NemotronLabs-VoiceChat-11B-GGUF --include "llamacpp/*" --local-dir .
llama-voicechat -m llamacpp/nemotron_voicechat_11b-stt-llm-Q4_0.gguf --mmproj llamacpp/mmproj-voicechat-perception-Q4_0.gguf --tts llamacpp/voicechat-tts-Q4_0.gguf --audio question.wav --tts-out answer.wav -ngl 99

Set VC_NO_BARGE=1 and VC_FORCE_BOS=1 for anything push-to-talk. The model
answers in English only.

v1.0.0

Choose a tag to compare

@sansamour sansamour released this 17 Aug 03:57

First release of the VoiceChat fork of llama.cpp. Speech in, speech out, one
duplex timeline, with tool calls.

Downloads

file what
llama-voicechat-bin-win-cpu-x64.zip Windows x64, CPU. Needs an AVX2 CPU (Haswell 2013 or newer).
llama-voicechat-bin-win-cuda-12.8-x64.zip Windows x64, CUDA 12.8. Self-contained.
cudart-llama-bin-win-cuda-12.8-x64.zip The CUDA runtime dlls. Unzip into the same folder as the CUDA build.

The CUDA build carries device code for sm_86, sm_89 and sm_120 (RTX 30xx, 40xx,
50xx) and PTX from sm_50 up, so anything Maxwell or newer runs.

Model

The four GGUF files are in the llamacpp/ folder of
hoidhxd/NVIDIA-NemotronLabs-VoiceChat-11B-GGUF:

hf download hoidhxd/NVIDIA-NemotronLabs-VoiceChat-11B-GGUF --include "llamacpp/*" --local-dir .
llama-voicechat -m llamacpp/nemotron_voicechat_11b-stt-llm-Q4_0.gguf --mmproj llamacpp/mmproj-voicechat-perception-Q4_0.gguf --tts llamacpp/voicechat-tts-Q4_0.gguf --audio question.wav --tts-out answer.wav

Set VC_NO_BARGE=1 and VC_FORCE_BOS=1 for anything push-to-talk.

What is in it

  • llama-voicechat: the 12.5 Hz duplex loop, the function channel for
    turn-taking and tool calls, --serve for a live session, --say to make a
    test clip without a mic
  • a new mtmd projector type for VoiceChat's causal FastConformer encoder
  • the speech generator: gemma3 backbone, mixture-of-Gaussians head, 31 stage
    residual VQ and the codec decoder, on plain ggml graphs
  • three converters that split the published VoiceChat GGUF into the four files
    llama.cpp loads, plus a numpy reference harness for the speech path

The model answers in English only.

Everything upstream llama.cpp does still works; this is a fork of
ggml-org/llama.cpp with its full history.