Repository navigation
The speech generator runs on the GPU now. In v1.0.0 it was the only part of the
loop still pinned to the CPU, and once the llm was offloaded it accounted for
almost all of a turn.
What changed
voicechat-tts.cpp builds its graphs into a ggml_backend_sched over
[device, cpu] instead of computing them on a CPU scratch arena. Weights and the
backbone kv cache live in a backend buffer, and the scheduler keeps a CPU
backend behind the chosen device so an op the device does not implement falls
back instead of aborting the turn.
New flag --tts-device <dev> (or VC_TTS_DEVICE) names that backend. It
defaults to whatever --device put the llm on, then to the first GPU there is,
then to the CPU - so an existing command line moves to the GPU with no change.
--tts-device CPU puts it back.
Measured
RTX 5060 Ti, one 17.4 s question, a session carrying the full tool block, llm
and speech generator on the same card:
| v1.0.0 | v1.0.1 | |
|---|---|---|
| open a session (load + 329 system frames) | 39 s | 22 s |
| one turn, 843 frames | 205 s | 36-47 s |
| per frame, whole loop | 0.24 s | 0.05 s |
| peak VRAM | 7943 MiB | 9977 MiB |
About 4.3x. The speech generator costs 719 MiB of weights plus 1210 MiB of kv
cache for a 180 s timeline.
What is left on the CPU is not reachable by a flag: the 31 stage residual RVQ
search (1024 candidates per stage per frame), the nucleus filter and gumbel
pick over the 1024 mixture components, and the 16 point ISTFT. Those are now
the floor.
With --tts-device CPU a --say clip is bit identical to v1.0.0, so nothing
about the speech path changed on that route.
Downloads
| file | what |
|---|---|
llama-voicechat-bin-win-cpu-x64.zip |
Windows x64, CPU. Needs an AVX2 CPU (Haswell 2013 or newer). |
llama-voicechat-bin-win-cuda-12.8-x64.zip |
Windows x64, CUDA 12.8. Self-contained. |
cudart-llama-bin-win-cuda-12.8-x64.zip |
The CUDA runtime dlls. Unzip into the same folder as the CUDA build. |
The CUDA build carries device code for sm_86, sm_89 and sm_120 (RTX 30xx, 40xx,
50xx) and PTX from sm_50 up, so anything Maxwell or newer runs.
Model
Unchanged from v1.0.0. The four GGUF files are in the llamacpp/ folder of
hoidhxd/NVIDIA-NemotronLabs-VoiceChat-11B-GGUF:
hf download hoidhxd/NVIDIA-NemotronLabs-VoiceChat-11B-GGUF --include "llamacpp/*" --local-dir .
llama-voicechat -m llamacpp/nemotron_voicechat_11b-stt-llm-Q4_0.gguf --mmproj llamacpp/mmproj-voicechat-perception-Q4_0.gguf --tts llamacpp/voicechat-tts-Q4_0.gguf --audio question.wav --tts-out answer.wav -ngl 99
Set VC_NO_BARGE=1 and VC_FORCE_BOS=1 for anything push-to-talk. The model
answers in English only.