Faster Qwen3-TTS iq4xs

[FNsi]

Currently vulkan is too slow, so I did some change with ai, it works. Just I cannot create a fork about kcpp since I forked original llama.cpp (and Id like to keep it.) then that's a issue.

Edit one sentence conclusion: no need to dequant 4bit tensor to f32 in fly for all backend

Summary from ai

Session Summary: Qwen3TTS Vulkan Backend Optimization

Initial Problem

The Qwen3TTS implementation was running at RTF=6.2 (0.16x realtime) on Vulkan GPU - meaning it took 6.2 seconds to generate 1 second of audio. The goal was to reach RTF < 1 (faster than real-time).

Investigation Phase

1. Analyzed timing data with QWEN3_TTS_TIMING enabled
2. Discovered the real bottleneck : Not graph building/allocation (which was only ~3 ms/frame), but the actual computation
3. Found the root cause : The code was casting ffn_down weights to F32 before every matmul, forcing the Vulkan backend to:
   • Dequantize 4-bit weights to F32 on-the-fly
   • Use F32 matmul instead of native quantized matmul (2x memory bandwidth)

The Fix (Minimal Change)

Removed unnecessary ggml_cast to F32 for ffn_down weights in 5 places in tts_transformer.cpp :

// BEFORE (slow):
struct ggml_tensor * ffn_down_f32 = 
ggml_cast(ctx0, layer.ffn_down, 
GGML_TYPE_F32);
cur = mul_mat(ctx0, ffn_down_f32, 
cur);

// AFTER (fast):
cur = mul_mat(ctx0, layer.ffn_down, 
cur);

Locations Fixed

1. build_prefill_forward_graph - Talker prefill
2. build_step_graph - Talker step
3. build_code_pred_graph - Code predictor non-AR
4. build_code_pred_prefill_graph - Code predictor prefill
5. build_code_pred_step_graph_impl - Code predictor step

Results

Test Case Before After "Hello world." RTF=6.2 RTF=0.925 "The quick brown fox..." RTF=~6.2 RTF=0.874 Long sentence RTF=~6.2 RTF=0.867

~7x speedup with just a 5-line change (10 lines removed, 5 lines added)!

What We Also Tried (But Didn't Work)

1. Graph caching - Attempted to cache step graphs to avoid rebuilding, but hit ggml scheduler assertion errors when reusing cached graphs across resets
2. Non-autoregressive code prediction - Built a parallel code predictor with RoPE and causal mask, but it produced incorrect results (model was trained autoregressively)
3. Different quantizations - Tested q8_0 vs iq4_xs, but iq4_xs was already optimal

Key Insight

The performance issue was not about algorithmic complexity or graph overhead. It was a simple issue where the code was unnecessarily forcing F32 computation when the Vulkan backend could handle quantized weights natively. This is a common pattern in ML inference: always let the backend use its native format when possible.

[FNsi]

Also there's some conv1d tensor didn't supported in ggml vulkan, that's the core issue some ops fallback to cpu increases the copies, but I don't think we can change ggml here.

[wbruna]

Just I cannot create a fork about kcpp since I forked original llama.cpp (and Id like to keep it.)

You can upload it to your lcpp fork in a separate branch, then make PRs for Kobold from it (I do this all the time). Simplest way is going to https://github.com/LostRuins/koboldcpp/pulls just after pushing to your fork, then choosing concedo_experimental as base branch.

[FNsi]

You can upload it to your lcpp fork in a separate branch, then make PRs for Kobold from it (I do this all the time). Simplest way is going to https://github.com/LostRuins/koboldcpp/pulls just after pushing to your fork, then choosing concedo_experimental as base branch.

Thanks for the guide, I may try it then, only file related here is tts_transformer.cpp

[FNsi]

Simplest way is going to https://github.com/LostRuins/koboldcpp/pulls just after pushing to your fork, then choosing concedo_experimental as base branch.

I don't think it's that easy and it's 44 commit ahead after I tried😅😅 so simply give up...

#2174

[FNsi]

tts_transformer.cpp

[FNsi]

feel free to test and make a pr, since i cannot simply figure out how to in my case.

[LostRuins]

for larger backend changes (e.g. vulkan) its probably best to PR directly to llama.cpp repo (that way both repos get the benefit)

[FNsi]

for larger backend changes (e.g. vulkan) its probably best to PR directly to llama.cpp repo (that way both repos get the benefit)

If you mean the 44 commit ahead it's only because my bad attempt (while only 1 commit is made by me rest are the mainline ahead commits)

Only 1 file with around 20 line change related (see upon single file) in that f32 removal case, which help running qwen3tts iq4xs gain 2-7 times speed boost.

Can u test and make it a pr cause I don't know how to...

[LostRuins]

alright ill try

[FNsi]

also I see the tech report, the qwen3tts ctx 8192 is origin and can be extended to 32768 by scaling, so...I guess I'd simply write that here as a feature request 😂

[LostRuins]

So I gave it a shot. The original commit at predict-woo/qwen3-tts.cpp@8783e53#diff-32bec1c446243103ad422160039fbe7d1921d50f1663af5d658902ee09207b91 mentioned that using F32 was for precision.

With your changes, it does run faster, however the quality seems to have dropped too. I am not sure if that is a good idea. Have you noticed a lower quality?

[LostRuins]

I'll add the change tentatively, but if the quality is too bad i might have to revert it. eca9f4c

[FNsi]

I'll add the change tentatively, but if the quality is too bad i might have to revert it. eca9f4c

I think it won't be much difference, just say, for those people already use 4bit weight.
did you just test with the same seed? If the quality actually drop then the dequant to f32 may have some magic i don't know now if you use the same 4bit weight😂😂 (people still can use their 8bit instead that 4bit or even f16 version, if they want more precision)

Also, with that speed, cpu can be real time in 4bit!! That's the most impressive thing!

[LostRuins]

Please give it a test in the rolling builds https://github.com/LostRuins/koboldcpp/releases/tag/rolling