Fix: IndexTTS-2 F16 weight storage - #247
Merged
0xShug0 merged 2 commits intoAug 15, 2026
Merged
Conversation
Folded weight-norm conv weights (BigVGAN vocoder) and transposed HF conv1d linear weights (GPT decoder) are derived host-side, so the "native" storage type cannot resolve through load_tensor and silently fell back to F32. Follow the checkpoint dtype instead so an F16 GGUF keeps F16 weights in the compute graph; honor the caller-provided storage type in the weight-norm conv loaders.
The CFM estimator linears were pinned to GGML_PREC_F32 to work around CPU backend noise in the CFM wavenet. CUDA matmul kernels handle reduced precision correctly, so let them follow the module default there and keep the F32 pin on other backends.
Owner
|
@IIIIIllllIIIIIlllll This is a very good improvment, but I need some time to do regression test for the existing 5 consumers of BigVGAN. |
Owner
|
@IIIIIllllIIIIIlllll Could you test the audio quality change before and after the PR for indextts 2/2.5 using the q8 gguf? |
Contributor
Author
No problem. Are there any criteria for the evaluation? |
Owner
Just human judgment :). On CUDA Q8 before and after the patch has log-mel similarity and frame-count drift (while F16 is identical). But the spoken words are correct, and the emotion sounds good. I don’t know whether the BigVGAN change will affect the behavior for quantized types on HIP. If they sound good to you, I'm ready to merge. |
0xShug0
marked this pull request as ready for review
August 15, 2026 15:17
Contributor
Author
|
@0xShug0 I can clearly perceive the difference between F16 and Q8 — it’s a fleeting, extremely jarring yet indescribable sensation, a difference that only becomes noticeable upon careful comparison. However, there is virtually no difference between Q8(main branch) and Q8(this pr). (I used to be really into HiFi.) |
Owner
|
@IIIIIllllIIIIIlllll Merged! |
IIIIIllllIIIIIlllll
added a commit
to IIIIIllllIIIIIlllll/audio.cpp
that referenced
this pull request
Aug 25, 2026
IndexTTS2 on the HIP backend was ~40% slower than Vulkan on Strix Halo (gfx1151). Per-stage timings plus a rocprofv3 kernel trace pinned the remaining HIP-specific overhead on top of 0xShug0#247 (which fixed the F32-forced derived GPT weights for all backends): - decode_cache_type only used an F16 KV cache on CUDA. HIP shares the ggml-cuda flash-attention kernels, which require F16 K/V (unlike the Vulkan shaders, which accept F32 K/V natively), so the whole cache was converted f32->f16 twice per decode step (convert_unary<float,__half> ~= 1.95 s, growing quadratically with the number of decode steps). - The CFM/emo derived conv weights were likewise kept as F32; on HIP the conv/im2col paths run measurably faster on F16, so default index_tts2.conv_weight_type to F16 there. Measured on MarkPC (Ryzen AI MAX+ 395, gfx1151), same text/seed 1245, bf16 GGUF: HIP wall 13017 -> ~9650 ms (-26%, ~1.0x realtime), within ~6% of Vulkan (9107 ms); GPT decode (2765 ms) and CFM (3571 ms) both beat Vulkan (3368 / 3714 ms). The remaining gap is the BigVGAN vocoder (2047 vs 1037 ms). CUDA (RTX 2080 Ti) regression-checked at 6184 ms vs 6183 ms baseline. Outputs validated with ASR (qwen3_asr): transcripts identical before/after and correct.
0xShug0
pushed a commit
that referenced
this pull request
Aug 25, 2026
#305) IndexTTS2 on the HIP backend was ~40% slower than Vulkan on Strix Halo (gfx1151). Per-stage timings plus a rocprofv3 kernel trace pinned the remaining HIP-specific overhead on top of #247 (which fixed the F32-forced derived GPT weights for all backends): - decode_cache_type only used an F16 KV cache on CUDA. HIP shares the ggml-cuda flash-attention kernels, which require F16 K/V (unlike the Vulkan shaders, which accept F32 K/V natively), so the whole cache was converted f32->f16 twice per decode step (convert_unary<float,__half> ~= 1.95 s, growing quadratically with the number of decode steps). - The CFM/emo derived conv weights were likewise kept as F32; on HIP the conv/im2col paths run measurably faster on F16, so default index_tts2.conv_weight_type to F16 there. Measured on MarkPC (Ryzen AI MAX+ 395, gfx1151), same text/seed 1245, bf16 GGUF: HIP wall 13017 -> ~9650 ms (-26%, ~1.0x realtime), within ~6% of Vulkan (9107 ms); GPT decode (2765 ms) and CFM (3571 ms) both beat Vulkan (3368 / 3714 ms). The remaining gap is the BigVGAN vocoder (2047 vs 1037 ms). CUDA (RTX 2080 Ti) regression-checked at 6184 ms vs 6183 ms baseline. Outputs validated with ASR (qwen3_asr): transcripts identical before/after and correct.
NairoDorian
pushed a commit
to NairoDorian/speech.cpp
that referenced
this pull request
Aug 26, 2026
0xShug0#305) IndexTTS2 on the HIP backend was ~40% slower than Vulkan on Strix Halo (gfx1151). Per-stage timings plus a rocprofv3 kernel trace pinned the remaining HIP-specific overhead on top of 0xShug0#247 (which fixed the F32-forced derived GPT weights for all backends): - decode_cache_type only used an F16 KV cache on CUDA. HIP shares the ggml-cuda flash-attention kernels, which require F16 K/V (unlike the Vulkan shaders, which accept F32 K/V natively), so the whole cache was converted f32->f16 twice per decode step (convert_unary<float,__half> ~= 1.95 s, growing quadratically with the number of decode steps). - The CFM/emo derived conv weights were likewise kept as F32; on HIP the conv/im2col paths run measurably faster on F16, so default index_tts2.conv_weight_type to F16 there. Measured on MarkPC (Ryzen AI MAX+ 395, gfx1151), same text/seed 1245, bf16 GGUF: HIP wall 13017 -> ~9650 ms (-26%, ~1.0x realtime), within ~6% of Vulkan (9107 ms); GPT decode (2765 ms) and CFM (3571 ms) both beat Vulkan (3368 / 3714 ms). The remaining gap is the BigVGAN vocoder (2047 vs 1037 ms). CUDA (RTX 2080 Ti) regression-checked at 6184 ms vs 6183 ms baseline. Outputs validated with ASR (qwen3_asr): transcripts identical before/after and correct. (cherry picked from commit d03b957)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While running IndexTTS on the 2080 Ti, I noticed that the performance of the F16‑quantized GGUF and the F32‑quantized GGUF was almost identical, so I made a fix to address the performance difference.
Here is the comparison before and after the fix:
Need your input: should we go ahead with this change?