V-cache precision bug persists in v1.4.1 on head_dim=128 models (Qwen3-14b) — KV recall degradation from ctx ~1100t

[primoco]

We tested v1.4.1 via llama-cpp-2 Rust bindings (bypassing llama-cli argument parsing) on Qwen3-14b (head_dim=128) using a KV recall benchmark: math problem at position 0, N tokens of unrelated filler, model asked to recall and solve. Temperature=0, deterministic.

We tested both tbq4_0 (ID=42, blck_size=256) and tbq4_1 (ID=46, blck_size=128). The _1 suffix fixed short-context regressions but the precision bug at 1500-2000t persists in both.

Results (91 tests, filler 0–2500t):

Config	500-1000t	1000-1500t	1500-2000t	Overall
spiritbuun tbq4_0 (ID=42)	100%	90.5%	38.9%	85.7%
AmesianX v1.4.1 tbq4_0 (ID=42)	69.2%	81.0%	38.9%	78.0%
AmesianX v1.4.1 tbq4_1 (ID=46)	92.3%	85.7%	33.3%	81.3%

The _1 suffix corrects the short-context regression (confirming the head_dim=128 auto-detection does NOT apply when using Rust bindings directly). But the 1500-2000t bucket is still 33-38% in both versions — worse than spiritbuun.

Symptom: model outputs off-topic text ("Finally, the…", "Starting with…") instead of the recalled matrix — consistent with KV cache corruption, not wrong computation.

Note on the v1.4.0 fix: The release note says the IWHT FP32 fix was verified on Qwen3.5-27B-Q4_K_M. Is that model head_dim=128 or 256? We may be hitting a different code path.

Note on Rust bindings: When using llama-cpp-2 bindings, the 6-priority cascade auto-detection does NOT run — we pass the raw GGML type ID directly. Users integrating via bindings must specify the correct _0/_1/_2 suffix manually. Worth documenting.

Reproduce: https://github.com/eullm/eullm/blob/main/bench/turboquant_math_accuracy.py

python bench/turboquant_math_accuracy.py collect \
  --label test --no-think --num-predict 2048 \
  --filler 200,500,1000,1500,2000,2500

---

[AmesianX]

Test Results: Qwen3-14B Q4_K_M (head_dim=128) — f16 vs TBQP3/TBQ3

We reproduced your test using your turboquant_math_accuracy.py script (patched for llama-server's OpenAI-compatible API instead of Ollama's /api/chat endpoint) on NVIDIA DGX Spark (GB10, 128GB).

Model & Environment

• Model: Aldaris/Qwen3-14B-Q4_K_M-GGUF (head_dim=128)
• Build: TurboQuant v1.5.0 (upstream rebase b7ad48e)
• CUDA: 13.0, compute_121a (GB10 native)

Build command:

cmake -DCMAKE_BUILD_TYPE=Release \
      -DBUILD_SHARED_LIBS=ON \
      -DGGML_CUDA=ON \
      -DGGML_BLAS=ON \
      -DGGML_CCACHE=OFF \
      -DCMAKE_EXE_LINKER_FLAGS="-lpthread -lm" \
      -DLLAMA_BUILD_TESTS=OFF \
      -DLLAMA_BUILD_EXAMPLES=OFF \
      -DLLAMA_BUILD_SERVER=ON \
      -DCMAKE_VERBOSE_MAKEFILE=ON \
      ..

make -j12

Server command (f16 baseline):

./llama-server -m ~/Models/qwen3-14b-q4_k_m.gguf \
    -t 4 -c 262144 -n 32768 --parallel 2 \
    --cont-batching --jinja --reasoning-format auto \
    --chat-template-kwargs '{"enable_thinking": false, "reasoning_effort": "medium"}' \
    --n-gpu-layers 999 --flash-attn on \
    -b 1024 -ub 512 --no-mmap \
    --cache-type-k f16 --cache-type-v f16 \
    --top-k 20 --temp 0.6 --top-p 0.95 --min-p 0.0 \
    --presence-penalty 0.0 --repeat-penalty 1.0 \
    --host 127.0.0.1 --port 8080

Server command (TBQ): same as above with --cache-type-k tbqp3 --cache-type-v tbq3

Test command:

python3 turboquant_math_accuracy.py collect \
  --url http://127.0.0.1:8080 \
  --label f16_baseline --no-think --num-predict 2048 \
  --filler 0,500,1000,1500,2000 -o math_f16.json -v

python3 turboquant_math_accuracy.py collect \
  --url http://127.0.0.1:8080 \
  --label tbqp3_tbq3 --no-think --num-predict 2048 \
  --filler 0,500,1000,1500,2000 -o math_tbq.json -v

python3 turboquant_math_accuracy.py compare math_f16.json math_tbq.json

Note: Your script uses Ollama's /api/chat endpoint. We patched it for llama-server's /v1/chat/completions (OpenAI-compatible). The patched version is available if you'd like to verify.

Results

Config	Pass/Total	Accuracy	KV Cache Memory
f16 / f16	22/65	33.8%	20,480 MiB
tbqp3 / tbq3	22/65	33.8%	4,160 + 4,000 = 8,160 MiB

Accuracy is identical (33.8% both). TBQ uses 2.5x less KV cache memory (8.2 GB vs 20.5 GB) with no accuracy loss.

Divergent Results (18 cases where f16 and TBQ disagree)

Test	f16	TBQ	Winner
direct_mat2x2_M1	❌	✅	TBQ
direct_mat2x2_M5	❌	✅	TBQ
direct_mat3x3_M8	✅	❌	f16
direct_scalar_S1	❌	✅	TBQ
direct_scalar_S2	❌	✅	TBQ
delayed_mat3x3_M8_1500t	✅	❌	f16
delayed_mat3x3_M8_2000t	✅	❌	f16
delayed_scalar_S1_500t	❌	✅	TBQ
delayed_scalar_S1_1000t	✅	❌	f16
delayed_scalar_S2_500t	✅	❌	f16
delayed_scalar_S2_1000t	✅	❌	f16
delayed_scalar_S2_1500t	❌	✅	TBQ
delayed_scalar_S3_500t	❌	✅	TBQ
delayed_scalar_S3_2000t	❌	✅	TBQ
delayed_scalar_S4_1500t	✅	❌	f16
delayed_scalar_S5_500t	✅	❌	f16
delayed_scalar_S5_1000t	✅	❌	f16
delayed_scalar_S5_1500t	❌	✅	TBQ

f16 wins: 9 cases, TBQ wins: 9 cases — perfectly balanced divergence.

Analysis

The 33.8% overall accuracy is a Qwen3-14B Q4_K_M model limitation, not a TBQ issue. The model struggles with 3x3 matrix multiplication and multi-step scalar arithmetic regardless of cache type.

The divergent results show random variance (coin-flip between f16 and TBQ), not systematic degradation. There is no correlation with filler length — TBQ wins at 1500t and 2000t as often as it loses, contradicting the hypothesis of progressive KV cache corruption.

Note on Rust Bindings

You mentioned using tbq4_0 (blck_size=256) with a head_dim=128 model via Rust bindings. When bypassing common.cpp auto-detection, the correct type for head_dim=128 is tbq4_1 (blck_size=128), not tbq4_0. Using _0 with head_dim=128 would cause a block size mismatch and explain degraded results. We recommend documenting this for binding users.

Regarding the "spiritbuun" comparison

Your table shows spiritbuun's tbq4_0 at 85.7% overall vs our tbq4_0 at 78.0%. Without knowing spiritbuun's exact configuration (model quantization, context length, sampling params), we cannot explain this difference. However, our f16 baseline (33.8%) suggests the test difficulty is quite different from what produced 85.7%, so the comparison may not be apples-to-apples.

[AmesianX]

Update: temp=0 Deterministic Retest — Still Identical

Following @TheTom's suggestion to use temp=0 for proper KV cache fidelity isolation, we re-ran the full benchmark with deterministic sampling.

Qwen3-14B Q4_K_M, temp=0, DGX Spark GB10:

Config	Pass/Total	%	Divergent
f16/f16	20/65	30.8%	f16 wins 7
tbqp3/tbq3	20/65	30.8%	TBQ wins 7

Identical accuracy, 7:7 balanced divergence. Consistent with our earlier temp=0.6 result (22/65 = 22/65, 9:9).

Both deterministic and non-deterministic sampling confirm: no TBQ degradation on head_dim=128 models. The 33.8% / 30.8% overall accuracy is a Qwen3-14B Q4_K_M model limitation, not a cache fidelity issue.

[primoco]

We've been running extensive accuracy benchmarks on Qwen3-14B and wanted to share findings that explain the discrepancies across all our earlier tests — including the F16 baseline issue you noted.

TL;DR: --ctx-size 16384 poisons F16 accuracy above ~1500t of actual context on Qwen3-14B (native context: 32768). All previous benchmarks — ours and likely yours — were measuring against a broken baseline.

---

The ctx_size bug

We noticed our F16 baseline was 82.4% (75/91) with --ctx-size 16384, dropping to 27.8% in the 1500–2000t context bucket. The exact same prompts with --ctx-size 32768 give 100% (91/91) across all filler distances including 2500t.

F16 is lossless — if accuracy degrades with F16, the attention computation itself is producing different results. Our hypothesis is RoPE frequency scaling applied when ctx_size < model native context, but we'd welcome confirmation from the llama.cpp team (opening a separate issue there).

This also explains your F16 baseline of ~33.8%: testing at longer filler distances on ctx=16384 pushes further into the degraded zone.

---

The type suffix for Qwen3

A second issue: Qwen3-14B has head_dim=128, which requires the _1 suffix variants (tbq4_1, tbq3_1, etc.). Using tbq4_0 (blck_size=256) on a head_dim=128 model corrupts the KV cache silently. Auto-detection works at llama-cli level but not via external bindings — worth documenting explicitly.

---

Clean results with correct setup (ctx=32768, tbq4_1, Qwen3-14B Q4_K_M)

Config	Accuracy	Tok/s
F16/F16	100% (91/91)	67.0
F16/tbq4_1	100% (91/91)	63.9
q8_0/tbq4_1	100% (91/91)	62.9

Zero accuracy loss vs F16 baseline, all filler distances 200–2500t, all context buckets.

---

How to replicate

We used EULLM Engine (v0.3.6, Ollama-compatible, built-in TurboQuant support via AmesianX v1.4.2):

# Broken baseline — ctx < native context
./eullm-tq run qwen3-14b.gguf \
  --cache-type-k f16 --cache-type-v f16 \
  --ctx-size 16384
# → 75/91 (82.4%), 1500-2000t bucket: 27.8%

# Correct baseline — ctx matches native context
./eullm-tq run qwen3-14b.gguf \
  --cache-type-k f16 --cache-type-v f16 \
  --ctx-size 32768
# → 91/91 (100%), all buckets 100%

# Recommended config — q8_0 K-cache + tbq4_1 V-cache
./eullm-tq run qwen3-14b.gguf \
  --cache-type-k q8_0 --cache-type-v tbq4_1 \
  --ctx-size 32768
# → 91/91 (100%), ~6% throughput cost vs F16

Benchmark script: eullm/eullm — bench/turboquant_math_accuracy.py
(91 math recall problems, filler 200–2500t, --no-think to disable Qwen3 extended reasoning)

Happy to share the full JSON results if useful.

[primoco]

Update: ctx_size bug confirmed on Gemma 4 — explains the F16 baseline discrepancy

We reproduced the same F16 degradation on Gemma 4 E4B Q4_K_M (head_dim=512):

Model	ctx=16K	ctx=32K
Qwen3-14B Q4_K_M	82.4%	100%
Gemma 4 E4B Q4_K_M	86.8%	100%

Two different vendors, architectures, and head_dim values — same pattern, same recovery at native ctx_size. We've filed this as a llama.cpp bug: ggml-org/llama.cpp#21441

Implication for TurboQuant benchmarks: any F16 baseline measured at ctx_size < model native context is unreliable. Your earlier F16=30.8% on Qwen3-14B was almost certainly this bug — we'd expect it to move to ~100% at ctx=32768, which would confirm zero accuracy cost from TurboQuant with correct setup.

Could you re-run your Qwen3-14B benchmark at --ctx-size 32768 to confirm?

[AmesianX]

Thanks for confirming the ctx_size bug on Gemma 4 as well, and for filing ggml-org/llama.cpp#21441.

You're right — our Qwen3-14B benchmarks used --ctx-size 262144 --parallel 2 (ctx_seq=131072), which exceeds Qwen3-14B's native context (40960). This likely explains the low F16 baseline (33.8%) we measured.

We'll re-run at --ctx-size 40960 --parallel 1 to confirm. Will report back with results.

[primoco]

Quick heads-up before your re-run: we tested F16/F16 on Qwen3-14B at ctx=40960 and it also degrades — 91.2% overall, 83.3% in the 1500-2000t bucket. The native context of our Qwen3-14B Q4_K_M appears to be 32768, not 40960.

ctx-size	F16/F16	1500-2000t bucket
16384	82.4%	27.8%
40960	91.2%	83.3%
32768	100%	100%

Suggest re-running at --ctx-size 32768 --parallel 1 instead.