v0.6.9 — llama.cpp UD GGUFs load and run faster than llama.cpp

llama.cpp "UD" GGUFs load, and run faster than llama.cpp on them.

unsloth now publishes only "UD" dynamic quants of Qwen3.8-27B. These mix Q3_K, IQ3_S, IQ4_NL and IQ4_XS into the usual k-quants, and sparkinfer couldn't load them. Now it can.

• GPU dequantizers for the four new types. Ported from ggml; bit-identical to gguf-py.
• Refit to Q4_K at load. The new types and the file's Q5_K attention-side matrices become Q4_K. SPARKINFER_GGUF_Q5K_PROJ=q8 keeps those Q5_K matrices near-lossless instead.
• New int8 and tensor-core kernels for FFN layers that mix Q4_K, Q5_K and Q6_K.

Measured

RTX 5090, Qwen3.8-27B-UD-Q4_K_M, llama.cpp 436f6f8 on the same file. Single stream, greedy:

ctx	decode, sparkinfer / llama.cpp	prefill, sparkinfer / llama.cpp
128	94.7 / 83.1	4,538 / 2,809
4K	93.2 / 82.0	8,756 / 3,839
16K	90.9 / 78.7	8,718 / 3,606

Concurrent requests. Before this release they declined batched decode on this file. Now they decode batched.

• Decode throughput (rows ÷ step time) is 608 tok/s at 16 rows and 938 at 32.
• llama-batched-bench on the same file gives 621 and 763.

Accuracy vs llama-server on the same file: top-1 0.956, KL 0.025. The Q5_K→Q4_K refit costs 1.2% perplexity.

Fix: dense k-quant GGUFs stop mixing prefill and decode

Since 0.6.4, a mixed step's decode rows on a dense k-quant GGUF were scored through the batched prefill's int8 projections. Those projections are coarser than the decode kernels.

• On Qwythos-9B Q4_K_M, those rows agreed with one forward per row 0.90–0.93 of the time, against 0.97–0.98 for the control.
• These models now prefill prompts in passes of their own, and throughput is no worse (Qwythos cb c16 / c32: 1,531 / 2,288 → 1,584 / 2,342 tok/s).
• NVFP4, MoE, Muse and Bonsai models keep mixing. SPARKINFER_MIXED_KQUANT=1 restores mixing on dense k-quant GGUFs.

Full notes: CHANGELOG.

What's Changed

• feat(gguf): llama.cpp UD GGUFs load and run fast (Qwen3.8-27B-UD-Q4_K_M decode 94.7 vs llama.cpp 83.1, packed decode at concurrency); fix: dense k-quant GGUFs stop mixing by @skyrocket2026 in #1267
• docs: 0.6.9 — llama.cpp UD GGUFs load (Qwen3.8-27B-UD-Q4_K_M decode 94.7 vs llama.cpp 83.1) by @skyrocket2026 in #1268

Full Changelog: v0.6.8...v0.6.9