Two fixes to the tq2_0 CUDA kernels take Maple-Preview from 52 to 377 tok/s on an RTX 4080 SUPER.
| Build | pp512 | tg128 | vs stock |
|---|---|---|---|
| stock fork | 1255.7 | 52.1 | — |
+ 0001 MMVQ for batch-1 MoE |
1173.4 | 254.8 | 4.9x |
+ 0002 SIMD ternary vec_dot |
1635.8 | 376.7 | 7.2x |
q4_k_m reference |
8130.5 | 298.2 | — |
The patched ternary build generates faster than Q4_K (377 vs 298 t/s) in 44% of the memory.
What the fixes are
- 0001 —
get_mmvq_mmid_max_batchreturned-1forTQ2_0. The call site testsne2 <= mmvq_mmid_max, andne2is a batch size >= 1, so the condition was never true — the fast ternary kernel was unreachable at every batch size and every MoE expert matmul fell back to dequantize-to-F32 + GEMM. - 0002 —
vec_dot_tq2_0_q8_1was a scalar 32-iteration loop. Rewritten to 4 elements per step with__vsub4+ggml_cuda_dp4a. - 0003 — enables
tq2_0intest-backend-ops. Ternary is commented out of the upstream suite (// TODO: implement for all backends), so these kernels had zero automated coverage — which is how 0001's bug survived.
Validated 103/103 against the CPU reference (72 MUL_MAT_ID + 31 MUL_MAT).
Still open
Prompt processing trails badly (1636 vs 8130 t/s) — that is the missing ternary MMQ kernels, which is real kernel authoring, not a guard fix. Measured on one GPU (sm_89) only.
Not submitted upstream: llama.cpp's AGENTS.md declines predominantly AI-generated PRs and exempts private forks.