Skip to content

v1.0.0 — ternary CUDA kernel fixes (7.2x generation)

Latest

Choose a tag to compare

@PascalAI2024 PascalAI2024 released this 06 Aug 02:20
· 9 commits to main since this release

Two fixes to the tq2_0 CUDA kernels take Maple-Preview from 52 to 377 tok/s on an RTX 4080 SUPER.

Build pp512 tg128 vs stock
stock fork 1255.7 52.1
+ 0001 MMVQ for batch-1 MoE 1173.4 254.8 4.9x
+ 0002 SIMD ternary vec_dot 1635.8 376.7 7.2x
q4_k_m reference 8130.5 298.2

The patched ternary build generates faster than Q4_K (377 vs 298 t/s) in 44% of the memory.

What the fixes are

  • 0001get_mmvq_mmid_max_batch returned -1 for TQ2_0. The call site tests ne2 <= mmvq_mmid_max, and ne2 is a batch size >= 1, so the condition was never true — the fast ternary kernel was unreachable at every batch size and every MoE expert matmul fell back to dequantize-to-F32 + GEMM.
  • 0002vec_dot_tq2_0_q8_1 was a scalar 32-iteration loop. Rewritten to 4 elements per step with __vsub4 + ggml_cuda_dp4a.
  • 0003 — enables tq2_0 in test-backend-ops. Ternary is commented out of the upstream suite (// TODO: implement for all backends), so these kernels had zero automated coverage — which is how 0001's bug survived.

Validated 103/103 against the CPU reference (72 MUL_MAT_ID + 31 MUL_MAT).

Still open

Prompt processing trails badly (1636 vs 8130 t/s) — that is the missing ternary MMQ kernels, which is real kernel authoring, not a guard fix. Measured on one GPU (sm_89) only.

Not submitted upstream: llama.cpp's AGENTS.md declines predominantly AI-generated PRs and exempts private forks.