I built leankv ([https://github.com/asmit383/leankv]) on top of TEAL — it uses your calibration + greedy per-layer threshold optimization, one level below the Triton kernel. As part of it I hand-wrote a fused sparse + int4 GEMV for batch-1 decode:
y[n] = Σ_k ( |x[k]| > t ? x[k] · dequant_int4(Wq[k,n]) : 0 )
The point is the fusion: int4 weights stay packed in HBM and are unpacked to fp16 in registers via LOP3.LUT, so dequant never round-trips through memory — a separate dequant kernel would write fp16 back to HBM and lose the byte savings. int4 symmetric group-wise (G=128), split-K, fp32 accum, uniform per-block sparsity skip (zero warp divergence).
Measured on an NVIDIA L4 (300 GB/s HBM):
- fp16 sparse GEMV ties the Triton kernel on HBM-bound shapes (85% BW util, rel-err 1e-5), and sparsity converts linearly to wall-clock (40%→0.60×, 60%→0.40×).
- the int4-fused version moves 4× fewer weight bytes, giving a further batch-1 speedup on top of the sparsity skip.
Two questions before I open a PR:
- Would you want this rewritten in Triton to match the current gpt-fast inference path, or contributed as an optional CUDA backend?
- Any conventions (kernel interface, calibration format, tests) you'd want me to follow?
I built leankv ([https://github.com/asmit383/leankv]) on top of TEAL — it uses your calibration + greedy per-layer threshold optimization, one level below the Triton kernel. As part of it I hand-wrote a fused sparse + int4 GEMV for batch-1 decode:
y[n] = Σ_k ( |x[k]| > t ? x[k] · dequant_int4(Wq[k,n]) : 0 )
The point is the fusion: int4 weights stay packed in HBM and are unpacked to fp16 in registers via LOP3.LUT, so dequant never round-trips through memory — a separate dequant kernel would write fp16 back to HBM and lose the byte savings. int4 symmetric group-wise (G=128), split-K, fp32 accum, uniform per-block sparsity skip (zero warp divergence).
Measured on an NVIDIA L4 (300 GB/s HBM):
Two questions before I open a PR: