Skip to content

v0.7.0 — trellis 3INST: lookup-free TCQ decode kernels

Choose a tag to compare

@cnygaard cnygaard released this 25 Jul 22:07
cfe4bc1

Trellis 3INST — lookup-free trellis decode

The trellis (TCQ) codebook gains the 3INST variant: each 16-bit trellis state is decoded arithmetically in-kernel (a hash + two fp16 halves) instead of through HYB's 512-entry shared-memory lookup table. Quality measured equal-or-slightly-better than HYB on SmolLM3-3B (SQNR 22.06 vs 21.1, wikitext-2 PPL 9.2425 vs 9.2454; AIME-2026 thinking 43.3% vs 40.4% avg@8, none significant), with a simpler and faster kernel path.

  • <R, IS_3INST> templated matvec/matmul kernels, zero dynamic shared memory
  • Select at quantize time with GLQ_TRELLIS_VARIANT=3inst (--codebook trellis --bpw {2,3,4})
  • Kernel storage layout: trellis checkpoints now store indices decoder-native (trellis_layout: "kernel"); loading them requires glq ≥ 0.7.0 — older versions abort with a layout error instead of decoding garbage
  • Serves on vLLM 0.25 under full CUDA-graph capture; HF eager path included
  • Fixes an upstream idle-warp out-of-bounds read in the trellis matmul preload
  • Public checkpoint: xv0y5ncu/SmolLM3-3B-trellis-3inst-4bpw-kernel