Skip to content

v0.7.1 — B=1 decode bracket fusion: 129 → 179 tok/s (+39%) on 3INST

Choose a tag to compare

@cnygaard cnygaard released this 25 Jul 22:14
905937e

Perf-only release: deletes the B=1 RHT/elementwise launch bracket around the 3INST trellis decode. No API breaks (new ops and trailing optional args are additive), bit-exact end to end — wikitext-2 PPL is unchanged at 9.2299 for every stage.

  • RS1 — fold the ×wscale elementwise into the matvec/matmul store
  • RS2/RS3 — FUSE_IN: input RHT + fp16 cast computed inside every matvec block (single-block pow2 and block-diagonal shapes); standalone input-RHT launches drop from 21.4% of decode time to 0.1%
  • RS4a — warp-shuffle the 5 lowest FHT butterfly stages (__shfl_xor_sync) in all RHT kernels + shared glq_fht.cuh helpers
  • S4b — shard-batched output RHT: fused QKV/gate_up shards run ONE grid.y launch instead of per-shard launches (7 → 4 per layer), torch.cat eliminated

Measured (SmolLM3-3B 3inst-4bpw, RTX PRO 6000, vLLM 0.25.0, random 128/256 ignore-eos, n=3)

before after
B=1 decode 129 tok/s 179 tok/s (+39%)
B=32 decode ~2330 2423–2437
PPL (wikitext-2) 9.2299 9.2299 (exact)
output-RHT launches/layer 7 4

B=1 is now at parity with bf16 serving (180 tok/s) on the same box while using 1.9 GB vs 5.8 GB of weights.

Validation

  • 163 trellis-suite + 21 vLLM tests green, incl. CPU bit-mirror anchors and fused-vs-sequential torch.equal gates at B=1 and B>1
  • nsys mechanism checks: input-RHT launch count ≈ 0, output-RHT 7→4/layer; ncu: fused matvec 64 regs, zero hot-path local traffic
  • Both fusions can be disabled at runtime if ever needed: GLQ_TRELLIS_FUSE_INPUT=0 and GLQ_TRELLIS_BATCH_OUT_RHT=0 restore the previous per-kernel paths