v0.7.1 — B=1 decode bracket fusion: 129 → 179 tok/s (+39%) on 3INST
Perf-only release: deletes the B=1 RHT/elementwise launch bracket around the 3INST trellis decode. No API breaks (new ops and trailing optional args are additive), bit-exact end to end — wikitext-2 PPL is unchanged at 9.2299 for every stage.
- RS1 — fold the
×wscaleelementwise into the matvec/matmul store - RS2/RS3 — FUSE_IN: input RHT + fp16 cast computed inside every matvec block (single-block pow2 and block-diagonal shapes); standalone input-RHT launches drop from 21.4% of decode time to 0.1%
- RS4a — warp-shuffle the 5 lowest FHT butterfly stages (
__shfl_xor_sync) in all RHT kernels + sharedglq_fht.cuhhelpers - S4b — shard-batched output RHT: fused QKV/gate_up shards run ONE grid.y launch instead of per-shard launches (7 → 4 per layer),
torch.cateliminated
Measured (SmolLM3-3B 3inst-4bpw, RTX PRO 6000, vLLM 0.25.0, random 128/256 ignore-eos, n=3)
| before | after | |
|---|---|---|
| B=1 decode | 129 tok/s | 179 tok/s (+39%) |
| B=32 decode | ~2330 | 2423–2437 |
| PPL (wikitext-2) | 9.2299 | 9.2299 (exact) |
| output-RHT launches/layer | 7 | 4 |
B=1 is now at parity with bf16 serving (180 tok/s) on the same box while using 1.9 GB vs 5.8 GB of weights.
Validation
- 163 trellis-suite + 21 vLLM tests green, incl. CPU bit-mirror anchors and fused-vs-sequential
torch.equalgates at B=1 and B>1 - nsys mechanism checks: input-RHT launch count ≈ 0, output-RHT 7→4/layer; ncu: fused matvec 64 regs, zero hot-path local traffic
- Both fusions can be disabled at runtime if ever needed:
GLQ_TRELLIS_FUSE_INPUT=0andGLQ_TRELLIS_BATCH_OUT_RHT=0restore the previous per-kernel paths