Skip to content

runNburn v0.5.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 01 Aug 15:30
v0.5.0
0bd2df0

runNburn 0.5.0 more than doubles Gemma 4 26B-A4B end-to-end throughput by removing row-by-row CUDA transfers, keeping selected expert work on device, reusing quantized down weights across rows, and parallelizing large KVarN compaction batches. The release keeps mmap-backed GGUF weights and the existing quantized formats intact; it does not introduce expanded weight caches or model sidecars.

CUDA performance

The official release rerun used an RTX 3090 24 GB, Gemma 4 26B-A4B Q4_K_M with the external Q8_0 assistant selected automatically at k=1, a 1,115-token prompt, and 100 generated tokens. Values are medians after one warmup and three measured runs against the latest v0.4.0 baseline on the same device and workload:

  • Prefill increased from 38.760 to 102.210 tok/s (+163.70%).
  • Decode increased from 14.170 to 17.426 tok/s (+22.98%).
  • End-to-end generation increased from 2.788 to 6.011 tok/s (+115.59%).
  • All three measured runs used auto-MTP k=1, produced the same 100-token output hash, and accepted 47/53 drafted tokens (88.7%). A separate 10-token Korean correctness run produced the established exact hash and the correct answer, Seoul.

The release-baseline comparison above is the official product regression gate, not an interleaved attribution experiment. Each implementation step was also measured independently with warmup plus ABABAB on the same RTX 3090 and 1,115-token Gemma workload:

  • Batched expert activations reduced prefill median by 8.12%, same-index 3/3.
  • Batched expert normalization and residual operations reduced it by 16.30%, same-index 3/3.
  • Chaining gate/up, GELU, and quantized down projection on device reduced it by 8.16%, same-index 3/3.
  • Device-resident expert group gather, FFN, and weighted scatter reduced it by 14.04%, same-index 3/3.
  • Four-row Q5_1/Q8_0 down-weight reuse reduced it by 17.15%, same-index 3/3.
  • Ordered parallel KVarN block compaction reduced it by 22.09%, same-index 3/3.

Validation

  • The official four-model regression suite passed on Qwen3.6 35B-A3B, Gemma 4 26B-A4B, Hy3 295B-A21B, and GLM-5.2. Qwen end-to-end throughput was within +0.35% of the latest baseline; Hy3 was within -1.68%; GLM showed no regression, but its 58.18% run variance is too high to claim an improvement.
  • Qwen matched its established 100-token reference hash. Hy3 and GLM produced exact token IDs across all three measured runs.
  • The CUDA backend library suite passed 623 tests with 45 explicitly ignored tests. The CPU, CUDA, Vulkan, OpenCL, and MediaTek build matrix passed.
  • The release workflow passed all 10 jobs and published 9 platform archives with 9 matching SHA-256 files.

Full changelog: v0.4.0...v0.5.0

Assets 20
Loading