v0.3.1: Super Fat GEMM Prefill Kernel Suite & Concurrency Scaling

v0.3.1: Super Fat GEMM Prefill Kernel Suite & Concurrency Scaling for NVIDIA DGX Spark GB10

Version 0.3.1 delivers the Super Fat GEMM prefill kernel suite for wide-layer MoE dispatch, along with formal upstream attribution compliance and stability fixes for high-concurrency serving on NVIDIA DGX Spark GB10 (sm_121 Blackwell).

What's New in v0.3.1

• Super Fat GEMM Prefill Kernel Suite (csrc/exl3_fat_gemm.cu, csrc/exl3_fat_gemm.cuh):
  • High-throughput tiled chunked prefill kernel engineered for wide-layer and high-context routed expert evaluation.
  • Performs batched matrix multiplication directly over unquantized activation states and trellis-dequantized expert weights with register-level unrolling.
  • Dispatched automatically through apply_exl3_batched_fat in src/vllm_exl3/exl3.py.
• Kernel Stability & Dispatch Fix:
  • Implements a guard for k == 4 in apply_exl3_batched_fat dispatch to prevent illegal memory indexing during 4-bit trellis tile reconstruction.
• Upstream Attribution & Notice Compliance:
  • Full third-party attribution prominently placed at the top of README.md and documented in THIRD_PARTY_NOTICES.md and NOTICE.
  • Credits to Mia's AI Lab (@MiaAI-Lab and @plotarmordev) for the routed-expert EXL3 serving path and Fat GEMM CUDA kernels (GLM-5.3-Flash-EXL3-2x-DGX-Sparks, commit 4b8d3c7).
  • Credits to ExLlamaV3 (@turboderp) for the EXL3 trellis quantization format, MCG codebook, and core dequantization math.
• Hardware Validation:
  • Fully validated on NVIDIA DGX Spark GB10 (sm_121 Blackwell) with 128 GiB Unified Memory.

---

Upstream Credits & Notices

This project builds upon foundational open-source engineering:

• ExLlamaV3: Authored by @turboderp (MIT License). Defines the EXL3 trellis format, MCG codebook, and low-bit quantization mechanics.
• Mia's AI Lab: Authored by @MiaAI-Lab and @plotarmordev (MIT License). Pioneered the routed-expert EXL3 serving path and Fat GEMM CUDA kernels.

Full license texts and commit notices are preserved in THIRD_PARTY_NOTICES.md.

v0.3.0: Native EXL3 CUDA Kernel Suite for NVIDIA DGX Spark GB10

v0.3.0: Native EXL3 CUDA Kernel Suite for NVIDIA DGX Spark GB10

Version 0.3.0 introduces a high-performance native CUDA kernel suite (csrc/) replacing the stock exllamav3_ext decode and prefill paths on NVIDIA Blackwell sm_121 (and Hopper sm_90) architectures.

What's New in v0.3.0

• In-Register Trellis Dequantization (csrc/exl3_dequant.cuh): Unrolls MCG bit extraction directly into hardware registers without intermediate global memory roundtrips.
• Active-Expert Batched GEMV (csrc/exl3_gemv.cu, csrc/p2b_batched.cu): Saturates 99.2% of the physical memory bandwidth floor (73.3 $\mu\text{s}$).
• 4-Phase Cooperative MoE Decode (csrc/p2b_moe.cu): End-to-end fused MoE decode reducing per-layer latency from $497\ \mu\text{s} \to 287.8\ \mu\text{s}$ ($1.73\times$).
• Power-of-Two Chunked Prefill GEMM (csrc/exl3_gemm.cu): Tiled matrix multiplication delivering 7.85 TFLOPS ($13.0\times$ faster than legacy prefill).
• vLLM Dispatch Control: Environmental toggle VLLM_EXL3_MOE_KERNEL=native (default) with seamless, zero-cost fallback to exllamav3.

---

Live Head-to-Head Benchmark Receipts

Measured simultaneously across physical NVIDIA DGX Spark GB10 machines (sm_121 Blackwell, 128 GiB Unified Memory) running GLM-5.3-Flash-EXL3-K2 via live vLLM HTTP streaming API:

Category	Baseline ExLlamaV3	Native EXL3 Suite	Baseline TTFT	Native TTFT	Net Speedup
Coding	14.9 tok/s	27.6 tok/s	2,343.8 ms	859.1 ms	+85.6%
Prose	13.7 tok/s	24.6 tok/s	355.4 ms	308.7 ms	+79.3%
Reasoning	18.9 tok/s	25.1 tok/s	482.2 ms	407.8 ms	+32.7%
Summary	17.1 tok/s	25.6 tok/s	409.6 ms	345.4 ms	+50.0%
Format	16.3 tok/s	24.0 tok/s	401.9 ms	349.8 ms	+47.7%
JSON	20.8 tok/s	25.6 tok/s	502.6 ms	414.1 ms	+23.3%
HTML	19.5 tok/s	23.1 tok/s	361.7 ms	323.1 ms	+18.6%
Narrative	14.0 tok/s	21.0 tok/s	395.4 ms	333.0 ms	+50.0%
Average Across Categories	16.9 tok/s	24.6 tok/s	656.6 ms	417.6 ms	+45.6%

---

Per-Step Decode Latency Breakdown (C1)

• 40 MoE Layers: Cut from $19.9\ \text{ms} \to 11.5\ \text{ms}$ ($497\ \mu\text{s} \to 287.8\ \mu\text{s}$ per layer), saving 8.4 ms in MoE compute alone per token.
• Total Per-Step Time: Reduced from $59.2\ \text{ms} \to 40.6\ \text{ms}$ (-31.4%), directly powering the +45.6% throughput gain.
• Prefill GEMM: 7.85 TFLOPS ($13.0\times$ faster), holding 1,875 tok/s cold prefill across 65k context.
• NVMe Storage Scaling: 8-worker parallel read reaches 3,563 MB/s ($3.0\times$ speedup over single-thread 1,185 MB/s), loading 96 GB weights in ~27 seconds.

---

Essential Serving Guidance

When running on DGX Spark or long-context instances, pass:

--long-prefill-token-threshold 1024

This prevents long prompt prefill from starving parallel decode steps and stalling the scheduler.

---

Third-Party Notices & Attribution

• Turboderp (@turboderp): EXL3 trellis format, MCG codebook, quantization math, and base extension headers (hadamard_inner.cuh, ptx.cuh).
• Mia's AI Lab (@MiaAI-Lab / @plotarmordev): Routed-expert EXL3 serving path, pointer table dispatch, and expert-map pinning.
• Full license texts and attribution records are maintained in THIRD_PARTY_NOTICES.md.

v0.2.3

Dense EXL3 for non-routed linears (non_routed_exl3 layers map, mul1 codebook, mixed EXL3/BF16 shards), tools/dense_overlay.py, the bf16_as_stored policy for packs that keep dense weights BF16 (DeepSeek-V4-Flash on stock vLLM 0.28), and the mul1 marker constant fix. See CHANGELOG.md.

v0.2.1

Fix: the glm53_exl3_plugin compat shim now provides a real glm53_exl3_plugin.exl3 submodule.

v0.2.0

First standalone release. Renamed from glm53_exl3_plugin 0.1.1 — identical behavior plus the package rename; the old import path remains as a deprecated shim.

Carried from 0.1.1:

• Fused-MoE per-expert row cap raised (TEMP_ROWS_FUSED 128 → 2048), fixing the >163k-token prefill stall.
• Non-routed layers delegate to a pack-declared source-format quant method (quantization_config.non_routed_quantization).