GPT-QModel v7.9.0

GPT-QModel 7.9.0 🚀

• Correctness: Fixed packing/inference overflow, underflow, FP32 reductions, BF16 bias cancellation, 8-bit BitBLAS zero points, and stale fused Torch metadata after checkpoint loads or tensor changes.
• MoE: Fixed skipped routed experts in MiniMax M2, Granite MoE/Hybrid, Bailing Hybrid, and AfMoE; added warnings for unresolved module paths.
• Memory: Removed retained FP8 weight expansion from the experimental NVFP4 path and dense-weight scratch from ExLlamaV2.
• Kernels: Added measured RTX 4090 (Ada, SM 8.9) and RDNA4 (gfx1200/gfx1201/gfx1250) launch tables to every tuned Triton pipeline — GPTQ GEMV/GEMM/dequant, ParoQuant GEMM, AWQ GEMM/GEMV, GGUF Triton, and bitsandbytes 4-bit — enforced by a hardware-free parity contract; the fused bitsandbytes 4-bit GEMM measured 1.4x-6.3x faster than the vendor HIP kernel on RDNA4 for M >= 16, while Ada keeps the vendor kernels. Added opt-in AllSpark W8A16 for compatible per-channel 8-bit GPTQ/AWQ on CUDA and RDNA4 (gfx1200/gfx1201). EXL3 2-bit quantization now fits RDNA4's 64 KiB shared memory.
• MLX: Accelerated GPTQ/RTN quantization, GPTQ/AWQ group-16 decode and short prefill, and GGUF K-quant/ternary packing.
• GSQ/RCO: Added teacher-KL optimization, per-projection bit allocation, and native MLX search for dense Llama/Qwen3, with token-based batching and device/disk teacher caches.
• Models: Added LFM3, LFM3-VL, and LFM3-Omni text-decoder quantization, OLMo/Helium support, and Qwen2 quantization resume.
• Compat: Added native Transformers/Optimum load hooks preserving dynamic settings, fused/MoE structure, rotation, and W4A policies. Replaced method-specific Marlin/Machete/Swordfish enums with BACKEND.MARLIN, BACKEND.MACHETE, and BACKEND.SWORDFISH; legacy string aliases remain supported by the hf_ APIs.

Full list of changes

What's Changed

• Fix RCO budget decoding when softmax saturates by @Qubitium in #3493
• refactor(backend): unify marlin, machete, and swordfish sub-kernels behind shared backends by @Qubitium in #3491
• Add teacher-KL and per-projection RCO allocation by @Qubitium in #3494
• fix(config): treat act_group_aware as a no-op when group_size <= 0 by @Lumea-art in #3497
• Optimize MLX GPTQ and RTN quantization kernels (4.95x / 2.6x) by @Qubitium in #3498
• Avoid MLX split-K pass for short prefill on Apple GPUs (1.58x) by @Qubitium in #3499
• Pack MLX GGUF Q6_K blocks four per threadgroup (1.15x, bit-exact) by @Qubitium in #3500
• Pack MLX GGUF TQ2_0 blocks four per threadgroup (1.31x, bit-exact) by @Qubitium in #3501
• Add native MLX RCO search and GPU/ANE profiling for Apple silicon by @Qubitium in #3495
• Add Qwen3 GSQ-RCO qualification and ANE gating by @Qubitium in #3502
• feat(lfm3): add LFM3, LFM3-VL and LFM3-Omni quantization support by @ZX-ModelCloud in #3496
• Avoid MLX qmv_wide weight re-reads for 6-11 row batches (2.18x) by @Qubitium in #3504
• Qualify complete Qwen3 teacher forward on ANE by @Qubitium in #3505
• Stop steering MLX dispatch once it no longer matches 0.32.4 by @Qubitium in #3507
• Restrict GSQ-RCO pipeline to GPU backends by @Qubitium in #3508
• Pack MLX GGUF Q4_K weights from one coalesced block read (1.08x, bit-exact) by @Qubitium in #3509
• Pack MLX GGUF Q5_K weights from one coalesced block read (1.13x, bit-exact) by @Qubitium in #3510
• Pack MLX GGUF TQ1_0 weights from registers (1.39x, bit-exact) by @Qubitium in #3511
• fix(rocm): stop mixing NVIDIA CUDA headers into hipcc builds by @ZX-ModelCloud in #3506
• fix(w4a): stage the NVFP4 FP8 operand per forward instead of caching it by @ZX-ModelCloud in #3492
• Size the MLX RCO forward pass by a token budget by @Qubitium in #3512
• Interpolate MLX RCO weights in the model dtype (1.17x, Torch parity) by @Qubitium in #3513
• Fold constant teacher entropy out of the MLX RCO KL step (1.09x) by @Qubitium in #3514
• feat(model): add native GPT-QModel load bridge for Transformers/Optimum by @ZX-ModelCloud in #3503
• [Kernel] AllSpark W8A16: CUDA (5090/4090) + ROCm (RDNA4) by @Lumea-art in #3490
• test(allspark): cover CUDA graph capture of the W8A16 GEMM by @Lumea-art in #3515
• Fix GPTQ v1 zero-point overflow during BitBLAS repacking by @Qubitium in #3517
• fix(bitblas): 8-bit zero points truncate to 4 bits and wrap at 128 by @Lumea-art in #3516
• Fix overflow and underflow across quantized packing paths by @Qubitium in #3519
• Faster MLX GPTQ/AWQ group-16 decode with multi-column warps (1.08x, oracle-validated) by @Qubitium in #3521
• feat(checkpoint): support qwen2 checkpoints by @Lumea-art in #3518
• Fix numeric range loss across quantized inference kernels by @Qubitium in #3520
• fix(qlinear): keep the kernel's device-rejection reason by @Lumea-art in #3522
• fix(qlinear): stop repeating the device-family sentence by @Qubitium in #3524
• fix(moe): quantize routed experts, warn on unresolved module specs by @ZX-ModelCloud in #3523
• fix(tests,jit): repair false-green gates and report the real ninja failure by @ZX-ModelCloud in #3525
• fix(triton): keep the GPTQ reduction in FP32 and repair kernel test imports by @ZX-ModelCloud in #3526
• feat(models): support OLMo and Helium via the Llama module tree by @ZX-ModelCloud in #3527
• feat(exl3): support 2-bit quantization within RDNA4 shared memory by @ZX-ModelCloud in #3528
• fix(qlinear): refresh fused state, preserve AWQ precision, and remove dense scratch by @ZX-ModelCloud in #3529
• release: bump to 7.9.0 and add README release notes by @Qubitium in #3530
• docs: archive release notes for 7.5.0 and earlier by @Qubitium in #3531
• feat(triton): tuned RTX 4090 (Ada) + RDNA4 kernel paths for every quantization pipeline by @Qubitium in #3532
• feat(bitsandbytes): RDNA4 fused 4-bit GEMM (2.4x-6.3x vs vendor HIP kernel) by @Qubitium in #3533
• fix(lint): clear the ruff gate and document 4090/RDNA4 kernel parity in the README by @Qubitium in #3534

New Contributors

• @Lumea-art made their first contribution in #3497

Full Changelog: v7.8.0...v7.9.0