Repository navigation
GPT-QModel 7.11.4
- Quantization: SmoothMSE keeps the unclipped range when clipping would increase error, and QQQ's GSQ calibration now uses the same FP32 activation scales as inference. AWQ samples routed-expert activations uniformly across the calibration set instead of favoring later batches. Paro retains mixed-shape batches, and SenseNova-Loop calibrates on every pass through its repeated layers.
- Quantization memory: AWQ and Paro limit how many activation rows they retain. AWQ reuses its expert sampling buffers and scales experts in place across GPUs, while EXL3 spreads expert Hessians across the configured GPU pool. ROCm allocator defaults reduce peak reserved VRAM for wide MACaQ layers.
- Inference: Marlin speeds up large-batch prefill and fixes GPTQ/AWQ output errors under GPU contention and MXFP8/MXFP4 execution. Portable QQQ and VPTQ Torch kernels now support compiled generation. Fixed crashes when loading another model during compiled inference and improved EXL3 BF16 execution and Paro dispatch for long prefills.
- Compat: Fixed Transformers
device_map="auto"when GPTQ, RTN, or AWQ layers spill to CPU or disk. Checkpoint loads preserve required FP32 tensors, and re-saves preserve quantized embeddings and output heads. Marlin now requires NVIDIA Ampere or newer.
Full list of changes
What's Changed
- Fix CUDA CI runtime for Python 3.14t and stable PyTorch 2.15 by @ZX-ModelCloud in #5103
- fix(rocm): stop pinning max_split_size_mb in the allocator default by @ZX-ModelCloud in #5108
- fix(calibration): drop Arrow-injected nulls from
messagesbefore the chat template (#5082) by @ZX-ModelCloud in #5112 - fix(sensenova): capture every loop visit during calibration by @ZX-ModelCloud in #5114
- Move AWQ CPU suites out of pull-request CI by @Qubitium in #5117
- fix(awq): cap packed token-row replay at the model position limit by @ZX-ModelCloud in #5120
- fix(hf): resolve an accelerate keyword map before validating GPU capabilities (#5084) by @ZX-ModelCloud in #5123
- fix(exl3): publish deferred runtime weights when a run exits early (#5083) by @ZX-ModelCloud in #5125
- perf(nf12): make the FP32 prefill-gate row count a runtime argument by @Qubitium in #5129
- fix(awq): skip the batched output copy for untracked inference (#5086) by @ZX-ModelCloud in #5130
- fix(structure): publish materialized parameters atomically (#5088) by @ZX-ModelCloud in #5132
- fix(cpu): keep FP32 activations in FP32 for TorchFused int4pack by @ZX-ModelCloud in #5133
- fix(quantization): validate embedding quant config before replay (#5124) by @Qubitium in #5136
- test(gpu): repair the #4387, #5014 and #5020 host gates by @Qubitium in #5138
- fix(compile): make the int4pack cast path a traceable custom op (#5110) by @Qubitium in #5139
- fix(gguf): reject conflicting full-subtype bits and format (#5106) by @Qubitium in #5140
- fix(allspark): size the fused split-K workspace in 64-bit by @ZX-ModelCloud in #5141
- fix(awq): keep the MoE block signature visible in bypass replay (#5095) by @Qubitium in #5142
- fix(exl3): give routed-expert Hessians a concrete owner device (#5093) by @ZX-ModelCloud in #5144
- fix(config): reject unknown smoother/fallback payload keys (#5122, #5107) by @Qubitium in #5143
- fix(generate): keep version counters on CPU-offloaded models (#5099) by @Qubitium in #5147
- fix(awq): address wide-prefill output and activation offsets in 64 bits by @ZX-ModelCloud in #5149
- test(exl3): inherit device visibility in the routed-owner regression (#5093) by @ZX-ModelCloud in #5150
- fix(looper): publish a quant kernel only after packing returns (#5089) by @ZX-ModelCloud in #5152
- fix(offload): gate disk offload on recursive module state (#5145) by @Qubitium in #5153
- fix(hf): restore the transformers-4 generate() input contract for legacy remote code (#5090) by @ZX-ModelCloud in #5154
- fix(calibration): keep all-zero extended attention masks (#5134) by @Qubitium in #5155
- fix(requantize): place embedding endpoints on the model device and dtype (#5116) by @Qubitium in #5156
- fix(load): keep _keep_in_fp32_modules_strict tensors fp32 on reload (#5151) by @Qubitium in #5157
- fix(loader): scope Defuser's cache flush to the converted model's devices (#5105) by @Qubitium in #5158
- fix(torch): serialize TorchLinear's first compile with model loads (#5127) by @Qubitium in #5159
- fix(bitsandbytes): keep the INT8 epilogue below the 2^31 element limit (#5097) by @ZX-ModelCloud in #5160
- fix(requantize): validate the whole norm request before packing embeddings (#5118) by @Qubitium in #5161
- fix(loader): keep runtime checkpoint keys when the HF source map would rename them (#5131) by @Qubitium in #5162
- fix(compile): let dynamo fall back per frame instead of replaying the forward (#5119) by @Qubitium in #5163
- fix(paro): address wide-prefill offsets and dispatch long prefills to cuBLAS (#5096) by @ZX-ModelCloud in #5164
- docs: align ROCm, checkpoint and calibration docs with shipped code (#5121) by @Qubitium in #5165
- fix(paro): keep the CPU rotation forward graph-safe (#5115) by @Qubitium in #5167
- fix(pangu): materialize non-layer replay modules for the last layer (#5135) by @Qubitium in #5169
- fix(bnb): widen the fused 4-bit GEMM row offsets to 64-bit (#5100) by @ZX-ModelCloud in #5170
- fix(embed): refuse fresh endpoint quantize before calibration (#5111) by @Qubitium in #5168
- perf(qqq,vptq): opt portable torch kernels into compiled generation (#5102) by @Qubitium in #5171
- fix(awq): score Hymba's mixer input instead of replaying the KV-reuse block (#5098) by @ZX-ModelCloud in #5172
- perf(fp8): lean W8A16 decode with exact codes and an FP32 scale epilogue by @Qubitium in #5173
- fix(awq): pack and load GEMV_FAST/LLM_AWQ on ROCm with the portable reader by @ZX-ModelCloud in #5176
- fix(awq): fold a scale group across a balanced MoE device pool (#5104) by @ZX-ModelCloud in #5177
- fix(fp8): measure the row-wise
_scaled_mmcontract before using it (#5174) by @Qubitium in #5178 - test(bnb): opt into the Triton policy at module construction (#4764) by @Qubitium in #5179
- fix(awq): never publish the shift table from a CUDA-graph capture (#5094) by @Qubitium in #5180
- fix(fp8): publish non-block FP8 as gptqmodel_fp8 so transformers refuses it (#5128) by @Qubitium in #5181
- fix(compile): scope the compiled forward to the model's device (#5101) by @Qubitium in #5182
- fix(gguf): round Q8_0 fallback ties away from zero by @ZX-ModelCloud in #5183
- fix(awq): bound token-row activation capture to sampling budget (#5137) by @Prajwal7-2002 in #5175
- perf(exl3): fuse the bf16 pre-scale and re-scale into one launch each (#5148) by @Qubitium in #5184
- fix(gguf): reject Q8_1 blocks whose FP16 auxiliary fields overflow by @ZX-ModelCloud in #5185
- fix(qqq): keep per-channel empty s_group buffers in checkpoints (#5126) by @Qubitium in #5186
- test: realign the qwen3-omni decoder walk and the hessian-release oracle (#4769) by @Qubitium in #5187
- perf(vptq): share one segment plan per k-means iteration by @ZX-ModelCloud in #5189
- fix(gsq): quantize QQQ calibration on the deployed FP32 activation grid by @ZX-ModelCloud in #5190
- fix(marlin): stop re-reading a group's scale slot after it is refilled (#4913) by @Qubitium in #5188
- fix(marlin): make the MXFP8 and MXFP4 microscaling paths work by @Qubitium in #5193
- chore(marlin): drop the sm_75 (Turing) path; require sm_80+ by @Qubitium in #5200
- fix(awq): make the routed-expert capture a real uniform reservoir (#5204) by @Qubitium in #5212
- fix(fp8): refuse every inverse-scale FP8 layout instead of publishing
fp8(#5202) by @Qubitium in #5214 - perf(marlin): use 128x64 tiles when the output grid is small by @Qubitium in #5216
- test(docs): cite models/base.py by symbol so the guard stops drifting (#5201) by @Qubitium in #5217
- test(awq): guard #5085's position cap on the capture-time token-row plan (#5215) by @Qubitium in #5220
- fix(exl3): build the torch kernel constants outside inference mode (#5213) by @Qubitium in #5223
- fix(quantization): score the unshrunk SmoothMSE range as a candidate by @ZX-ModelCloud in #5221
- fix(exl3): map the scalar bf16 conversion in the HIP shim (#5208) by @Qubitium in #5227
- fix(requant): stop writing the no-op dynamic scale_search key (#5206) by @Qubitium in #5229
- fix(nf12): keep the decoded weight out of InferenceMode so grad-mode forwards work (#5211) by @Qubitium in #5230
- fix(hf): keep native RoPE initializers ahead of the remote-code shim (#5226) by @Qubitium in #5231
- fix(paro): join mixed-shape calibration captures on the row axis (#5198) by @Qubitium in #5234
- fix(save): keep quantized embed/lm_head endpoints in the re-save shell (#5207) by @Qubitium in #5235
- perf(awq): merge routed-expert reservoir captures in place (#5233) by @Qubitium in #5236
- perf(marlin): take whole output tiles before striping K (hybrid schedule) by @Qubitium in #5222
- fix(awq): publish the shift table only after its device work completes (#5191) by @Qubitium in #5237
- fix(qqq): build the CPU packed cache outside inference mode (#5197) by @Qubitium in #5238
- fix(load): require every kernel's checkpoint payload (#5218) by @Qubitium in #5239
- fix(compile): keep nested torch_compile wrappers trace-clean (#5224) by @Qubitium in #5240
- fix(awq): gate GEMV ROCm refusal on the target device, not the torch build (#5219) by @Qubitium in #5241
- feat(fp8): make FP8 load-only and remove the invented gptqmodel_fp8 export by @Qubitium in #5242
- fix(loader): place non-persistent buffers when a single-device map skips hooks (#5232) by @Qubitium in #5243
- fix(vptq): skip the per-layer compile when the GIL is disabled (#5228) by @Qubitium in #5244
- refactor(mlx): remove the invented FP8 Metal kernels and loader by @Qubitium in #5245
- fix(hf): reselect kernels after
device_map="auto"resolves to an offload map (#5194) by @Qubitium in #5247 - fix(bitsandbytes): chunk fp16 INT8 activation quant above 2^31 elements (#5205) by @Qubitium in #5248
- fix(offload): keep strict fp32 tensors fp32 in the lazy meta shell (#5203) by @Qubitium in #5249
- fix(paro): make the dense/split-K prefill crossover shape-aware (#5210) by @Qubitium in #5250
- fix(compile): serialize every compiled call with model loads (#5209) by @Qubitium in #5251
- fix(awq): launch GEMV Triton rows on grid axis 0 and address them in 64 bits (#5192) by @Qubitium in #5252
- fix(paro): index fused rotation rows in 64 bits (#5196) by @Qubitium in #5253
- Update version.py by @Qubitium in #5254
- Summarize GPT-QModel 7.11.4 in the README by @Qubitium in #5255
- fix(optimize): stop forcing expandable_segments:True before import gptqmodel (#5199) by @n-o-t-mugen in #5256
- fix(fp8): resolve the W8A8
_scaled_mmcontract outside forward (#5195) by @Qubitium in #5257 - fix(paro): make the non-identity rotation trace- and capture-clean (#5225) by @Qubitium in #5258
Full Changelog: v7.11.2...v7.11.4