Skip to content

GPTQModel v7.7.0

Latest
Latest

Choose a tag to compare

View all tags
@Qubitium Qubitium released this 23 Oct 10:42
v7.7.0
bcb44c1
This commit was created on GitHub.com and signed with GitHub鈥檚 verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

GPTQModel 7.7.0 馃殌

  • CPU inference: Added the shared AVX512-BF16 int4_qzero_group_scaled_cpu kernel for GPTQ/AWQ group sizes 16/32/64/128/256/512 and prioritized it ahead of ATen when supported. Added packed/low-precision CPU decode paths for QQQ, VPTQ, EXL3, FP8, NVFP4, and BitsAndBytes, plus oneDNN BF16x3 accumulation and shape-aware BitsAndBytes INT8 dispatch. GPTQ/AWQ measured up to 2.02x faster than ATen; BitsAndBytes INT8 reached 1.28x at M=64.
  • GGUF: Added fused CPU Q4_0/Q8_0/Q4_K/Q5_K/Q6_K forward with FP32 accumulation under oneDNN BF16x3, fused bias, per-shape autotuning, and no retained dense weights. K-quant fused paths measured up to 1.92x faster than dense at M=7.
  • Quantized storage: Removed persistent dense, dequantized, expanded, and INT4-to-INT8 inference caches across GPTQ/AWQ, QQQ, EXL3, VPTQ, FP8, FP4, GGUF, BitsAndBytes, and INT8. Only packed weights, scales, zero points, and metadata persist; temporary dequantization uses bounded tiles.
  • Quantization quality: Added token-normalized GPTQ loss reporting, on-device embedding diagonal flooring and VPTQ Hessian damping, and QQQ vLLM compatibility/CUDA saturation fixes.
  • Models and runtimes: Added LLaVA quantization support, fused VPTQ CUDA inference, and hardened remote LoRA adapter loading.
  • Docs and tooling: Added the GPTQ FSDP/PEFT LoRA example, multi-GPU MoE quantization guide, and independent Torch-oracle accuracy and CPU speed benchmarks.
  • Dependencies: Refreshed runtime and optional dependency floors to current stable releases, including PyTorch 2.14.1, Transformers 5.20.2, Datasets 5.1.0, and Optimum 2.3.0. vLLM/SGLang extras retain their upstream framework pins and should be installed in dedicated environments.

Upgrade notes

  • Python 3.10 remains supported; dependency markers select compatible NumPy, filelock, Pillow, PyArrow, vLLM, SGLang, and MLX-LM releases.
  • Runtime floors are now PyTorch 2.14.1 and Transformers 5.20.2.
  • BACKEND.GPTQ_TORCH_INT8 / TORCH_INT8 accept native 8-bit GPTQ sources only. AWQ_TORCH_INT8 / TORCH_INT8_AWQ are retired.
  • Dense/dequantized inference-cache opt-ins now warn and remain disabled. Only packed weights, scales, zero points, and metadata persist.
  • vLLM/SGLang extras remain dedicated-environment integrations because their latest releases pin older Torch/Transformers versions.

Validation

  • Wheel and sdist built for 7.7.0; twine check passed.
  • Core requirements resolved on Python 3.10, 3.11, 3.12, and 3.14.
  • Focused CPU/storage/config suite: 193 passed, 2 skipped.
  • PR CI passed: Ruff, setuptools compatibility, and CodeQL.
Full list of changes

What's Changed

  • [MODEL] support openvla by @ZX-ModelCloud in #3258
  • Fix XPU AWQ fused kernel: cast zero points to int8 by @jiqing-feng in #3259
  • BitBLAS: support bf16 checkpoints via fp16 compute; fix multi-GPU illegal memory access (#1192) by @ZX-ModelCloud in #3254
  • [MODEL] support qwen2_audio by @ZX-ModelCloud in #3260
  • Fix ROCm JIT builds for GPTQ block updates by @Qubitium in #3261
  • Use AWQ Triton GEMM on ROCm by @Qubitium in #3262
  • ROCm/gfx950: fused GPTQ 4-bit GEMV for single-token decode by @Qubitium in #3263
  • ROCm/gfx950: widen fused GPTQ GEMV coverage to narrow projections by @Qubitium in #3264
  • ROCm/gfx950: extend fused GPTQ GEMV to 2-bit and 8-bit weights by @Qubitium in #3265
  • gitignore hipify-generated ROCm sources by @Qubitium in #3266
  • ROCm/gfx950: drop triton.autotune from GPTQ dequant kernel by @Qubitium in #3267
  • ROCm/gfx950: fused GPTQ GEMM for small-batch (M=2..16) forwards by @Qubitium in #3268
  • fix(pack): make pack_gpu bit-exact with the CPU block packer by @Qubitium in #3269
  • perf(pack): pack quantized layers on the accelerator by default by @Qubitium in #3270
  • fix(awq): pick a device-capable AWQ GEMM pack kernel off CUDA by @Qubitium in #3271
  • perf(awq): vectorize the AWQ pack loop in the torch and GEMM packers by @Qubitium in #3272
  • fix(awq): let the FORMAT.GEMV packer register its bias buffer by @Qubitium in #3273
  • perf(awq): vectorize the AWQ GEMV pack loop by @Qubitium in #3274
  • perf(awq): vectorize the AWQ GEMV_FAST packer by @Qubitium in #3275
  • perf(gptq): batch the pack_gpu qweight packer for 2/4/8-bit by @Qubitium in #3276
  • fix(paroquant): build the fused rotation kernel on ROCm/gfx950 by @Qubitium in #3277
  • fix(qqq): initialize the bias attribute when register_buffers=False by @Qubitium in #3278
  • perf(qqq): reuse the fused CUDA block update in the QQQ lifecycle by @Qubitium in #3279
  • perf(telemetry): stop spawning the vendor SMI tool when it cannot report by @Qubitium in #3280
  • perf(awq): select the best clip with torch.where instead of masked scatter by @Qubitium in #3281
  • perf(paroquant): hoist the per-rotation constants out of the rotation backward by @Qubitium in #3282
  • fix(exll3): keep HIPify from mangling indexed kernel launches by @Qubitium in #3283
  • fix(exll3): cast kernel pointers for hipFuncSetAttribute by @Qubitium in #3284
  • fix(exll3): keep the cuBLAS-only status out of the HIP build by @Qubitium in #3285
  • fix(exll3): use __activemask() as the warp-shuffle mask by @Qubitium in #3286
  • fix(exll3): replace the NVIDIA-only codebook PTX with portable bitwise ops by @Qubitium in #3287
  • fix(exll3): let the quant decoders build without the CUDA-only PTX header by @Qubitium in #3288
  • feat(exll3): build the extension on ROCm and accelerate the EXL3 quant lifecycle by @Qubitium in #3289
  • fix(hadamard): build the fast Hadamard transform on ROCm by @Qubitium in #3290
  • perf(exl3): vectorize the global-scale tile sampling by @Qubitium in #3291
  • perf(exl3): cache the codebook decode in the ROCm trellis quantizer by @Qubitium in #3292
  • perf(exl3): run the trellis cost accumulation in fp32 on ROCm by @Qubitium in #3293
  • perf(paroquant): fuse the rotation gradient accumulation into one ROCm kernel by @Qubitium in #3294
  • perf(paroquant): fuse the two backward rotations into one ROCm kernel by @Qubitium in #3295
  • perf(paroquant): fuse the backward rotation sweep and its gradient into one ROCm kernel by @Qubitium in #3296
  • perf(awq): hoist the per-ratio forward scaffolding out of the AWQ scale search by @Qubitium in #3297
  • perf(awq): keep the scale-search weight master on the module device by @Qubitium in #3298
  • fix(paroquant): stop the compiled large-train quant path from aborting the lifecycle by @Qubitium in #3299
  • perf(awq): trim the per-ratio clip-search launches by @Qubitium in #3301
  • Recover post-quant accuracy for free: RMSNorm gain tuning as an optional requant pass by @ToukoUrsin in #3255
  • perf(paroquant): drop per-step host dispatch from the rotation path by @Qubitium in #3302
  • perf(awq): accumulate the scale-search reconstruction loss on device by @Qubitium in #3303
  • perf(exl3): give each quantization worker its own tile scratch and stream by @Qubitium in #3308
  • perf(exl3): order the quantization streams against the caller, not the device by @Qubitium in #3309
  • perf(exl3): run each worker's module quantization on its own stream by @Qubitium in #3310
  • perf(exl3): allocate tile-quantizer outputs uninitialized by @Qubitium in #3311
  • docs(exl3): record MI350X quantization-lifecycle concurrency analysis by @Qubitium in #3312
  • perf(exl3): allocate the LDLQ outputs and tile scratch uninitialized by @Qubitium in #3313
  • docs(exl3): record the MI350X wave accounting and kernel instruction floor by @Qubitium in #3314
  • perf(paroquant): fold the STE detach algebra into autograd nodes by @Qubitium in #3315
  • perf(paroquant): fold the channel scales into the fused rotation by @Qubitium in #3316
  • perf(paroquant): fuse the asymmetric STE into one autograd node by @Qubitium in #3317
  • perf(paroquant): run the affine STE quantize/dequantize in fused Triton kernels by @Qubitium in #3318
  • perf(paroquant): take the fused Triton forward on value-only quantize calls by @Qubitium in #3319
  • perf(paroquant): fold the frozen-theta rotation backward into one reverse sweep by @Qubitium in #3320
  • perf(paroquant): batch the best-state snapshot into one copy per dtype group by @Qubitium in #3321
  • perf(paroquant): overlap the rotation-gradient reduce tile walk by @Qubitium in #3322
  • perf(paroquant): drop the dead pre-epoch train evaluation by @Qubitium in #3323
  • perf(paroquant): fold stage loss on device instead of per-step readback by @Qubitium in #3324
  • perf(paroquant): fold the STE scale-divisor factor into the fused forward by @Qubitium in #3325
  • perf(paroquant): merge identical stage optimizer param groups by @Qubitium in #3326
  • perf(paroquant): skip the stage angle reset when no angle slot is padded by @Qubitium in #3327
  • perf(paroquant): fold the step loss into the epoch total in place by @Qubitium in #3328
  • perf(paroquant): reuse one unit seed for every scalar-loss backward by @Qubitium in #3329
  • Warn when dynamic rules cannot be honored by fused-MoE inference engines (#1455) by @ZX-ModelCloud in #3300
  • feat(gemma3): declare Gemma 3 VL modality and add image calibration support by @ZX-ModelCloud in #3307
  • perf(paroquant): fold grouped-stage losses on the device by @Qubitium in #3330
  • perf(exl3): skip the redundant device->host copy of the cached L factor by @Qubitium in #3331
  • perf(awq): drop one kernel per clamp grid step and one fixup pass per scale by @Qubitium in #3332
  • perf(awq): drop the redundant symmetric clamp in the pseudo-quantizer by @Qubitium in #3333
  • perf(awq): fold the per-step weight restore into the next scale multiply by @Qubitium in #3334
  • perf(exl3): fuse the tensor-core tile permute into one gather by @Qubitium in #3335
  • perf(exl3): build the tensor-core tile layout with a permute, not a gather by @Qubitium in #3336
  • perf(paroquant): fuse the symmetric quantize/dequantize STE into one kernel by @Qubitium in #3337
  • perf(paroquant): publish the bounded scale from the affine STE forward by @Qubitium in #3342
  • perf(paroquant): make the stage AMP mode configurable via opt_amp by @Qubitium in #3343
  • perf(gptq): build g_idx with tensor ops instead of a python loop by @Qubitium in #3344
  • perf(paroquant): default opt_amp to auto (no AMP on ROCm) by @Qubitium in #3345
  • perf(gptq): collapse group bookkeeping for uniform 128-column blocks by @Qubitium in #3346
  • perf(qqq): collapse group bookkeeping for uniform 128-column blocks by @Qubitium in #3347
  • perf(quant): skip redundant maxq.to(dev) in find_params by @Qubitium in #3348
  • perf(gptq,qqq): drop redundant block-buffer staging in the 128-column loop by @Qubitium in #3349
  • fix(paroquant): skip stage CUDA-graph capture on ROCm where it deadlocks by @Qubitium in #3350
  • perf(paroquant): drop parent lock around the pack call by @Qubitium in #3351
  • perf(threadx): start device warmups at pool construction, not on the first task by @Qubitium in #3352
  • perf(paroquant): persist resolved fused-rotation launch configs across processes by @Qubitium in #3353
  • perf(quantize): join the background linear prewarm before the capture forward by @Qubitium in #3354
  • perf(startup): import torch._dynamo lazily instead of at gptqmodel import by @Qubitium in #3355
  • perf(startup): resolve the optional datasets dependency lazily by @Qubitium in #3356
  • chore(startup): drop the unused import torch._dynamo from models/base.py by @Qubitium in #3357
  • fix(fallback): reject malformed fallback threshold strings by @peter-positron in #3341
  • perf(rocm): quantize with rocBLAS to skip hipBLASLt Tensile-library loads by @Qubitium in #3358
  • perf(rocm): build the linalg-warmup SPD input without a BLAS GEMM by @Qubitium in #3359
  • perf(rocm): keep the quantizer range search off the host-sync path by @Qubitium in #3360
  • fix(qwen3-vl): accept pre-tokenized calibration rows by @peter-positron in #3338
  • fix(mimo): preserve layer-specific attention inputs in replay by @peter-positron in #3339
  • fix(writer): refuse incomplete declared auxiliary source tensors by @peter-positron in #3340
  • Generalize RMSNorm gain tuning to NormConfig with selectable norm types by @Qubitium in #3306
  • fix(norm): correct gain probing, shared parameters, and legacy API compatibility by @Qubitium in #3362
  • refactor(norm): remove legacy RMSNorm tuning API by @Qubitium in #3363
  • perf(rocm): warm the quantization kernels behind the model load by @Qubitium in #3361
  • perf: skip disk write benchmark for in-memory quantization by @Qubitium in #3364
  • perf(rocm): prewarm GPTQ GAR topk before quantization by @Qubitium in #3365
  • perf(rocm): prewarm cold Hessian Cholesky kernel by @Qubitium in #3366
  • perf(rocm): use Torch VRAM metrics on virtual GPUs by @Qubitium in #3367
  • perf(quant): collapse symmetric empty-group detection by @Qubitium in #3368
  • docs(peft): document LoRA loading and merge behavior by @ZX-ModelCloud in #3369
  • perf(awq): coalesce scale-search replay batches by @Qubitium in #3371
  • perf(awq): prepare scale-search forward once by @Qubitium in #3373
  • fix(quantizer): bypass zero scales without NaN by @ZX-ModelCloud in #3370
  • fix(qqq): reject unsupported activation ordering by @ZX-ModelCloud in #3372
  • perf(awq): select clip bounds with one argmin by @Qubitium in #3374
  • perf(awq): select clip bounds with one argmin by @Qubitium in #3375
  • feat(vptq): native VPTQ quantization and Torch inference path by @ZX-ModelCloud in #3305
  • perf(gptq): keep MSE parameter selection on device by @Qubitium in #3376
  • perf(qqq): keep range search and maxq on device by @Qubitium in #3377
  • perf(vptq): keep empty-cluster fallback on device by @Qubitium in #3378
  • perf(vptq): keep Hessian preprocessing on device by @Qubitium in #3379
  • perf(gptq): use maxq mirror for fused-path selection by @Qubitium in #3380
  • perf(vptq): accumulate loss chunks on device by @Qubitium in #3381
  • perf(vptq): combine Hessian usability checks on device by @Qubitium in #3382
  • perf(vptq): defer k-means inertia read until after centroid update by @Qubitium in #3383
  • perf(quant): neutralize dead channels with device masks by @Qubitium in #3384
  • perf(awq): select best scale on device by @Qubitium in #3385
  • perf(exl3): accumulate proxy-error traces on device by @Qubitium in #3386
  • perf(qqq): track quantizer readiness on the host by @Qubitium in #3387
  • perf(exl3): update Hessian diagonals through views by @Qubitium in #3389
  • perf(awq): combine post-quantization NaN guards by @Qubitium in #3390
  • perf(nvfp4): accumulate activation amax on device by @Qubitium in #3391
  • perf(gptq): cache per-sample keep-mask flags per batch by @Qubitium in #3392
  • perf(awq): compute previous-subset scale once per stage by @Qubitium in #3388
  • perf(weight-only): compute dequant error on device by @Qubitium in #3393
  • perf(gptq): reuse zero-scale mask during sanitization by @Qubitium in #3394
  • perf(vptq): validate index bounds with one reduction by @Qubitium in #3395
  • perf(paroquant): combine identity-rotation host reads by @Qubitium in #3396
  • perf(awq): vectorize square-mask lengths by @Qubitium in #3397
  • perf(awq): vectorize kept-token gather by @Qubitium in #3398
  • perf(awq): coalesce previous-subset scale reads by @Qubitium in #3399
  • perf(qqq): drop redundant post-loss device sync by @Qubitium in #3400
  • perf(mask): avoid host branch for numeric masks by @Qubitium in #3401
  • perf(awq): drop redundant entry NaN scan by @Qubitium in #3402
  • perf(vptq): keep Hessian damping on device by @Qubitium in #3403
  • Fix GPTAQ token and loss reporting by @ZX-ModelCloud in #3406
  • perf(cpu): add shared AVX512-BF16 GPTQ/AWQ inference kernel by @Qubitium in #3408
  • perf(gptq): keep embedding diagonal floor on device by @Qubitium in #3404
  • feat(vptq): add fused CUDA inference kernel by @ZX-ModelCloud in #3405
  • docs: explain multi-GPU MoE quantization by @ZX-ModelCloud in #3410
  • perf(cpu): optimize quantization kernels and runtime caches by @Qubitium in #3412
  • [MODE] support llava quantization by @ZX-ModelCloud in #3411
  • perf(exl3): use oneDNN BF16x3 CPU matmul by @Qubitium in #3415
  • fix: block untrusted remote LoRA adapters by @mikemikimike in #3409
  • perf(bitsandbytes): enable packed AVX512-BF16 CPU decode by @Qubitium in #3414
  • fix(qlinear): do not cache dense dequantized weights by @Qubitium in #3417
  • fix(exl3): avoid dense dequant cache with chunked CPU decode by @Qubitium in #3418
  • perf(cpu): vectorize VPTQ/EXL3 decode and align ATen int4 groups by @Qubitium in #3419
  • perf(cpu): shape-dispatch BnB INT8 and compile EXL3 decode by @Qubitium in #3420
  • perf(fp8): compile CPU dequant matmul decode by @Qubitium in #3421
  • perf(fp4): use native CPU decode for NVFP4 linear by @Qubitium in #3422
  • perf(bnb): shape-dispatch packed CPU 4-bit decode by @Qubitium in #3423
  • perf(qqq): packed CPU inference without dense weight caches by @Qubitium in #3426
  • fix(torch): remove persistent dense inference weight caches by @Qubitium in #3427
  • Fix QQQ vLLM compatibility and CUDA saturation by @ZX-ModelCloud in #3425
  • fix(int8): retire expanded low-bit runtime weights by @Qubitium in #3428
  • docs: add GPTQ FSDP PEFT LoRA example by @ZX-ModelCloud in #3424
  • perf(cpu): tune and validate optimized quantized kernels by @Qubitium in #3430
  • chore(release): prepare 7.7.0 by @Qubitium in #3431

New Contributors

Full Changelog: v7.6.0...v7.7.0

Contributors

Qubitium, jiqing-feng, and 4 other contributors
Assets 3
Loading