Skip to content

v0.15.0

Latest
Latest

Choose a tag to compare

View all tags
@dhuangnm dhuangnm released this 29 Oct 17:18
· 25 commits to main since this release
0.15.0
04546e0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
LLM Compressor v0 15 0 Banner

LLM-Compressor v0.15.0

This release adds HIGGS mixed-precision quantization, MoNe expert pruning, block-wise Quantization-Aware Distillation (QAD), layerwise compression with a rewritten offloading path, checkpoint-space REAP workflows, faster observers, and MXFP6 support.

It also includes bug fixes, expanded model support, performance improvements, and dependency updates.

Key Highlights ✨

  • HIGGS autoquantization (#3028): get_higgs_config selects optimal quantization schemes for each layer to maximize estimated accuracy under a target average bitwidth.
  • Block-wise QAD (#3029): QADModifier distills against cached, unquantized subgraph outputs without requiring a teacher model.
  • Layerwise compression and offloading (#3162, #3221, #3228): Converts large checkpoints one subgraph at a time using CPU/pinned-memory staging and background prefetching.
  • REAP and MoNe (#3281, #3343): Enables direct checkpoint pruning using per-expert saliency reports. MoNe collapses low-saliency experts into "novice" vectors for higher throughput.
  • Faster observers (#3348, #3233): Exponential E8M0 grids make MXFP4 MSE observation approximately 6× faster. A shared MSE/iMatrix Triton kernel delivers 1.2–3.4× NVFP4A16 speedups.
  • Model-free PTQ and MXFP6 (#3262, #3250, #3286): Adds offline SpinQuant R1/R2 rotations, converter chains, max_workers="auto", and MXFP6A16/A8 support.
  • Expanded model and hardware coverage: Adds a GPTQ Triton kernel for Intel XPU (#3339) and Transformers MtpModel loading for MTP layers (#3225).

HIGGS Mixed-Precision Quantization

HIGGS introduces automatic mixed-precision quantization configuration.

  • Uses HiggsMSECollectorConverter to collect per-layer MSE.
  • Calculates heuristic sensitivity and solves an integer linear program (ILP) to meet the target bitwidth.
  • Collection is model-free, does not write a checkpoint, and takes under 10 seconds for Llama-3.1-8B-Instruct.
  • Composes with both the oneshot and ModelFreePtq entry points.
  • Based on the HIGGS paper.

At a 6.0-bit target, the following WikiText perplexity results were achieved:

Configuration WikiText perplexity
HIGGS + QuantizationModifier 11.05
HIGGS + GPTQ 10.99

Quantization-Aware Distillation (QAD)

QADModifier enables block-wise distillation after QuantizationModifier or GPTQModifier.

  • AWQ integration: The supported composition order is AWQ → Quantization → QAD.
  • Shared calibration: Reuses one calibration dataset with the preceding modifier.
  • Memory efficiency: Caches inputs and teacher outputs with CPU/disk offloading and prefetching.
  • Joint training: Supports training N targets per subgraph.
  • Loss and optimization: Supports FP32 MSE loss, AdamW with FP32 master weights, dynamic loss scaling, gradient accumulation, and gradient clipping.
  • Validation: Uses a deterministic 10% validation hold-out.
  • Weight observation: Re-observes weights before training, at each epoch, and before finalization.
  • Fixed quantization parameters: Setting reobserve_weights=False keeps the preceding quantization parameters fixed.

Requirements:

  • A sequential pipeline.
  • A preceding weight-quantization modifier.
  • Single-process calibration.

REAP and MoNe Expert Pruning

REAP

  • report_path stores per-expert router weights, routed-token counts, and saliency in JSON format.
  • Setting sparsity=0.0 enables report-only operation.
  • ExpertPruner prunes checkpoints directly from safetensors.
  • Supports uniform per-layer and non-uniform cross-layer sparsity while respecting the num_experts_per_tok floor.
  • Supports 2D and 3D expert layouts, per-expert router vectors, and hash-routed layers.
  • Skips GPT-OSS because of incompatible routing (#3223).

MoNe

  • Collapses low-saliency experts into novice vectors.
  • Keeps the router at full R + N width.
  • Repacks novice experts into the checkpoint.
  • Writes moe_sparsity_config with method="novice".

MoNe/REAP statistics are now reduced in float32 on the accelerator, fixing distributed crashes and desynchronization (#3397).

Sequential Pipeline and Offloading

  • Layerwise decompression: Added to the sequential pipeline, avoiding upfront decompression of quantized models before passing them to oneshot.
  • Layerwise compression: Subgraphs can be eagerly compressed after calibration.
  • Background staging: Overlaps preparation of the next subgraph with current-layer calibration.
  • Activation prefetching: Always enabled.

Prefetching Fixes

  • Retains buffer references to prevent AWQ DDP divergence (#3258).
  • Skips unsupported record_stream operations (#3270).
  • Falls back to synchronous fetching outside CPU/CUDA (#3274).

Pipeline and Tracing Improvements

  • Basic calibration now uses the sequential pipeline as one untraced target (#3336).
  • log_sequential_error now reports signal-to-quantization-noise ratio (SQNR) (#3165).
  • Attention is treated as a sequential target. Non-eager attention is now the default to eliminate the O(seq_len²) attention mask (#3255).
  • Tracing fixes cover autowrapped modules, concrete keyword arguments, and Hub kernels on meta tensors (#3264, #3332, #3345).

Observer Improvements

  • Exponential E8M0 grids now consider only powers of two, reducing MXFP4 observation time by approximately 6×.
  • Magnitude normalization now matches llama.cpp iMatrix behavior (#3348).
  • A unified MSE/iMatrix Triton kernel adds tensor weighting and shared eager implementations (#3233).
  • Fused module mappings are centralized in FUSED_MODULE_MAPPINGS (#3224).
  • end_calibration now calibrates activation quantization parameters for already-quantized modules traced through by the sequential pipeline (#3245).
  • Fixed observer-kernel compilation with Triton 3.9 (#3406).

Model-Free Post-Training Quantization (PTQ)

  • SpinQuantConverter folds RMSNorm scales and R1/R2 Hadamard rotations directly into the residual stream in safetensors (#3262).
  • max_workers="auto" is now the default (#3250).
  • The pipeline delegates to convert_checkpoint, removing duplicated device, validation, and configuration logic (#3206).
  • Validation no longer runs observers on meta tensors (#3247).

Model Support

  • Compressed 3D MoE repacking supports nested fused projections and native 3D keys. Quantized-but-uncompressed experts now raise an error (#3200).
  • Kimi-K3 fused modules are constructed on the execution device rather than the meta device (#3259).
  • MTP layers load through Transformers MtpModel. Unsupported architectures fall back to unchanged MTP weights (#3225).
  • The DeepSeek-V3 example now uses an NVFP4 backbone and FP8 MTP in a single oneshot call (#3402).

Documented Checkpoints

  • GLM-5.3 MXFP4xFP8 (#3380).
  • GLM-5.3 NVFP4 (#3230).
  • Nemotron 3.5 Super VL FP8 (#3291).

Performance Improvements

  • Approximately 6× faster MXFP4 observation through exponential E8M0 grids (#3348).
  • 1.2–3.4× faster end-to-end NVFP4A16 through the unified MSE/iMatrix kernel (#3233).
  • Improved pipeline throughput by overlapping background subgraph staging with current-layer computation (#3228).

Bug Fixes

  • Fixed distributed MoNe/REAP statistics through accelerator-based float32 reductions (#3397, #3405).
  • resave_config now uses the loaded model revision (#3390).
  • Forwarded trust_remote_code and revision to the compressed-tensors check (#3313).
  • Fixed nested disable_hooks behavior to restore the enclosing state (#3328).
  • Fixed loading of numbered recipe stage keys (#3316).
  • Prevented Kimi-K3 fused modules from being placed on the meta device (#3259).
  • Fixed AWQ DDP divergence caused by prefetched buffers (#3258).
  • Added safe handling for unsupported record_stream and background prefetch operations (#3270, #3274).
  • Ensured activation quantization parameters are calibrated for modules traced through by the sequential pipeline (#3245).
  • Added early group-size checks to flag narrow layers (#3252).
  • Prevented chat datasets from prepending a second BOS token (#3289).
  • Fixed tracing with concrete keyword arguments so downstream modules are not pruned (#3332).
  • Prevented Hub kernels from running on meta tensors (#3345).
  • Changed torchrun behavior without init_dist from raising an error to issuing a warning (#3231, #3202).
  • Fixed sparsification-reporting imports on Python 3.11 (#3378).
  • Fixed observer-kernel compilation with Triton 3.9 (#3406).

Breaking Changes and Removals

Removed Legacy Sparsity Components

The following legacy components have been removed (#3377, #3382):

  • SparseGPTModifier
  • WandaPruningModifier
  • ConstantPruningModifier
  • MagnitudePruningModifier
  • The OBCQ compatibility shim.
  • Shared sparsity utilities.

Other Breaking Changes

  • Python 3.11 is now the minimum supported Python version (#3299).
  • preprocessing_func is deprecated (#3227).
  • sequential_prefetch has been permanently enabled (#3364).
  • Running torchrun without init_dist now produces a warning instead of an error (#3231).

Dependencies and Infrastructure

  • Updated compressed-tensors to version 0.20.1 (#3398, #3407).
  • Relaxed dependency upper bounds (#3393).
  • Raised the Transformers upper bound to 5.21.0 and gated KV cache tests to version 5.22 and later (#3401).
  • Added Python 3.14 to GPU test configuration (#3349, #3331).
  • Added PuLP as the optional higgs extra.

Documentation

  • Added the Open Research Hub (#3280).
  • Added a Simplified Chinese README (#3188).
  • Refreshed README model highlights, Key Models pages, and quantization documentation (#3394, #3376, #3379, #3384).

Tests and Continuous Integration

  • Extended XPU coverage to AWQ, GPTQ, SmoothQuant, and KV cache (#3049, #3339).
  • Updated DDP comparisons to use deterministic algorithms (#3381, #3334).
  • Consolidated dataset padding and BOS-token test coverage (#3196, #3295).
  • Moved AWQ + NVFP4 evaluation to Qwen3-8B for more stable recovery (#3275).

Full Changelog

New Contributors

Contributors

mgoin, bartowski1182, and 27 other contributors
Assets 4
Loading