Skip to content

v0.13.0

Latest
Latest

Choose a tag to compare

View all tags
@dsikka dsikka released this 11 Aug 21:08
· 34 commits to main since this release
0.13.0
de46bfd
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
ab6cbf09-a397-40d4-82da-b790a4b0fb39

Key Highlights

  • REAP Expert Pruning — #2864: New modifier for structurally pruning Mixture-of-Experts (MoE) models by removing individual experts based on calibration-based saliency scores. Based on the REAP the Experts paper.
  • Arbitrary Bit-Width Quantization (Humming) — ct#732, ct#785: Dense packing for non-power-of-2 bit widths (3, 5, 6, 7) with no wasted bits, plus 16 new WxAy presets covering W2–W8 weights with A4, A8, or A16 activations.
  • Observer Fusion and Deletion — #2865: Refactored observer lifecycle and significantly reduced memory usage for large models due to observer statistics persisting after calibration.
  • Expanded MoE Architecture Support — #2847: Extended MoE linearization to support a broader range of architectures, including Transformers v5.13.0 models.
  • Improved XPU Compatibility — #2776, #2884: Migrated torch.cuda calls to torch.accelerator for Intel XPU support.
  • AutoRound Sub-Bit Quantization — #2895: Added sub-bit quantization, including W2A16 attention / W4A16 MLP mixed precision.
  • Pre-Quantized Model Support — #2909: oneshot now provides experimental support for pre-quantized models, provided the targeted layers have not been previously quantized.

REAP Expert Pruning

REAP (Router-weighted Expert Activation Pruning) structurally compresses MoE models by permanently removing individual experts based on saliency scores computed during calibration. The algorithm is introduced in the REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression paper.

REAP can be combined with quantization modifiers. For example, users can prune low-saliency experts first and then quantize the remaining model to FP8 or NVFP4.

Arbitrary Bit-Width Quantization (Humming)

Dense Packing for Non-Standard Bit Widths

The pack_quantized compressor in compressed-tensors now uses dense cross-element packing (ct#732). Previously, 3-, 5-, 6-, and 7-bit formats used padded packing that wasted bits.

The new implementation:

  • Packs 32 consecutive intB elements into exactly num_bits int32 words.
  • Uses no wasted bits and splits elements across int32 boundaries when needed.
  • Supports activation quantization in addition to weight-only schemes.

Expanded WxAy Quantization Presets

A new _int_wnam() helper generates valid integer WxAy combinations (ct#785), adding 16 presets covering W2–W8 weights with A4, A8, or A16 activations, including W3A8, W5A16, and W6A8.

All presets use group-128 symmetric weights and token-wise dynamic symmetric activations and are supported in vLLM as of vllm#46390.

W2–W7 weight-only (A16) schemes were also added as standalone presets (ct#760), extending the previous W4A16 and W8A16 presets.

Observer Fusion and Deletion

Observer lifecycle management was refactored to fix a memory leak where statistics persisted after calibration (#2865):

  • A dedicated fusion_handler manages fused observer groups.
  • Statistics are deleted only after the full fusion group completes.
  • Weight observers skip redundant observation when statistics already exist, reducing recomputation for AWQ/GPTQ workflows.

Expanded MoE Support

MoE linearization now supports a broader range of architectures, including models introduced in Transformers v5.13.0 (#2847). Import patterns were also refactored for backwards compatibility.

Cohere2MoE SpinQuant support was added (#2867), including special handling for its parallel transformer block where one input_layernorm feeds attention, MLP, and the router.

Lifecycle Improvements

Calibration Events

Calibration events are now first-class lifecycle hooks (#2783, #2784):

  • Added on_calibration_start, on_sequential_epoch_end, and on_calibration_end.
  • Calibration start/end logic is handled by the Modifier base class.
  • Renamed calibration_epoch_start/end to calibration_start/end.

Calibration Requirement Check

Each modifier now declares requires_calibration_data() (#2947), replacing the hardcoded pipeline registry list. GPTQ, AutoRound, SparseGPT, Wanda, SmoothQuant, AWQ, and REAP explicitly require calibration.

Pipeline Device Movement

Device movement logic has been removed from pipelines (#2846). load_offloaded_model now handles distributed dispatch and disk offloading, simplifying pipeline logic.

Distributed Improvements

  • Module Parallel Calibration — #2785: Weight calibration can run in parallel across distributed workers.
  • Suspend Distributed Timeout — #2868: Supports saving very large or disk-offloaded models taking more than 10 minutes.
  • AutoRound DDP — #2844, #2934: Added Qwen MoE DDP example and fixed rank-local device placement.
  • DDP Smoke Tests — #2769: Added comprehensive DDP tests with subsequent stability fixes in #2840, #2857, #2863, and #2943.

Performance

  • torch.compile for MSE Observer — #2384: Added chunked execution support for torch.compile, with significant speedups for activation quantization. Pass enable_compile=True to oneshot to enable.
  • IntermediatesCache pin_memory Fix — #2813: Fixed a CUDA OOM issue with nested dispatchers.
  • Reduced Default Save Shard Size — #2927: Reduced the default shard size to 20 GB for improved network transfer performance.

XPU Compatibility

All main-path torch.cuda calls have been migrated to torch.accelerator (#2884). A torch.cuda linter was added to CI (#2776), along with XPU Docker and testing infrastructure (#2945).

New Model Support

  • DeepSeek V4 Pro — #2858, with automatic MTP weight copying (#2951).
  • GLM 5.2 — #2869
  • GLM 4.6 — #2343
  • HunyuanMoE V3 — #2928
  • Gemma 4 — #2816
  • Mellum2 — #2832
  • Cohere2MoE SpinQuant + NVFP4 — #2867
  • Input Embedding Quantization example — #2830
  • New AWQ/SmoothQuant mappings for Step3p5 (#2770), Granite (#2797), Nanbeige (#2966), and Qwen3.5 MoE (#2718, #2727).

Breaking Changes

  • Removed sparsity-preserving logic from GPTQ — #2860
  • Removed IMatrixGatherer; functionality consolidated into observers — #2920
  • Deprecated GPTQ actorder=group — #2893
  • Deprecated reindex_fused_weights — #2737
  • QuIP now defaults to input (v) rotations only — #2815

New Contributors

Full Changelog: 0.12.0...0.13.0

Contributors

jethac, suluner, and 18 other contributors
Assets 4
Loading