Skip to content
Qubitium edited this page Nov 17, 2026 · 14 revisions

🧩 MACaQ: MXFP4 Weights and Activations

🏠 Home · 🧭 Choose a method · 🚀 Quickstart · 🧰 Help

On this page

MACaQ compresses weights and, optionally, activations to MXFP4 using calibration data to protect important signals. This guide is for model builders comparing four-bit floating-point W4A4 inference with a weight-only W4A16 path.

GPT-QModel includes a native port of Huawei CSL's MACaQ, a calibrated quantizer for low-precision weights and activations. This page describes source revision f9594695c.

🧭 Choose and understand

Choose this when… Consider another path when…
You can calibrate a supported model and want MXFP4 weights plus four-bit activations You need calibration-free packing: compare BitsAndBytes or FP8 weight-only
You want to isolate activation rounding by comparing "w4a4" and "w4a16" You need a GPTQ INT4 checkpoint/runtime: see GPTQ
Your runtime accepts native MACaQ state, or you want transform-free W4A16 for stock MLX Your deployment consumes GGUF: choose its supported formats in GGUF

Before you start: install GPT-QModel, select a supported model, and prepare a representative calibration corpus. Input widths must be divisible by both 32 and had_block; there is no automatic padding. The LM head is not quantized. Native Triton execution is declared for NVIDIA CUDA and AMD ROCm gfx1201; use the Torch reference on other supported targets. Backend details describe the remaining limits.

Read in three passes: use it → understand the solve → tune controls. Still choosing? Start with Choosing a Method and Dtype.

🚀 Use it

Quantize, save, and reload

import torch
from gptqmodel import BACKEND, CalibrationConfig, GPTQModel
from gptqmodel.quantization import MacaqConfig

calibration_texts = [
    "Quantization reduces model storage and inference bandwidth.",
    "Use a representative calibration corpus for real workloads.",
]
model = GPTQModel.load(
    "meta-llama/Llama-3.2-1B-Instruct",
    quantize_config=MacaqConfig(
        n_flip_iters=256,
        had_block=32,
        pre_alpha=0.75,
        activation_precision="w4a4",
    ),
    device="cuda:0",
    dtype=torch.bfloat16,
)
model.quantize(CalibrationConfig(dataset=calibration_texts, batch_size=1), backend=BACKEND.AUTO)
model.save("./llama-macaq")
quantized = GPTQModel.load("./llama-macaq", backend=BACKEND.MACAQ_TRITON)

API demo only: the two short strings show the call sequence, not an accuracy recipe. Use a representative corpus and evaluate the resulting model. Save uses safetensors. On a compatible Apple Silicon MLX model, reload with backend=BACKEND.MLX; install the MLX extra first.

🧠 Go deeper

Method, format, and lifecycle

Choice Value Meaning
Method METHOD.MACAQ / "macaq" Calibration-based MACaQ solve, not a GPTQ variant or calibration-free conversion
Config MacaqConfig Import from gptqmodel.quantization
Format FORMAT.MXFP4_MACAQ / "mxfp4_macaq" by default Packed MXFP4 weights plus enabled MACaQ transform state; "mxfp4" when both transforms are disabled
Default activation precision "w4a4" MXFP4 weights and runtime MXFP4 activations
Alternative "w4a16" Same 4-bit weights with non-MXFP4 activations

The default per-Linear solve captures a calibration Hessian, computes an activation-aware diagonal pre-scale (D1), applies a block-Hadamard rotation, rounds weights to OCP MXFP4, screens E8M0 exponent offsets using the Hessian, and optionally refines E2M1 levels with greedy flips. D1 and rotation can be disabled independently. Calibration uses a combined block subset; packed replay with the selected activation precision supplies inputs to later blocks. MACaQ shares calibration infrastructure with GPTQ, but uses a different grid, solve, checkpoint, and runtime.

Supported quantization continuation runs record the packed module's actual compute_dtype so BH and pre_scale restore in the saved dtype even when the source model config declares another dtype. This prevents a resume schema mismatch without changing MACaQ's inference checkpoint format or relaxing continuation admission rules.

The solver now drops unused pre-rotation weight/Hessian references after rotation and releases the rotated Hessian after constructing the flip metric. This reduces temporary memory overlap before exponent screening and the flip sweep, especially when several modules quantize concurrently. Packed codes and E8M0 exponents are unchanged, and no new option is required. See buffer-lifetime regression coverage.

Exponent screening accumulates E2M1 codes in UINT8 and widens signed-grid indices to INT32 for gathering, avoiding full INT64 planes while preserving the weight and activation tie rules. Compiled flip sweeps use fullgraph=True and keep converged iterations in the graph with masked no-op updates, avoiding a data-dependent early return whose Dynamo resume frames could retain the Hessian working set. A future graph break raises rather than silently retaining those frames. With that cycle removed, the flip path no longer forces a full gc.collect() after each Linear; processor finalization retains one collection as the end-of-run cleanup backstop. These are automatic memory/performance changes; compile_flip and the packed format retain their existing contracts. See flip-sweep implementation.

OCP MXFP4 uses E2M1 values and one E8M0 scale byte per 32 values: nominally 4.25 bits/weight, before enabled pre-scale, Hadamard, bias, and other model state. mxfp4_macaq is a checkpoint contract, not a new number system. Generic MXFP4 or GGUF MXFP4 checkpoints are not interchangeable with it: the tensor layout and enabled pre_scale/BH state must also match. See rotation-free weight-only MXFP4 for the transform-free format.

🔬 How MACaQ protects the output

The intuition is to balance large channels, spread outliers with a rotation, then spend rounding effort where calibration says an error matters most. The recipe comes from Muller et al., 2026.

Let $X \in \mathbb{R}^{T\times K}$ contain $T$ calibration input rows and $W \in \mathbb{R}^{N\times K}$ be a linear weight. The dense output is $XW^\top$. MACaQ collects $H=X^\top X$, a positive diagonal pre-scale $D=\operatorname{diag}(p)$, and a block-Hadamard rotation $R$. In ideal arithmetic,

$$ Z=XDR,\qquad V=WD^{-1}R,\qquad ZV^\top=XW^\top, $$

where $R^\top R=I$. The implementation stores the transform in BF16, so this identity explains the design rather than promising exact finite-precision cancellation.

For column $j$, balancing uses

$$ p_j=\frac{\operatorname{std}(W_{:,j})^\alpha} {\left(\sqrt{H_{jj}/T}\right)^{1-\alpha}}. $$

The source clamps small denominators and limits $p_j$ to $[10^{-4},10^4]$. Weight standard deviation uses the unbiased estimate across output rows, except for a single output row: that case uses population standard deviation (zero), then the same positive $10^{-8}$ floor as zero-variance columns. This keeps narrow single-output projections finite. Non-finite weight statistics, activation RMS or resulting pre-scales raise ValueError instead of emitting NaN artifacts; empty or non-2-D weights are also rejected. Default $\alpha=0.75$ is a recipe choice, not a learned parameter. With transformed Hessian $G=R^\top DHD R$, the weight objective is

$$ \min_{Q\ \text{on the MXFP4 grid}}|Z(V-Q)^\top|_F^2 =\sum_{i=1}^{N}(V_i-Q_i)^\top G(V_i-Q_i). $$

Here $V_i,Q_i$ denote column-vector versions of weight row $i$. The practical solver approximates this discrete problem:

  1. Round each block of 32 values to signed E2M1 magnitudes ${0,0.5,1,1.5,2,3,4,6}$ with a shared power-of-two scale.
  2. Screen candidate scale exponents $\lfloor\log_2(\max|V_{\mathrm{block}}|)\rfloor-2+\mathrm{offset}$ using the diagonal of the metric.
  3. Freeze exponents and greedily move individual values to adjacent E2M1 levels. For a decoded-value change $c$ at coordinate $j$, objective change is $c^2G_{jj}-2c,g_j$, where $g=G(V_i-Q_i)$.

The refinement metric adds ridge * mean(diag(G)) to the diagonal, rather than an absolute ridge. W4A4 also rounds transformed activations at runtime, adding an error source beyond this weight objective. W4A16 keeps the packed weights and avoids that extra activation rounding. See the implementation and OCP MX specification.

Configuration reference

All fields below are declared in MacaqConfig; inherited controls are covered in QuantizeConfig Reference.

Field Default Contract
bits 4 Forced to 4
block_size, group_size 32, 32 block_size must be 32; group_size is set to it
desc_act, sym False, True Fixed by this format
had_block 32 Power of two, at least 32; input width must divide into these blocks
rotation "hadamard" Per-layer "hadamard" or "none"; legacy null normalizes to "hadamard"
do_flip True Enable greedy code-level refinement
n_flip_iters 256 Nonnegative maximum sweep count
fast_flip True Active-set sweep
compile_flip True Compiled sweep on eligible NVIDIA CUDA with the GIL enabled or ROCm gfx1201; eager on other targets
ridge 1.0 Nonnegative multiplier of the mean Hessian diagonal added to the flip metric
pre_mode "balance" "balance" or "none"
pre_alpha 0.75 D1 exponent in [0, 1]
exp_screen True Hessian-weighted exponent search
exp_offsets (1, 0, -1) Nonempty exponent-offset sequence; serialized as a list
activation_precision "w4a4" "w4a4" or "w4a16"

lm_head=True is rejected. dynamic supports exclusions and per-module overrides of had_block, rotation, do_flip, n_flip_iters, fast_flip, compile_flip, ridge, pre_mode, pre_alpha, exp_screen, exp_offsets, and activation_precision. Other dynamic keys raise; bits, symmetric packing, and MXFP4 scale-block size are not per-module knobs.

The Python option remains rotation, but checkpoint configuration serializes this per-layer choice as macaq_rotation and writes the shared model-wide rotation key as null. Loaders and the QuantizeConfig factory restore macaq_rotation to the MACaQ setting, including "none"; it must not request a second model-wide rotation.

Rotation-free weight-only MXFP4

from gptqmodel.quantization import MacaqConfig

config = MacaqConfig(
    rotation="none",
    pre_mode="none",
    activation_precision="w4a16",
)

This keeps calibrated Hessian-weighted exponent screening and optional flips while removing the activation rotation and D1 pass. It trades the default transform's accuracy benefit for less runtime work; evaluate that tradeoff on your model. rotation here selects a per-layer transform, not the model-wide rotation used by GPTQ. Keeping pre_mode="balance" with rotation="none" applies D1 only. W4A4 still quantizes activations and generally benefits more from rotation than W4A16.

Settings Saved format Transform buffers
rotation="none", pre_mode="none" FORMAT.MXFP4 / "mxfp4" Neither BH nor pre_scale
Any enabled transform FORMAT.MXFP4_MACAQ / "mxfp4_macaq" BH for Hadamard; pre_scale for balance

The config chooses mxfp4 only when the base settings and every non-excluded dynamic override disable both transforms; an explicit format does not override this decision. Both formats retain method="macaq". A bare format="mxfp4" without that method still resolves to GGUF.

On MLX, a transform-free W4A16 layer becomes stock mlx.nn.QuantizedLinear(mode="mxfp4", group_size=32, bits=4). Packed codes and E8M0 scales transfer without requantization. For a compatible model whose quantized layers all meet this contract, convert_gptq_to_mlx_weights(..., stock_compatible=True) followed by MLX-LM save helpers produces an artifact readable by stock mlx_lm.load. See checkpoint conversion; GPTQModel.export() has been removed. W4A4 or layers retaining transforms use the custom MlxMACaQLinear runtime for packed transfer. See the source guide.

Backends and packed storage

Backend Runtime and limits
MACAQ_TORCH / macaq_torch Portable Torch reference; FP16/BF16/FP32 activations, no training. Temporary reconstruction is limited to the current forward; no dense weight cache.
MACAQ_TRITON / macaq_triton NVIDIA CUDA or ROCm gfx1201 with Triton. W4A4 uses activation packing and tl.dot_scaled on packed E2M1 tiles. Other ROCm architectures are not qualified for this path.
MLX / mlx Apple Silicon/Metal packed transfer into stock QuantizedLinear for transform-free W4A16, otherwise MlxMACaQLinear; FP16/BF16 checkpoint holders, no adapters or training. See native MLX execution.

Quantization preflight resolves AUTO to Triton on eligible configured CUDA/ROCm targets, otherwise Torch; an explicit Torch request remains Torch. Triton validation checks the selected GPU ordinal. The Torch/Triton modules require input width divisible by both 32 and had_block, support any output width, and do not auto-pad. A nontransient Triton execution exception disables Triton for that module and uses its Torch forward. An out-of-memory exception permits a later retry; the current Torch fallback can also fail for lack of memory. Inspect warnings instead of assuming every call stayed native.

The Torch forward follows the current activation dtype (FP16/BF16/FP32), including after moving the module with .to(...); decoding and transform/bias operands no longer use a stale constructor dtype. Eligible Triton forwards also derive compute dtype from the current input: W4A16 passes it to the kernel helper, and rotation-free W4A4 rebuilds its identity BH when device or dtype changes. A constructor's compute_dtype is not a permanent forward-dtype override. The Torch final linear operation and Triton W4A16 helper use linear_fp32_reduce: NVIDIA low-precision products and bias stay FP32 until the output cast, while eligible CPU/ROCm calls fuse bias into the matmul epilogue. This protects cancellation before rounding without retaining decoded weights.

Checkpoint buffers are W_packed (uint8, [out_features, in_features / 2]), W_scales (E8M0 bytes, [out_features, in_features / 32]), optional bias, and only the enabled transform buffers: pre_scale for balance and BH for Hadamard. MLX repacks the same nibble codes to uint32 words without changing E8M0 bytes. The custom runtime skips absent transforms; for W4A4 it uses mx.qqmm through 16 flattened rows, while larger calls explicitly quantize/dequantize activations before mx.quantized_matmul. W4A16 uses packed-weight matmul directly. Torch W4A16 also skips disabled transform passes; Triton W4A4 uses an in-memory identity block when no BH is stored so its fused activation pack remains available. Runtimes restore the input activation dtype and retain only quantized weights between calls.

Scope and troubleshooting

  • A usable calibration Hessian is required for the calibrated solve. When the configured low-sample fallback threshold triggers, the port instead uses MXFP4 RTN with an identity metric, no D1 pre-scale, and no flips. Configured exponent screening remains active under that identity metric, choosing by plain squared error. Without a triggered fallback, an unusable Hessian raises. This recovery path does not make the normal workflow calibration-free; see the fallback implementation.
  • W4A16 avoids the additional runtime activation-quantization error; it does not change the saved 4-bit weight grid.
  • This port does not include the upstream MoE-specific kernels, W4A8 FP8 lane, cuDNN/CUTLASS backend, Diffusers/TTS pipelines, GGUF writer, or vLLM plugin. Do not infer those integrations from the method name.
  • Report output quality separately from speed. Repository measurements are model/device/recipe-specific, not guaranteed performance or quality for another workload.

🛠️ First things to check

If you see… Try this
ImportError for MacaqConfig from the package root Import from gptqmodel.quantization
Divisibility error Check each input width against 32 and had_block; choose compatible blocks or exclude the module
Unexpected quality loss Evaluate W4A16 with the same weights to separate weight error from activation error; revisit calibration coverage
Torch fallback warning Check the selected GPU ordinal, architecture and Triton availability
Low-sample fallback in logs Inspect which modules received calibration tokens before interpreting quality or timing

Next: validate on held-out data with Evaluation, then choose a matching Inference Backend.

Non-finite MXFP4 activation blocks are not converted into plausible finite codes: Torch packing and Triton activation quantization propagate NaN/Inf through the E8M0 NaN scale byte 0xFF. Torch/Triton dequantization treats that byte as NaN, not 2**128; the rotated activation path checks values after rotation too. Finite-block quantization and packed layout are unchanged. Activation and decode source · Torch packing source.

Source files

See also

Clone this wiki locally