Repository navigation
MACaQ
🏠 Home · 🧭 Choose a method · 🚀 Quickstart · 🧰 Help
On this page
MACaQ compresses weights and, optionally, activations to MXFP4 using calibration data to protect important signals. This guide is for model builders comparing four-bit floating-point W4A4 inference with a weight-only W4A16 path.
GPT-QModel includes a native port of Huawei CSL's MACaQ, a calibrated quantizer for low-precision weights and activations. This page describes source revision f9594695c.
| Choose this when… | Consider another path when… |
|---|---|
| You can calibrate a supported model and want MXFP4 weights plus four-bit activations | You need calibration-free packing: compare BitsAndBytes or FP8 weight-only |
You want to isolate activation rounding by comparing "w4a4" and "w4a16"
|
You need a GPTQ INT4 checkpoint/runtime: see GPTQ |
| Your runtime accepts native MACaQ state, or you want transform-free W4A16 for stock MLX | Your deployment consumes GGUF: choose its supported formats in GGUF |
Before you start: install GPT-QModel, select a supported model, and prepare a representative calibration corpus. Input widths must be divisible by both 32 and had_block; there is no automatic padding. The LM head is not quantized. Native Triton execution is declared for NVIDIA CUDA and AMD ROCm gfx1201; use the Torch reference on other supported targets. Backend details describe the remaining limits.
Read in three passes: use it → understand the solve → tune controls. Still choosing? Start with Choosing a Method and Dtype.
import torch
from gptqmodel import BACKEND, CalibrationConfig, GPTQModel
from gptqmodel.quantization import MacaqConfig
calibration_texts = [
"Quantization reduces model storage and inference bandwidth.",
"Use a representative calibration corpus for real workloads.",
]
model = GPTQModel.load(
"meta-llama/Llama-3.2-1B-Instruct",
quantize_config=MacaqConfig(
n_flip_iters=256,
had_block=32,
pre_alpha=0.75,
activation_precision="w4a4",
),
device="cuda:0",
dtype=torch.bfloat16,
)
model.quantize(CalibrationConfig(dataset=calibration_texts, batch_size=1), backend=BACKEND.AUTO)
model.save("./llama-macaq")
quantized = GPTQModel.load("./llama-macaq", backend=BACKEND.MACAQ_TRITON)API demo only: the two short strings show the call sequence, not an accuracy recipe. Use a representative corpus and evaluate the resulting model. Save uses safetensors. On a compatible Apple Silicon MLX model, reload with backend=BACKEND.MLX; install the MLX extra first.
| Choice | Value | Meaning |
|---|---|---|
| Method |
METHOD.MACAQ / "macaq"
|
Calibration-based MACaQ solve, not a GPTQ variant or calibration-free conversion |
| Config | MacaqConfig |
Import from gptqmodel.quantization
|
| Format |
FORMAT.MXFP4_MACAQ / "mxfp4_macaq" by default |
Packed MXFP4 weights plus enabled MACaQ transform state; "mxfp4" when both transforms are disabled |
| Default activation precision | "w4a4" |
MXFP4 weights and runtime MXFP4 activations |
| Alternative | "w4a16" |
Same 4-bit weights with non-MXFP4 activations |
The default per-Linear solve captures a calibration Hessian, computes an activation-aware diagonal pre-scale (D1), applies a block-Hadamard rotation, rounds weights to OCP MXFP4, screens E8M0 exponent offsets using the Hessian, and optionally refines E2M1 levels with greedy flips. D1 and rotation can be disabled independently. Calibration uses a combined block subset; packed replay with the selected activation precision supplies inputs to later blocks. MACaQ shares calibration infrastructure with GPTQ, but uses a different grid, solve, checkpoint, and runtime.
Supported quantization continuation runs record the packed module's actual compute_dtype so BH and pre_scale restore in the saved dtype even when the source model config declares another dtype. This prevents a resume schema mismatch without changing MACaQ's inference checkpoint format or relaxing continuation admission rules.
The solver now drops unused pre-rotation weight/Hessian references after rotation and releases the rotated Hessian after constructing the flip metric. This reduces temporary memory overlap before exponent screening and the flip sweep, especially when several modules quantize concurrently. Packed codes and E8M0 exponents are unchanged, and no new option is required. See buffer-lifetime regression coverage.
Exponent screening accumulates E2M1 codes in UINT8 and widens signed-grid indices to INT32 for gathering, avoiding full INT64 planes while preserving the weight and activation tie rules. Compiled flip sweeps use fullgraph=True and keep converged iterations in the graph with masked no-op updates, avoiding a data-dependent early return whose Dynamo resume frames could retain the Hessian working set. A future graph break raises rather than silently retaining those frames. With that cycle removed, the flip path no longer forces a full gc.collect() after each Linear; processor finalization retains one collection as the end-of-run cleanup backstop. These are automatic memory/performance changes; compile_flip and the packed format retain their existing contracts. See flip-sweep implementation.
OCP MXFP4 uses E2M1 values and one E8M0 scale byte per 32 values: nominally 4.25 bits/weight, before enabled pre-scale, Hadamard, bias, and other model state. mxfp4_macaq is a checkpoint contract, not a new number system. Generic MXFP4 or GGUF MXFP4 checkpoints are not interchangeable with it: the tensor layout and enabled pre_scale/BH state must also match. See rotation-free weight-only MXFP4 for the transform-free format.
The intuition is to balance large channels, spread outliers with a rotation, then spend rounding effort where calibration says an error matters most. The recipe comes from Muller et al., 2026.
Let
where
For column
The source clamps small denominators and limits ValueError instead of emitting NaN artifacts; empty or non-2-D weights are also rejected. Default
Here
- Round each block of 32 values to signed E2M1 magnitudes
${0,0.5,1,1.5,2,3,4,6}$ with a shared power-of-two scale. - Screen candidate scale exponents
$\lfloor\log_2(\max|V_{\mathrm{block}}|)\rfloor-2+\mathrm{offset}$ using the diagonal of the metric. - Freeze exponents and greedily move individual values to adjacent E2M1 levels. For a decoded-value change
$c$ at coordinate$j$ , objective change is$c^2G_{jj}-2c,g_j$ , where$g=G(V_i-Q_i)$ .
The refinement metric adds ridge * mean(diag(G)) to the diagonal, rather than an absolute ridge. W4A4 also rounds transformed activations at runtime, adding an error source beyond this weight objective. W4A16 keeps the packed weights and avoids that extra activation rounding. See the implementation and OCP MX specification.
All fields below are declared in MacaqConfig; inherited controls are covered in QuantizeConfig Reference.
| Field | Default | Contract |
|---|---|---|
bits |
4 |
Forced to 4 |
block_size, group_size
|
32, 32
|
block_size must be 32; group_size is set to it |
desc_act, sym
|
False, True
|
Fixed by this format |
had_block |
32 |
Power of two, at least 32; input width must divide into these blocks |
rotation |
"hadamard" |
Per-layer "hadamard" or "none"; legacy null normalizes to "hadamard"
|
do_flip |
True |
Enable greedy code-level refinement |
n_flip_iters |
256 |
Nonnegative maximum sweep count |
fast_flip |
True |
Active-set sweep |
compile_flip |
True |
Compiled sweep on eligible NVIDIA CUDA with the GIL enabled or ROCm gfx1201; eager on other targets |
ridge |
1.0 |
Nonnegative multiplier of the mean Hessian diagonal added to the flip metric |
pre_mode |
"balance" |
"balance" or "none"
|
pre_alpha |
0.75 |
D1 exponent in [0, 1]
|
exp_screen |
True |
Hessian-weighted exponent search |
exp_offsets |
(1, 0, -1) |
Nonempty exponent-offset sequence; serialized as a list |
activation_precision |
"w4a4" |
"w4a4" or "w4a16"
|
lm_head=True is rejected. dynamic supports exclusions and per-module overrides of had_block, rotation, do_flip, n_flip_iters, fast_flip, compile_flip, ridge, pre_mode, pre_alpha, exp_screen, exp_offsets, and activation_precision. Other dynamic keys raise; bits, symmetric packing, and MXFP4 scale-block size are not per-module knobs.
The Python option remains rotation, but checkpoint configuration serializes this per-layer choice as macaq_rotation and writes the shared model-wide rotation key as null. Loaders and the QuantizeConfig factory restore macaq_rotation to the MACaQ setting, including "none"; it must not request a second model-wide rotation.
from gptqmodel.quantization import MacaqConfig
config = MacaqConfig(
rotation="none",
pre_mode="none",
activation_precision="w4a16",
)This keeps calibrated Hessian-weighted exponent screening and optional flips while removing the activation rotation and D1 pass. It trades the default transform's accuracy benefit for less runtime work; evaluate that tradeoff on your model. rotation here selects a per-layer transform, not the model-wide rotation used by GPTQ. Keeping pre_mode="balance" with rotation="none" applies D1 only. W4A4 still quantizes activations and generally benefits more from rotation than W4A16.
| Settings | Saved format | Transform buffers |
|---|---|---|
rotation="none", pre_mode="none"
|
FORMAT.MXFP4 / "mxfp4"
|
Neither BH nor pre_scale
|
| Any enabled transform |
FORMAT.MXFP4_MACAQ / "mxfp4_macaq"
|
BH for Hadamard; pre_scale for balance |
The config chooses mxfp4 only when the base settings and every non-excluded dynamic override disable both transforms; an explicit format does not override this decision. Both formats retain method="macaq". A bare format="mxfp4" without that method still resolves to GGUF.
On MLX, a transform-free W4A16 layer becomes stock mlx.nn.QuantizedLinear(mode="mxfp4", group_size=32, bits=4). Packed codes and E8M0 scales transfer without requantization. For a compatible model whose quantized layers all meet this contract, convert_gptq_to_mlx_weights(..., stock_compatible=True) followed by MLX-LM save helpers produces an artifact readable by stock mlx_lm.load. See checkpoint conversion; GPTQModel.export() has been removed. W4A4 or layers retaining transforms use the custom MlxMACaQLinear runtime for packed transfer. See the source guide.
| Backend | Runtime and limits |
|---|---|
MACAQ_TORCH / macaq_torch
|
Portable Torch reference; FP16/BF16/FP32 activations, no training. Temporary reconstruction is limited to the current forward; no dense weight cache. |
MACAQ_TRITON / macaq_triton
|
NVIDIA CUDA or ROCm gfx1201 with Triton. W4A4 uses activation packing and tl.dot_scaled on packed E2M1 tiles. Other ROCm architectures are not qualified for this path. |
MLX / mlx
|
Apple Silicon/Metal packed transfer into stock QuantizedLinear for transform-free W4A16, otherwise MlxMACaQLinear; FP16/BF16 checkpoint holders, no adapters or training. See native MLX execution. |
Quantization preflight resolves AUTO to Triton on eligible configured CUDA/ROCm targets, otherwise Torch; an explicit Torch request remains Torch. Triton validation checks the selected GPU ordinal. The Torch/Triton modules require input width divisible by both 32 and had_block, support any output width, and do not auto-pad. A nontransient Triton execution exception disables Triton for that module and uses its Torch forward. An out-of-memory exception permits a later retry; the current Torch fallback can also fail for lack of memory. Inspect warnings instead of assuming every call stayed native.
The Torch forward follows the current activation dtype (FP16/BF16/FP32), including after moving the module with .to(...); decoding and transform/bias operands no longer use a stale constructor dtype. Eligible Triton forwards also derive compute dtype from the current input: W4A16 passes it to the kernel helper, and rotation-free W4A4 rebuilds its identity BH when device or dtype changes. A constructor's compute_dtype is not a permanent forward-dtype override. The Torch final linear operation and Triton W4A16 helper use linear_fp32_reduce: NVIDIA low-precision products and bias stay FP32 until the output cast, while eligible CPU/ROCm calls fuse bias into the matmul epilogue. This protects cancellation before rounding without retaining decoded weights.
Checkpoint buffers are W_packed (uint8, [out_features, in_features / 2]), W_scales (E8M0 bytes, [out_features, in_features / 32]), optional bias, and only the enabled transform buffers: pre_scale for balance and BH for Hadamard. MLX repacks the same nibble codes to uint32 words without changing E8M0 bytes. The custom runtime skips absent transforms; for W4A4 it uses mx.qqmm through 16 flattened rows, while larger calls explicitly quantize/dequantize activations before mx.quantized_matmul. W4A16 uses packed-weight matmul directly. Torch W4A16 also skips disabled transform passes; Triton W4A4 uses an in-memory identity block when no BH is stored so its fused activation pack remains available. Runtimes restore the input activation dtype and retain only quantized weights between calls.
- A usable calibration Hessian is required for the calibrated solve. When the configured low-sample
fallbackthreshold triggers, the port instead uses MXFP4 RTN with an identity metric, no D1 pre-scale, and no flips. Configured exponent screening remains active under that identity metric, choosing by plain squared error. Without a triggered fallback, an unusable Hessian raises. This recovery path does not make the normal workflow calibration-free; see the fallback implementation. - W4A16 avoids the additional runtime activation-quantization error; it does not change the saved 4-bit weight grid.
- This port does not include the upstream MoE-specific kernels, W4A8 FP8 lane, cuDNN/CUTLASS backend, Diffusers/TTS pipelines, GGUF writer, or vLLM plugin. Do not infer those integrations from the method name.
- Report output quality separately from speed. Repository measurements are model/device/recipe-specific, not guaranteed performance or quality for another workload.
| If you see… | Try this |
|---|---|
ImportError for MacaqConfig from the package root |
Import from gptqmodel.quantization
|
| Divisibility error | Check each input width against 32 and had_block; choose compatible blocks or exclude the module |
| Unexpected quality loss | Evaluate W4A16 with the same weights to separate weight error from activation error; revisit calibration coverage |
| Torch fallback warning | Check the selected GPU ordinal, architecture and Triton availability |
| Low-sample fallback in logs | Inspect which modules received calibration tokens before interpreting quality or timing |
Next: validate on held-out data with Evaluation, then choose a matching Inference Backend.
Non-finite MXFP4 activation blocks are not converted into plausible finite codes: Torch packing and Triton activation quantization propagate NaN/Inf through the E8M0 NaN scale byte 0xFF. Torch/Triton dequantization treats that byte as NaN, not 2**128; the rotated activation path checks values after rotation too. Finite-block quantization and packed layout are unchanged. Activation and decode source · Torch packing source.
Documentation home · Quickstart · Troubleshooting · Repository · PyPI
Version scope: public source 54888582e (package version 7.12.0); older pinned evidence is identified on individual pages. Choose a method · Dtypes and formats. Maintained by ModelCloud.ai.
🧭 1 · Choose and understand
🚀 2 · Use it
- Installation and Quickstart
- Loading and Inference
- Evaluation
- Serving
- Troubleshooting and FAQ
- Memory, Offload, Multi-GPU
- MLX / Apple Silicon
- Tenstorrent / Block Floats
- Hugging Face Integration
- Multimodal and MoE Models
🎛️ Methods and representations
Each guide: choose → examples → math / advanced
- GPTQ / GPTAQ / RTN
- AWQ
- QQQ
- MACaQ / MXFP4
- EXL3
- VPTQ
- Paro
- FP8 and NVFP4
- NF12: lossless BF16 storage
- GGUF families
- BitsAndBytes: NF4 / FP4 / INT8
🧰 Refinement and execution
🧠 3 · Theory and advanced reference