Skip to content

Rotation Smoothing and Norm Tuning

Jump to bottom
Qubitium edited this page Oct 29, 2026 · 5 revisions

Rotations, Smoothing, and Norm Tuning

Documentation home · Quickstart · Lifecycle

On this page

Overview

GPT-QModel provides several independent ways to change the numerical problem around quantization. They operate at different points in the lifecycle and are not interchangeable:

Technique When it runs What changes Main entry point
Orthogonal rotation Before weight quantization Floating-point weights, fused normalization gains, and the activation basis QuantizeConfig(rotation="hadamard") or rotation="random"
Weight-block smoothing Weight-only quantization or a supported fallback/packing path Weight blocks used to fit quantization parameters; some methods also return a compensating scale SmootherConfig(smooth=SmoothMAD(...)), or Fallback(smooth=...)
Group Aware Reordering (GAR) GPTQ Hessian-based weight solve Internal processing order of input columns and groups GPTQConfig(act_group_aware=True)
FOEM GPTQ-style weight solve Error-feedback update, optionally with GPTAQ native-input correction GPTQConfig(foem=FOEMConfig(...))
Floating activation policies Calibration/replay and supported inference Activation encoding and scale policy; weights remain GPTQ INT4 activation="w4afp8" or activation={"mode": "w4a_nvfp4", ...}
Producer activation calibration After loading an NVFP4 stream Persistent FP32 producer global scales, not weights model.calibrate_activations(samples)
Norm-only distillation After quantization and reload Selected normalization gains and biases, not packed quantized tensors model.requantize(calibration, norm=NormConfig(...))

Terminology: the Smooth* classes in this page implement weight-block clipping, normalization, or quantization-range fitting. They are not a calibration-driven SmoothQuant implementation that transfers per-channel activation magnitude into adjacent weights. There is no SmoothQuant alpha in SmootherConfig. FOEM's alpha has a different meaning: the strength of the GPTAQ correction.

This reference follows source revision c1976918cfe3703fbcf6d1ce2440a82b26394375 (v7.7.0), including gptqmodel/quantization/ and its callers. Backend and hardware restrictions apply independently of whether a configuration object can be constructed. In particular, portable mathematical helpers do not imply that the complete rotation or W4A inference lifecycle is portable to every accelerator.

Workflow at a glance

Choose where to intervene in quantization. Basis changes, weight-range fitting and norm training are independent options.

Reading the diagram: These lanes are independent choices, not a mandatory stack. Shared model rotation is not supported for QQQ/EXL3 checkpoints. Smoothing is not activation-based SmoothQuant.

Text version of the workflow
  • Rotation before GPTQ: Check rotation support → Fuse norms and rotate → Quantize and replay → Reload with rotation.
  • Optional weight smoothing: Choose a supported path → Apply the weight policy → Quantize weight blocks.
  • Post-load norm tuning: Reload the saved student → Select tagged norms → Distill teacher logits → Save norm replacements.

Open full-size SVG

Concepts and lifecycle

Choose the right intervention

  1. Change the basis before quantization: use rotation for a supported Llama/Qwen2 model and a rotation-aware runtime. Normalization gains are folded into weights; MLP down-projection inputs need an online Hadamard transform afterwards.
  2. Adjust weight ranges without activations: use an explicit weight-only config such as RTNConfig, with an optional smoother. Smoothing is opt-in. For sparse calibration coverage, configure the separate GPTQ/QQQ fallback smoother.
  3. Improve the Hessian solve: GAR changes the processing order while preserving group membership. FOEM changes error propagation. FOEM has its own solver; do not assume it inherits every ordinary GPTQ behavior merely because it subclasses GPTQ.
  4. Fit activation representation: W4A policies and producer calibration control activation rounding and transport. They do not turn an INT4 checkpoint into an FP4-weight checkpoint.
  5. Refine a finished checkpoint: norm tuning trains only explicitly eligible normalization parameters against an unquantized teacher. Save and reload freshly quantized models before tuning.

These options need evaluation on the intended model, calibration corpus, and serving backend. The source does not promise that enabling more techniques always improves accuracy. Keep calibration and held-out evaluation data separate; producer calibration explicitly places responsibility for that separation on the caller.

Rotations

Configuration and supported runtime paths

The shared configuration field is:

rotation: Optional[str] = field(default=None, metadata={"choices": ["hadamard", "random"]})
Value Meaning
None Do not rotate the model.
"hadamard" Build a randomized signed Hadamard orthogonal matrix. This is randomized, not simply a fixed unsigned Walsh matrix.
"random" Draw an FP64 Gaussian matrix, use QR factorization, and adjust signs using the diagonal of R.

BaseQModel.quantize() restricts this path to instances of LlamaQModel and Qwen2QModel. It directly accesses Llama-style embedding, attention, MLP, and normalization attributes; a model having a vaguely similar architecture is not enough. get_orthogonal_matrix() rejects an unknown mode. The loader also checks that a saved rotation value is "hadamard" or "random".

Constraint Implemented behavior
Quantization backend GPTQ_TORCH, GPTQ_TRITON, GPTQ_W4AFP8, or GPTQ_W4A_NVFP4; AUTO is resolved to GPTQ_TORCH in the rotation branch. Other backends raise NotImplementedError.
Saved checkpoint The loader requires FORMAT.GPTQ or FORMAT.GPTQ_V2 plus one of the supported rotation backends. The shared config field does not make rotation a supported QQQ or EXL3 round trip. Leave it unset for those formats.
Tied embeddings If model.config.tie_word_embeddings is true, it is set to False and the LM-head weight is cloned before rotation. Already untied models do not take this cloning branch.
Disk offload A truthy rotation disables offload_to_disk during config initialization, with a warning; rotation needs materialized weights.
FP64 work Basis rotations use FP64 intermediates and cast back to the original weight dtype; several rotated tensors are placed on CPU.
MPS The caller substitutes CPU for the FP64 rotation device, but the exact linear Hadamard helper still calls .cuda(). This is not an end-to-end MPS/CPU rotation guarantee.
CUDA/ROCm apply_exact_had_to_linear() explicitly uses Torch's CUDA device API. On ROCm this is the HIP-backed Torch API, but native extension availability must be checked separately. No blanket ROCm compatibility guarantee follows from the config field.

Example for a supported source model, with a prepared calibration dataset:

from gptqmodel import BACKEND, GPTQConfig, GPTQModel

config = GPTQConfig(bits=4, group_size=128, rotation="hadamard")
model = GPTQModel.load("path/to/unquantized-llama", quantize_config=config)
model.quantize(calibration, backend=BACKEND.GPTQ_TORCH)
model.save("llama-int4-rotated")

# Use a runtime that applies the saved rotation policy.
reloaded = GPTQModel.load("llama-int4-rotated", backend=BACKEND.GPTQ_TORCH)

Do not add the same Hadamard transform externally to a runtime that already applies it. Conversely, exporting rotated weights to a runtime that ignores rotation metadata can produce incorrect inference.

Weight transformations

The core signature is copied from rotation/rotation.py:

@torch.inference_mode()
def rotate_model(model: PreTrainedModel, rotate_mode: str, device: torch.device, lm_head_name: str,
                 layers_node: str, Q=None):

It returns (model, Q). Supplying Q bypasses matrix generation; this is an internal API, not a separate QuantizeConfig field.

Before this call, fuse_layer_norms() multiplies norm gains into adjacent input projections and resets the norm weights to ones. Where a norm has bias, fuse_ln_linear() also folds the bias into the adjacent linear's bias. The implementation targets the named Llama-style norms and projections, rather than arbitrary normalization graphs.

For a linear weight stored as (out_features, in_features), the rotation sequence is:

Module Main transform
model.embed_tokens Right-multiply embedding weights by Q.
LM head Right-multiply by Q.
self_attn.q_proj, k_proj, v_proj Right-multiply by Q.
self_attn.o_proj Left-multiply by Q.T; transform output bias by Q.T if present.
mlp.up_proj, gate_proj Right-multiply by Q.
mlp.down_proj Left-multiply by Q.T, then apply an exact input-side Hadamard; transform output bias by Q.T if present.
v_proj output and o_proj input Additional matching per-head Hadamard transforms, using hidden_size // num_attention_heads.

The per-head value/output transforms cancel through the attention path. The MLP has an elementwise SiLU * up activation between projections, so its down projection instead needs the matching transform on the live activation. rotate_mlp_output() sets online_full_had=True, online_partial_had=False, had_dim=-1, had_K, and K on that projection. Calibration/replay and packed inference must agree about this transform.

The loader's _setup_rotation_online_had() finds BaseQuantLinear modules whose names end with mlp.down_proj, reconstructs had_K/K from in_features, and enables the full online transform. It warns when no such quantized module is found. The rotation mode is persisted; those Hadamard matrices are recomputed, not separately required as checkpoint tensors.

Hadamard dimensions and implementation selection

rotation/hadamard_utils.py contains both mathematical helpers and fixed small Hadamard matrices. get_hadK(n, transpose=False) selects, in order, factors 172, 156, 140, 108, 60, 52, 36, 28, 40, 20, and 12 when divisible; it asserts that the quotient is a power of two. Otherwise, n itself must be a power of two and K=1. Because this is ordered branching with assertions, do not treat it as a search over every possible factorization of an arbitrary width.

Helper Contract
matmul_hadU(X, transpose=False) Portable normalized Hadamard multiplication over the last dimension, including the selected small K matrix.
matmul_hadUt(X) Calls the transposed version.
random_hadamard_matrix(size, device) Applies a normalized Hadamard to a random diagonal of FP64 signs and transfers the result to device.
matmul_hadU_cuda(X, hadK, K) Uses the selected fast/fallback Walsh transform plus the small matrix when K > 1; the name does not mean every selected implementation is CUDA-only.
apply_online_hadamard(x, online_full_had, online_partial_had, had_K, K, had_dim=-1) Returns x when both flags are false, or when had_K is absent and K != 1; otherwise applies a full last-axis transform or reshaped partial transform.
apply_exact_had_to_linear(module, had_dim=-1, output=False) Mutates an nn.Linear weight and, for output-side transforms, its bias. Explicit had_dim must be a power of two; internally moves work to CUDA in FP32 and restores device/dtype.

Backend selection in import_fast_hadamard_transform() is lazy and cached:

  1. Installed fast_hadamard_transform Python package.
  2. GPT-QModel's vendored JIT Hadamard extension, when available.
  3. TorchAO's Triton Hadamard helper, when importable.
  4. Portable Torch butterfly implementation, with a warning.

The Torch fallback requires a power-of-two transformed width. The TorchAO wrapper uses CUDA FP16/BF16 inputs, power-of-two widths up to 256 ** 2, and at most two normalized tiled passes; other inputs fall back to Torch. _HadamardAutograd preserves the chosen forward and supplies a portable FP64 butterfly backward with one final cast when gradients are needed. This matters when an online rotation is used in a gradient-based workflow; the upstream operator need not provide its own autograd formula.

The lower-level vendored API in gptqmodel/utils/hadamard.py is:

def hadamard_transform(x: torch.Tensor, scale: Optional[float] = None) -> torch.Tensor:

This direct wrapper is CUDA-only, accepts FP16/BF16/FP32 according to its documented contract, transforms the last dimension, and defaults scale to 1.0. That default is unnormalized. The rotation helpers explicitly supply normalization such as 1 / sqrt(n).

Extension control/helper Meaning
hadamard_supported() Checks torch.cuda.is_available(), not successful compilation.
hadamard_available() Checks the extension API's availability after the CUDA guard.
hadamard_runtime_error() Returns the CUDA requirement message or the extension's last error.
hadamard_op(op_name="fast_hadamard_transform") Resolves an extension operator.
GPTQMODEL_HADAMARD_BUILD_ROOT Overrides the JIT build root.
GPTQMODEL_HADAMARD_FORCE_REBUILD Forces extension rebuilding.
GPTQMODEL_EXT_VERBOSE Extension build verbosity.

The extension sources are gptqmodel_ext/hadamard/hadamard.cpp and fast_hadamard_transform_cuda.cu; registered required operations include the ordinary transform and 12N, 20N, 28N, and 40N variants. A working CUDA toolchain is needed when building the JIT path. An unavailable native extension can select a slower fallback; it does not remove .cuda() from the weight-rotation helper.

Weight smoothing

Public configuration surface

Import smoother descriptors from gptqmodel.quantization:

from gptqmodel import RTNConfig
from gptqmodel.quantization import SmoothMAD, SmootherConfig

config = RTNConfig(
    bits=4,
    group_size=128,
    smoother=SmootherConfig(smooth=SmoothMAD(k=2.75)),
)

SmootherConfig is a BasePreProcessorConfig with class code "smoother" and one field, smooth: Optional[SmoothMethod] = None. Its to_dict() produces {"code": "smoother", "smooth": ...}. A bare SmootherConfig() is an inactive descriptor, not a default smoothing algorithm.

PreProcessorConfig accepts these fields:

Field Default Accepted form and behavior
preprocessors New empty list List of typed preprocessor configs, smoother descriptors, dictionaries, or strings; None becomes an empty list.
smoother None SmootherConfig, SmoothMethod, dictionary, or method-name string. Preferred direct smoother field.
smooth None Backward-compatible alias; normalized to the active SmoothMethod. Use smoother in new code.

Normalization prefers an explicit non-None smoother over smooth. If neither supplies one, the first SmootherConfig in preprocessors is selected. All smoother entries are removed from their original positions, and the selected smoother is appended after non-smoother preprocessors. Duplicate remaining preprocessor codes are rejected. Consequently, the normalized order can differ from the order written in the constructor, and multiple smoother entries do not compose a chain of clipping operations.

Equivalent typed/dictionary forms:

from gptqmodel import RTNConfig
from gptqmodel.quantization import SmoothPercentile, SmootherConfig

typed = RTNConfig(
    smoother=SmootherConfig(smooth=SmoothPercentile(percentile=99.0)),
)
dictionary = RTNConfig(
    preprocessors=[{
        "code": "smoother",
        "smooth": {"type": "percentile", "percentile": 99.0,
                   "group_size_threshold": 128},
    }],
)
shorthand = RTNConfig(smoother="percentile")

Every smoother and its defaults

All constructors below also accept group_size_threshold: int = 128. The base descriptor is SmoothMethod(name: str, group_size_threshold: int = 128).

Class Canonical serialized type Constructor-specific arguments and defaults Implemented operation on a (rows, group_columns) block
SmoothPercentile "percentile" percentile: float = 99.0 Find a per-row percentile of absolute weights; clip symmetrically to its positive/negative threshold.
SmoothPercentileAsymmetric "percentile_asym" low: float = 0.5, high: float = 99.5 Find separate per-row quantiles of raw signed weights and clamp to those bounds. Parser also accepts "percentile_asymmetric".
SmoothMAD "mad" k: float = 2.75 Clip around each row's median, with radius k * median(abs(w - median(w))) * 1.4826. The factor converts raw Gaussian MAD to a sigma-like width.
SmoothMSE "mse" steps: int = 32, maxshrink: float = 0.8 In the dedicated RTN/fallback branch, choose a per-row range multiplier by bounded ternary search and refinement, then quantize/dequantize. Not an operation implemented inside smooth_block().
SmoothOutlier "outlier" pct: float = 1.0 Per row, clip at the k-th smallest absolute weight, where k = round(width * (1 - pct/100)), clamped to [1, width]. pct <= 0 is a no-op.
SmoothSoftNorm "softnorm" k: float = 3.0 Center each row, divide by centered RMS, clamp normalized values to [-k, k], then undo centering/scaling. This clips outliers; it does not persist standardized weights.
SmoothLog "log" percentile: float = 99.0, mu: float = 8.0 Transform magnitudes with log1p(mu * abs(w)) / log1p(mu), compute the percentile, invert its threshold, and clip original weights symmetrically. mu is floored at eps.
SmoothRowCol "rowcol" axis: str = "row" Return normalized weights and a scale factor that the caller must reapply after quantization. Row mode uses per-row RMS; "col" uses the mean of per-column RMS values, yielding a single scalar factor, not one independent factor per column.

Important default discrepancy: SmoothMAD() has k=2.75, but _build_smooth_method_from_dict() defaults a missing MAD k to 3.0. Thus smoother="mad" and smoother={"type": "mad"} are not numerically identical to smoother=SmoothMAD(). Specify k explicitly for reproducibility.

Parser details:

  • A string is treated as a method name; dictionary type takes precedence over name.
  • "none" resolves to no method. A dictionary with neither type nor name also resolves to no method.
  • Unknown names raise ValueError; the method parser strips whitespace and lowercases names.
  • A missing or None dictionary group_size_threshold becomes 128.
  • The descriptor classes do not perform comprehensive numerical range validation. Use meaningful percentiles, positive widths, and ordered asymmetric bounds rather than expecting constructor errors for every invalid setting.
  • Quantiles are implemented with kthvalue and linear interpolation. Percentiles at or below zero return the minimum; at or above 100 return the maximum.

fallback_smooth.py: numerical contract

The helper signatures are:

def smooth_block(
    block: torch.Tensor,
    fallback: Fallback,
    *,
    eps: float = 1e-8,
    group_size: Optional[int] = None,
) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:

def mse_optimal_quant(
    block: torch.Tensor,
    qcfg: QuantizeConfig,
    maxq: int,
    *,
    steps: int,
    maxshrink: float,
    eps: float = 1e-8,
) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:

smooth_block() reads fallback.smooth. Without a method, it returns (block, None). A supplied negative group_size is replaced with the actual block width. If the supplied effective size is below the method's threshold, the function returns the original block. With group_size=None, this threshold check is not performed. Callers commonly pass the configured effective group size, which need not equal the width of a short final group.

Most clipping calculations are in FP32 and cast back to block.dtype. Row/column normalization follows the incoming block's arithmetic and returns (block / scale_factor, scale_factor). The caller must multiply both reconstructed weights and quantizer scales by that factor. axis="col" selects the scalar column-RMS-average behavior; all other axis values take the row branch after lowercasing.

mse_optimal_quant() returns (dequantized_block, scale, zero). It uses qcfg.sym, an integer maxq, and FP32 work:

  • Symmetric fitting uses the row maximum absolute value, its negative bound, and zero point (maxq + 1) / 2.
  • Asymmetric fitting starts with each row's minimum and maximum and derives a rounded zero point.
  • maxshrink is clamped to [1e-3, 1]; the range multiplier lies between that value and 1.
  • The requested steps is converted to max(int(log(max(steps, 2)) / log(1.5)) + 1, 3) ternary-search iterations, followed by three nearby refinement candidates. It is not a literal steps-point exhaustive grid, despite the descriptor's older grid-search wording.
  • Error is mean squared reconstruction error per row. Scale is floored by eps.

SmoothMSE is recognized separately by RTN.quantize() and the GPTQ/QQQ fallback implementations, before they call smooth_block(). Therefore its group_size_threshold is not enforced by those dedicated MSE branches. Passing a SmoothMSE descriptor to a path that only calls smooth_block() does not invoke this MSE search.

Do not confuse SmoothMSE with MSEConfig: the latter controls the ordinary quantizer's shrink objective/workspace, defaults to norm=0.0 (disabled), and is a separate qcfg.mse field.

Where smoothing actually runs

Path Smoother source Behavior/limitation
Native whole-model RTN RTNConfig.smoother, normalized into qcfg.smooth Quantizes weights without entering the activation/Hessian lifecycle; clipping or row/column rescaling is applied group by group, with a separate MSE branch.
GPTQ low-sample fallback qcfg.fallback.smooth _fallback_quantize() uses the fallback smoother. A top-level smoother is not a substitute for fallback.smooth in this branch.
QQQ fallback Fallback smoother Has the corresponding smooth_block()/dedicated MSE behavior.
GGUF/FP8/BitsAndBytes direct packing Resolved weight-only smoother All three call the shared _apply_optional_smoother() defined in gguf.py. This helper does not implement SmoothMSE and rejects any returned post-quantization scale factor, so active SmoothRowCol is unsupported on all three direct-packing paths.
Ordinary Hessian GPTQ solve Not a generic smoothing execution stage ModulePreProcessor records a plan; it does not itself run clipping on every module. Do not infer a SmoothQuant pass or universal smoothing from the presence of preprocessors.

Native MLX RTN dispatch additionally requires qcfg.smooth is None. Configuring a smoother selects the Torch RTN path rather than silently omitting smoothing on MLX.

Direct packers use their own smoothing block width, not necessarily the top-level config's group_size: GGUF uses its tensor quantization block size, FP8 uses the column dimension of its block shape for block scaling (otherwise the full row), and BitsAndBytes uses bnb_block_size. The default smoother threshold of 128 can therefore disable clipping for smaller packing blocks. Set group_size_threshold explicitly when clipping those blocks is intended. The shared helper checks this threshold before it can reject a rescaling smoother; an inactive SmoothRowCol may appear to succeed without doing anything.

Fallback defaults are strategy=FallbackStrategy.RTN, threshold="0.5%", and smooth=None. Its enum values are "rtn", "midpoint", "mean", "median", and "stdclip". A percentage threshold expresses weak coverage relative to calibration tokens; the fallback is intended for cases such as under-routed MoE experts. Dictionary normalization has a different missing-threshold default ("1.0%"), so specify the threshold when exact reproducibility matters.

from gptqmodel import GPTQConfig
from gptqmodel.quantization import Fallback, FallbackStrategy, SmoothMAD

config = GPTQConfig(
    bits=4,
    group_size=128,
    fallback=Fallback(
        strategy=FallbackStrategy.RTN,
        threshold="0.5%",
        smooth=SmoothMAD(k=2.75, group_size_threshold=128),
    ),
)

The compatibility WeightOnlyConfig defaults to method=WeightOnlyMethod.RTN, smooth=None. Its method values are "rtn", "gguf", "fp8", "bitsandbytes", and "nvfp4". New usage should prefer the corresponding explicit method-specific configuration. The weight-only per-module config clone accepts dynamic smoother overrides, with legacy smooth as a fallback key.

Preprocessor pipeline

PreProcessorCode is a string enum with exactly these members:

Member Value Typed configuration and defaults
SMOOTHER "smoother" SmootherConfig(smooth=None)
AUTO_MODULE_DECODER "auto_module_decoder" AutoModuleDecoderConfig(source_dtype="auto", target_dtype=torch.bfloat16, passthrough_forward_policy="native", passthrough_save_policy="decode")
TENSOR_PARALLEL_PADDER "tensor_parallel_padder" TensorParallelPadderConfig(); padding is derived, not a user-supplied column count.

ModulePreProcessor is a no-forward planning processor: ExecutionConfig(require_fwd=False, fwd_replay_after_process=False). It writes ordered preprocessor_pipeline descriptors into each NamedModule.state and specialized auto_module_decoder/tp_pad_info state for downstream consumers. Its process() is intentionally side-effect free; it does not require calibration inputs. If a feature is no longer configured, it removes stale associated state.

import torch
from gptqmodel import RTNConfig
from gptqmodel.quantization import (
    AutoModuleDecoderConfig,
    SmoothPercentile,
    SmootherConfig,
    TensorParallelPadderConfig,
)

config = RTNConfig(
    bits=4,
    group_size=128,
    preprocessors=[
        AutoModuleDecoderConfig(target_dtype=torch.bfloat16),
        TensorParallelPadderConfig(),
        SmootherConfig(smooth=SmoothPercentile(percentile=99.0)),
    ],
)

Decoder requirements and planning behavior:

  • source_dtype is currently restricted to "auto".
  • The target must normalize to FP16 or BF16. FP32 is not an accepted decoder target here.
  • Forward policy is "native" or "decode"; save policy must be "decode".
  • The plan labels quantization source decoding "defer_until_quantization". Native source forwarding is a pre-quantization optimization, not permission to keep a floatx source wrapper in a saved standard dense/W4 artifact.
  • Use typed AutoModuleDecoderConfig when specifying policies: the dictionary-to-preprocessor normalizer constructs this config from source_dtype and target_dtype, rather than forwarding every policy key from that dictionary.

The padder computes target_multiple = lcm(2, 4, 8) and includes a positive group_size in that LCM. Padding is (target_multiple - columns % target_multiple) % target_multiple. It stores metadata only when padding is nonzero. transformers.Conv1D uses its transposed storage convention to determine columns; other weights use the first dimension for rows and the product of remaining dimensions for columns. Padding metadata is not a claim that every serving engine accepts every resulting shape.

Floatx source decoding and dtype support

gptqmodel/quantization/dtype.py is the shared compatibility/decode layer used around floatx source modules. Its role is distinct from activation calibration:

API Purpose and important defaults
available_float8_dtype_names() / available_float8_dtypes() Report only names/dtypes present in the installed Torch: candidates are float8_e4m3fn, float8_e5m2, float8_e4m3fnuz, float8_e5m2fnuz, and float8_e8m0fnu.
available_float4_packed_dtype_names() / available_float4_packed_dtypes() Conditionally report float4_e2m1fn_x2; absence of this Torch dtype is not the same as absence of a portable uint8 decoder.
get_device_dtype_support(device=None, *, validate=False) Returns cached DeviceDTypeSupport, separating architecture-advertised and runtime-probed linear dtypes.
device_supports_dtype(device, dtype, *, require_validation=False) Checks the advertised set by default; requests probing when required.
device_supports_native_fp8(device=None, *, require_validation=False) Compatibility wrapper for E4M3FN linear support.
device_supports_native_fp4(device=None, *, require_validation=False) Checks packed FP4 linear support, if that dtype exists.
dequantize_f8_e4m3(...) / dequantize_fp8(...) Decode floating weights with optional scale metadata, axis=0, and target_dtype=torch.bfloat16. The latter forwards to the former.
dequantize_f4_e2m1(...) Decode two E2M1 values per packed byte, with the same scale/axis/target defaults.
decode_e8m0_scale(scale) Decode uint8/E8M0FNU exponent bytes to FP32 powers of two; byte 255 is NaN.
quark_floatx_formats(config) Inspect global and per-layer Quark weight declarations; distinguish FP8/NVFP4 from MX formats using E8M0 scale declarations and block/group shape.

The dequantizer signature pattern, copied from the FP4 helper, is:

def dequantize_f4_e2m1(
    tensor: torch.Tensor,
    *,
    scale: Optional[torch.Tensor] = None,
    scale_inv: Optional[torch.Tensor] = None,
    axis: Optional[int] = 0,
    target_dtype: torch.dtype = torch.bfloat16,
) -> torch.Tensor:

Providing both scale and scale_inv raises. scale multiplies decoded values. For compatibility, scale_inv is interpreted by magnitude: a maximum absolute value at most 1 multiplies, otherwise it divides. Check checkpoint metadata conventions instead of assuming the argument name always means division. Scale expansion supports scalar, per-axis, same-shape, and divisible block layouts; incompatible shapes raise.

The portable FP4 fallback interprets the low nibble first, high nibble second, with E2M1 magnitudes (0, 0.5, 1, 1.5, 2, 3, 4, 6) and a sign bit. It doubles the last dimension. TorchAO unpack/decode functions are used when available; otherwise a Torch-only path warns once. Eligible CPU FP8/FP4 decoding can use a JIT native extension, with reference fallbacks and layout-dependent dispatch.

Architecture advertising is deliberately not a backend guarantee. The CUDA map includes FP16/FP32, BF16 at major capability >= 8, available FP8 dtypes at major >= 9 or exactly (8, 9), and packed FP4 at major >= 10. Non-CUDA/other devices conservatively advertise FP16/FP32/BF16. With validate=True, the CUDA path runs small operations; FP4 probing also needs TorchAO NVFP4 support. The non-CUDA path simply accepts those three portable dtypes without executing a probe. This generic map is broader than the concrete GB10-targeted NVFP4 backend described below, and is not a ROCm capability certification.

DeviceDTypeSupport is an immutable dataclass containing device, optional (major, minor) capability, advertised_linear_dtypes, and validated_linear_dtypes. Its supports(dtype, require_validation=False) reads the advertised set unless validation is explicitly requested. A record obtained with validate=False initializes its validated_linear_dtypes to the advertised set too; calling record.supports(..., require_validation=True) does not itself run probes. Obtain a record with get_device_dtype_support(..., validate=True), or use device_supports_dtype(..., require_validation=True), to request the validation path. Even then, the non-CUDA shortcut above applies. A returned support record is not evidence that every quantized kernel supports the dtype or that an optional extension built successfully.

Group Aware Reordering (GAR)

GAR reorders Hessian-sensitive columns without mixing the membership of complete quantization groups. In ordinary GPTQConfig, desc_act resolves to False by default and an unspecified act_group_aware resolves to True for METHOD.GPTQ. The declaration uses Optional[bool] = None so explicit user choices can be distinguished during normalization.

from gptqmodel import GPTQConfig

# Explicit form, useful when comparing runs.
config = GPTQConfig(
    bits=4,
    group_size=128,
    desc_act=False,
    act_group_aware=True,
)

desc_act=True with no explicit GAR choice automatically disables GAR with a warning. Explicitly enabling both raises. GAR requires a positive group size; disable it for group_size=-1. It needs Hessian statistics, so it is not a weight-only RTN technique.

Permutation construction

The production functions in gar.py are:

def compute_local_perms(
    diag_H: torch.Tensor,
    groupsize: int,
    *,
    return_values: bool = False,
) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]:

def compute_global_perm(
    diag_H: torch.Tensor,
    groupsize: int,
    metric: str = "max",
    *,
    precomputed_values: torch.Tensor | None = None,
) -> torch.Tensor:
Step/helper Behavior
compute_local_perms() Reshape complete groups and sort their Hessian diagonal values descending. On CUDA, positive group sizes <= 192 use sorted topk; other cases use torch.sort. Can also return the sorted values.
compute_global_perm() Sort complete groups by descending score; supported metrics are "max", "mean", "sum", and "median". Production GPTQ uses the default max metric, reusing the first sorted local value.
compose_final_perm(local_perms, global_perm, groupsize) Add original group offsets, reorder the groups, and flatten. Semantics: perm[new_position] = old_index.
extend_perm_with_tail(perm, total_columns) Append untouched trailing columns not included in full groups.
invert_perm(perm) Construct the inverse satisfying inverse[perm] == arange(len(perm)).

The GPTQ solve applies the permutation to weight columns and both Hessian axes. After solving, it restores column order and reorders group scales/zeros back. The trailing partial group is preserved, not silently dropped. Static groups take a separate branch rather than this dynamic GAR branch.

Global group-score ties use stable argsort if supported by Torch, with a process-cached support check and warning on older implementations. Do not assume the local topk and sort branches give a universal stable tie order. Standalone helpers only operate on complete groups; callers must append the tail before treating the result as a full-column permutation.

gar_ref.py contains loop-based compute_local_perms_original, compute_global_perm_original, and compose_final_perm_original. Its global metric is max, its loops extract Python scalars, and its final tensor is constructed on CPU. It is a reference implementation, not the selected production fast path. The reference composer's prose is less explicit than its code; use the new-position-to-old-index semantics above when integrating a permutation.

FOEM's solver implements desc_act and static groups but does not call GAR's permutation helpers. Do not interpret the inherited act_group_aware field as proof of GAR processing inside FOEM.

FOEM and GPTAQ correction

Configuration

FOEMConfig lives in config.py and is exported from gptqmodel:

Field Default Meaning/validation
alpha 0 Numeric GPTAQ correction coefficient. 0 disables native-input correction inside FOEM.
beta 0.2 Numeric FOEM feedback coefficient.
device "auto" Device selection for native activation capture when that capture is needed; accepts a nonempty string or torch.device. It is not an independent inference-backend selector.

The dataclass checks numeric types for alpha/beta; it does not enforce finiteness or a [0, 1] range. The optional MLX implementation applies stricter beta validation. Use nonnegative, finite coefficients; the lifecycle schedules native capture when alpha > 0, while FOEM.__init__() treats any nonzero alpha as needing native inputs. Negative alpha is therefore not a supported tuning recipe just because construction accepts it.

from gptqmodel import FOEMConfig, GPTQConfig, GPTQModel

config = GPTQConfig(
    bits=4,
    group_size=128,
    desc_act=False,
    act_group_aware=False,
    foem=FOEMConfig(alpha=0, beta=0.2, device="auto"),
)
model = GPTQModel.load("path/to/unquantized-model", quantize_config=config)
model.quantize(calibration)
model.save("int4-foem")
Coefficients Intended relation stated in FOEMConfig
alpha=0, beta=0 GPTQ-equivalent feedback rule.
alpha>0, beta=0 GPTAQ correction; the config documentation recommends alpha=0.25.
alpha=0, beta>0 FOEM-only feedback; the config documentation suggests exploring beta in [0.1, 0.25].
alpha>0, beta>0 Combined correction; the documented example is alpha=0.25, beta=0.2, without a guarantee of beating FOEM alone.

These are algorithmic relationships, not a claim of bitwise identity between every separate solver implementation. In GPTQProcessor, an existing gptaq config takes precedence over foem. To combine FOEM with GPTAQ, set FOEMConfig.alpha; do not simultaneously set gptaq and expect both processor classes to run. Dynamic per-module foem values accept FOEMConfig or dictionaries.

Current serialization caveat: GPTQConfig._update_meta_payload() emits its foem entry only in the branch where both gptaq and foem are non-None; with FOEM alone, that method emits gptaq: null instead. Do not rely on this serializer to preserve a FOEM-only experiment recipe. Record/reapply the coefficients when reproducing quantization; setting gptaq merely to force serialization would change processor selection. The saved quantized weights do not need FOEM's solver to run inference.

The FOEM/native-capture path is rejected by the weight-only lifecycle and by EXL3 quantization. Positive alpha requires native reference activations and an additional NativeProcessor; plan for the extra activation storage rather than treating it as a free coefficient change.

Solver internals

FOEM(GPTQ) accepts module: NamedModule, optional qcfg, and region_timer; its quantize(self, blocksize=128) returns the ordinary eight-element quantizer result: reconstructed weight, scales, zeros, group indices, duration, average loss, damping, and sample count.

  • add_batch() serializes updates with the instance lock.
  • process_batch() explicitly uses FP32 inputs and accumulates H; for correction mode it also accumulates dXXT from native versus current inputs. Its sample normalization counts batch requests/examples, not the flattened token count of the newer GPTQ accumulation path.
  • Linear/Transformers Conv1D inputs are flattened and transposed; Conv2d inputs use nn.Unfold.
  • Dead columns are neutralized before solving. Static-group quantizers and descending activation permutations have their own branches.
  • The inherited Hessian inverse routine remains eager, because compiling around its damp-recovery exception loop can hang.
  • A copy W_raw retains the pre-feedback weight. For the current column, the feedback error is ((w - q) - (w - w_raw) * beta) / d, where d is the current diagonal of the inverse factor. Updates propagate through the inverse factor; the next within-block column receives an additional beta correction.
  • For nonzero alpha, the additional matrix is P = alpha * triu(dXXT @ Hinv.T, diagonal=1) @ Hinv. Both within-block and cross-block updates include this correction.
  • The solver restores activation permutation/layout, produces g_idx, sanitizes scales/zeros, and rejects NaN average loss.

Hinv here is the upper Cholesky factor produced by the existing inverse path. It is not an invitation to pass the raw Hessian directly to the MLX function.

Native MLX FOEM

gptqmodel/quantization/mlx_foem.py implements the alpha-zero group update using mx.fast.metal_kernel. Its actual signature is:

def foem_quantize_weight_mlx(
    weight,
    inverse_hessian,
    *,
    bits: int = 4,
    group_size: int = 128,
    beta: float = 0.2,
    sym: bool = True,
    return_residuals: bool = False,
):
Requirement Direct MLX helper Automatic dispatch from foem.py
Platform Uses MLX and Metal Available only on macOS (darwin), ARM64, with mlx.core importable.
Input matrix Nonempty rank 2; (out_features, in_features); MLX FP16/BF16/FP32 Torch CPU/MPS weight with at least 256 * 1024 elements; inverse factor on the same device.
Inverse factor Shape (in_features, in_features), FP32, finite, positive diagonal Supplied by inherited Hessian inversion.
Bits Integers in 2..8 Integer config bits in 2..8.
Groups 16, 32, 64, or 128, dividing input width Same, and blocksize == group_size.
Feedback beta in [0, 1]; Boolean sym and return_residuals alpha == 0, beta in [0, 1], no desc_act, no static groups, and ordinary MSE search disabled (not qcfg.mse).

The default blocksize=128 means that merely choosing a smaller supported group size does not automatically meet the blocksize/group-size equality gate. When any automatic-dispatch condition fails, the Torch FOEM implementation remains the path; there is no config flag that overrides the gate.

One Metal thread processes a weight row within a group. The kernel computes group quantization parameters and sequential error feedback in FP32; cross-group correction uses _accurate_matmul_mlx. It returns FP32 reconstructed weights, scales, and zeros, plus optional per-weight residuals. The Torch bridge uses DLPack, synchronizes MPS before export, reconstructs the loss from residuals, and converts the result back to the module's shape/dtype. This is quantization-time acceleration, not a persistent dequantized inference weight cache.

Floating activation policies and calibration

Configuration and requirements

activation defaults to None. With a W4A policy, GPTQ weights remain native INT4 in INT32-packed storage. The policy controls activation operands and encoded transport between selected operators.

Mode Encoding Scale representation
"w4afp8" E4M3 activation codes Dynamic FP32 scale per token/row.
"w4a_nvfp4" E2M1 activation codes E4M3 local scale per 16 values, plus an FP32 global scale.

ActivationConfig defaults are mode="w4afp8", recipe=None, attention=None, mlp=None, and global_scales=None. For NVFP4, a missing recipe resolves to "least_squares". An omitted attention override inherits the stream policy; the distinct ActivationAttentionConfig class defaults its own mode to "w4afp8" when explicitly constructed. Do not confuse the two defaults.

from gptqmodel import GPTQConfig

config = GPTQConfig(
    bits=4,
    group_size=128,
    sym=True,
    desc_act=False,
    activation={
        "mode": "w4a_nvfp4",
        "recipe": "least_squares",
        "attention": {"mode": "w4afp8"},
        "mlp": {"mode": "w4afp8", "layers": [0, 1]},
    },
)

The example keeps attention and the first two MLP layers on FP8 while other MLP layers use the stream's NVFP4 policy. MLP overrides only promote to "w4afp8"; indices must be a nonempty list/tuple of unique nonnegative integers and are sorted. Attention can use FP8 without a recipe, or NVFP4 with a resolved recipe. Attention/MLP refinements are restricted to an NVFP4 base stream.

Config validation requires METHOD.GPTQ, GPTQ/GPTQ_V2 format, bits=4, group_size=128, sym=True, desc_act=False, INT32 packing, and lm_head=False. Dynamic weight-layout overrides for bits, sym, group_size, or desc_act are rejected in this mode.

The concrete W4ANVFP4Linear backend supports Linux CUDA FP16/BF16, group size 128, input width divisible by 128 and output width divisible by 32, no automatic padding, adapters, or training. This release's implementation targets GB10/SM 12.1 (MIN_COMPUTE_CAPABILITY=(12, 1)) and requires Triton packing, Torch's float4_e2m1fn_x2, and torch.nn.functional.scaled_mm, ScalingType, and SwizzleType. Do not substitute the generic FP4 dtype capability map for these stricter kernel requirements. The loaded encoded producer stream is Llama-specific. See FP8 and NVFP4 and Kernel Backends Reference for the broader backend surface; these paths are not an MLX/ROCm activation-stream implementation.

Recipes in activation_floatx.py

Canonical recipe Local scale fitting Extra behavior
"nvidia" One maximum-based E2M1 M=6 candidate Complete E2M1 range.
"four_six" Compare candidates based on M=4 and M=6 Select lower actual block squared error.
"least_squares" Start with both candidates, refine each code assignment, then refine the winner Default NVFP4 recipe; accept only strict error improvements.
"least_squares_grid" Least-squares result plus an E4M3 encoding-neighborhood search Search offsets -8..8, clamped to positive finite encoding bytes 1..126.
"nvidia_headroom" Local nvidia fitting Legacy headroom-calibrated global maximum policy.
"least_squares_headroom" Local least-squares fitting Legacy headroom-calibrated global maximum policy.

Legacy names "lsq", "lsq_headroom", and "lsq_grid" normalize to their least_squares* equivalents. Recipe matching is against the supported names; unknown names raise. The recipe fitter is not learned-scale QAT.

The principal helper signatures are:

def fp8_token_qdq(x: torch.Tensor) -> torch.Tensor:

def nvfp4_global_scale(amax: torch.Tensor | float, *, grid_dtype: torch.dtype | None = None,
                       recipe: str = "least_squares") -> torch.Tensor:

def nvfp4_block_qdq(x: torch.Tensor, global_scale: torch.Tensor | float,
                    recipe: str = "least_squares") -> torch.Tensor:
  • FP8 QDQ converts to FP32, finds the row maximum, uses amax / 448 (or 1 for a zero row), clamps/casts to E4M3FN, reconstructs, and restores the input dtype. Empty inputs are returned unchanged; non-finite inputs raise.
  • NVFP4 QDQ requires the last dimension divisible by 16, finite inputs, and positive finite global scale. It reshapes to blocks of 16, fits E4M3 local scales, uses finite-saturating E2M1 nearest/ties-to-even rounding, reconstructs, and restores the input dtype. Empty inputs are returned unchanged.
  • E2M1 magnitudes are 0, 0.5, 1, 1.5, 2, 3, 4, and 6. E4M3's maximum used here is 448.
  • nvfp4_global_scale() requires a finite nonnegative maximum. The denominator is 448 * 6 for nvidia or headroom recipes and 448 * 4 for the optimized non-headroom recipes. Zero maximum yields scale 1.
  • Optional grid_dtype can be FP16, BF16, or FP32. Scale stays FP32, retaining a positive FP32 value if source-grid rounding would underflow to zero. Producer maximum calibration does not request BF16 grid rounding.

NVFP4ActivationHeadroom is a separate bounded-histogram collector, not the post-load producer calibration algorithm. Its constructor defaults are:

def __init__(self, *, anchor_percentile: float = 1.0,
             upper_percentile: float = 99.99, rho: float = 16384.0,
             num_bins: int = 512, log2_min: float = -40.0,
             log2_max: float = 40.0):

It records one maximum per 16-value block in fixed log2 bins and tracks a running maximum, without retaining activation tensors. Percentiles must be in (0, 100] and rho in (0, 28672). collect() requires width divisible by 16 and finite block maxima; empty inputs do nothing. compute_amax() needs nonzero observations. It estimates an upper percentile (or literal maximum when requested), computes an anchor while excluding bins below upper / 1e6, and proposes rho * anchor. If that does not exceed the upper value, it warns and uses the upper value instead. GPTQ's headroom probe freezes this scale before the Hessian capture pass.

The replay helpers in gptq_processor.py require nn.Linear, wrap a solved target's forward with FP8/NVFP4 input QDQ, and avoid a second QDQ when the encoded-stream replay pre-hook is already active. They do not turn every arbitrary module into a supported W4A target.

Post-load producer maximum calibration

The public method and underlying functions are:

def calibrate_activations(self, samples: List[torch.Tensor], *, progress=None) -> Dict[str, Any]:

@torch.inference_mode()
def calibrate_nvfp4_producers(core, samples, *, quantize_config=None, progress=None) -> dict:

@torch.inference_mode()
def measure_nvfp4_producers(core, samples) -> dict:

Use the public method for normal checkpoint saving. It supplies the model's quantize config so the fitted scales are persisted:

from gptqmodel import BACKEND, GPTQModel
from gptqmodel.quantization.activation_calibration import measure_nvfp4_producers

# This checkpoint must carry an NVFP4 activation policy and load with its
# supported encoded Llama stream on compatible hardware.
model = GPTQModel.load("path/to/w4a-nvfp4-checkpoint", backend=BACKEND.GPTQ_W4A_NVFP4)
model.model.eval()

# fit_samples and held_out_samples are separate nonempty lists of unpadded,
# one-dimensional token-ID tensors prepared with this model's tokenizer.
report = model.calibrate_activations(fit_samples)
held_out = measure_nvfp4_producers(model.model, held_out_samples)
model.save("w4a-nvfp4-calibrated")

Preconditions are checked before calibration:

  • The public wrapper requires an already quantized model and quantize config.
  • The core must already have _w4a_stream_mode == "w4a_nvfp4" installed and must be in evaluation mode.
  • Samples must be nonempty, unpadded, one-dimensional token vectors. Capture disables KV caching and rejects unexpected positional decoder arguments or nonempty past-key-value state.
  • Stream and attention recipes must be one of nvidia, four_six, least_squares, or least_squares_grid; legacy headroom recipes are rejected.
  • If a config is supplied, its activation/attention/MLP promotion policy must match the installed policy.
  • Every enumerated NVFP4 boundary must have an idle NVFP4BoundaryQuantizer. Mixed FP8 boundaries are not given NVFP4 producer scales.

Calibration captures first-decoder inputs to CPU, then processes producers in layer/topological order. Each boundary observes a finite nonempty token tensor and accumulates absolute maximum, token count, and call count. Its scale is computed with that boundary's runtime recipe and frozen before later boundaries are measured. Actual calibrated encoded outputs become the next layer's cached inputs. Only one layer's samples are retained at a time; the fit set is still cached on CPU, so memory grows with sample count and sequence length.

The result contains algorithm="producer_maximum", the stream recipe, global_scales, per-boundary records (recipe, amax, global_scale, tokens, samples), upstream_activations="calibrated_encoded_carriers", and weight_updates=0. An optional progress(key, record) callback runs after each producer is calibrated.

On failure, the function restores producer scale bits/calibrated flags, runtime global-scale policy, config activation payload, and HF quantization metadata, and clears observers. With a supplied config, successful scales are validated through config reconstruction and copied into core.config.quantization_config. Without one, the caller must export the returned scale map.

measure_nvfp4_producers() installs temporary forward hooks, decodes each producer's encoded output to FP32, and collects SSE, reference energy, element count, maximum, and calls. It returns relative_rmse = sqrt(sse / max(energy, 1e-30)). It does not update scales, and hooks are removed in a finally block. This measures local activation reconstruction, not end-to-end model accuracy.

Post-quantization norm tuning

Public workflow

Norm tuning performs norm-only knowledge distillation, related to Norm Tweaking. It requires an already quantized student, calibration data, and an unquantized causal-language-model teacher. The teacher must have compatible token IDs and output-vocabulary semantics: the implementation directly compares logits from the same input IDs and has no vocabulary remapping stage.

from gptqmodel import BACKEND, GPTQModel, NormConfig, NormType

# Save fresh quantize() results and reload before entering this workflow.
model = GPTQModel.load("path/to/saved-quantized-model", backend=BACKEND.TORCH)

selected = model.norm_module_names(
    types=(NormType.RMSNORM, NormType.LAYERNORM),
    skip=("q_norm", "k_norm"),
)

result = model.requantize(
    calibration=calibration,
    norm=NormConfig(
        teacher="path/to/unquantized-teacher",
        types=(NormType.RMSNORM, NormType.LAYERNORM),
        steps=400,
        lr=1e-3,
        reg=1e-3,
        seed=0,
        skip=("q_norm", "k_norm"),
    ),
)
stats = result["norm"]
model.save("norm-tuned-model")

NormConfig is passed to the keyword-only norm= parameter of requantize(), not attached as a QuantizeConfig.norm field. NormConfig.teacher can also be an already loaded torch.nn.Module. If it is a path/ID, _requantize_norm() loads AutoModelForCausalLM in the student's dtype on the input-embedding device, with the student's trust_remote_code; it releases an internally loaded teacher afterwards. Passing an already loaded teacher lets the caller control its placement and ownership.

NormConfig and eligible modules

Field Default Contract
teacher None in declaration, but required by validation Unquantized reference path/HF ID or loaded module; None raises.
types (NormType.RMSNORM,) Selected families. Accepts a member/string or iterable; normalize to a validated, deduplicated tuple. Empty selection raises; None selects RMSNorm.
steps 400 Must be positive.
lr 1e-3 Finite and positive Adam learning rate.
reg 1e-3 Finite and nonnegative pull toward checkpoint norm parameters.
seed 0 Seeds local batch sampling and Torch.
skip () Module-name suffixes to exclude; converted to a tuple. Pass a tuple/list of suffix strings, not a single string.
NormType member Value Recognition
RMSNORM "rmsnorm" nn.RMSNorm, or class name ending in RMSNorm. Includes supported 1 + weight gain parameterizations.
LAYERNORM "layernorm" nn.LayerNorm, or class name ending in LayerNorm; tunes floating weight and optional bias.

Eligibility comes from model definitions, not from sweeping every norm-like class in the model. get_norm_modules() reads :norm tags in module-tree variants. norm_module_names(types=None, skip=()) expands layer-relative tagged names over decoder layers, filters by family/suffix, and includes pre_lm_head_norm_module when appropriate. An unsupported tagged class raises; absent paths can be skipped. A definition with no matching declared modules cannot be norm-tuned by this interface.

resolve_norm_params() validates floating trainable gains, matching floating bias shape, and finite values. It deduplicates module names. The tuner also deduplicates aliases of the same nn.Parameter object; conflicting effective offsets/scales on that object raise. This is parameter-identity deduplication, not a general check for distinct parameter objects that happen to share underlying storage.

The gain handler probes a private copy of each module instance with small stored gains -0.5 and 0.5, zeroed bias, alternating-sign inputs, and that module's normalization epsilon. It distinguishes effective gain weight from 1 + weight with maximum absolute tolerance 0.05. Empty/scalar gains, non-finite probes, incompatible output shapes, ambiguous gain parameterizations, or failed copies/forwards raise. Exclude unsupported instances with NormConfig.skip; a matching class-name suffix alone does not guarantee a successful probe.

Exact public requantize() signature

From gptqmodel/models/base.py:

def requantize(
    self,
    calibration: Union[List[Dict[str, Union[List[int], torch.LongTensor]]], List[str], List[int]],
    embed_quant_config: Optional[Union[QuantizeEmbedConfig, QuantizeEmbed]] = None,
    calibration_concat_size: Optional[int] = None,
    calibration_sort: Optional[str] = "desc",
    batch_size: int = 1,
    tokenizer: Optional[PreTrainedTokenizerBase] = None,
    backend: Optional[BACKEND] = BACKEND.AUTO,
    adapter: Adapter = None,
    adapter_calibration_dataset: Union[List[Dict[str, Union[List[int], torch.LongTensor]]], List[str], List[int]] = None,
    calibration_data_min_length: int = 10,
    calibration_concat_separator: Optional[str] = None,
    embed_quant_mode: Optional[QuantizeEmbed] = None,
    *,
    norm: Optional[NormConfig] = None,
) -> Dict[str, List[Dict[str, str]]]:

The annotation is broader lifecycle legacy typing: the actual "norm" payload is a statistics dictionary, not a list of string dictionaries. The model must already be quantized, norm must be a NormConfig, and at least one embedding target or norm target must be supplied. Calibration is prepared through the ordinary dataset helper, using the concat/sort/batch/minimum-length settings above. Norm-only processing does not switch the loaded quantized kernel merely because backend= is present on this shared signature; load with an autograd-capable backend first.

Objective and optimization

The lower-level tuning function is:

def tune_norm_params(
    model: nn.Module,
    teacher: nn.Module,
    batches: List[Dict[str, torch.Tensor]],
    module_names: List[str],
    config: NormConfig,
) -> Dict[str, Any]:

Each step samples one prepared batch with replacement using random.Random(seed). An absent attention mask becomes all ones. Both models run with use_cache=False; the teacher uses no_grad(). The loss is the attention-mask-weighted mean of KL(teacher || student) from FP32 log_softmax outputs over vocabulary. It compares the next-token distribution at every unmasked input position; there is no separate labels/cross-entropy objective or configurable distillation temperature.

Only selected norm parameters have gradients enabled. The model and teacher remain in eval mode, because quantized linear implementations can switch layouts in train mode; eval mode does not disable autograd. The optimizer is Adam over FP32 master copies. After each finite-gradient step, the master values are copied back to the original parameter dtype.

Regularization sums per-parameter mean squared deviation from the initial checkpoint parameter, divided by the square of its scale, and multiplies by reg. A gain's scale is mean(abs(weight + detected_offset)), floored at 1e-6. Bias uses the same gain scale so a zero-initialized bias is not effectively frozen. Regularization is relative to the effective gain, which is important for Gemma-style near-zero stored weights representing gain near one.

The first step raises if gradients do not reach a selected parameter, suggesting an autograd-capable kernel such as Torch. Non-finite gradients cause a skipped optimizer step and increment skipped_steps; they do not silently overwrite parameters. Histories are logged at step 1, every max(1, steps // 10) steps, and the final step. On exit, gradients are cleared, original requires_grad flags and model/teacher training modes are restored, and TorchLinear streaming settings are restored. Torch's RNG is seeded as part of tuning; the implementation does not save/restore the previous global RNG state.

result["norm"] key Value
modules Selected module paths, used for persistence.
types Requested normalized family strings.
params Labels such as model.layers.0.input_layernorm.weight, including eligible biases.
steps Requested number of steps.
skipped_steps Number skipped for non-finite gradients.
history Logged records containing step, kl, and unweighted penalty.

The writer tracks the tuned module prefixes and, for a loaded checkpoint, uses replacement-only saving so other checkpoint tensors are preserved. Norm tuning does not repack or optimize qweight, qzeros, quantization scales, or group indices. It cannot repair an unsupported inference backend or replace end-to-end quality evaluation.

Replacement-only saving currently requires source checkpoint shards in safetensors format. Keep the original checkpoint available: the writer uses it as the source of untouched tensors and files. Prefer a new sibling output directory; a destination nested inside the checkpoint source is explicitly rejected. Rewritten shards are serialized again, so preserving untouched tensor values does not mean the entire shard file remains byte-for-byte identical. The implementation is in gptqmodel/models/writer.py, including _save_module_replacement_safetensors() and _copy_missing_checkpoint_files().

Required save/reload boundary and embedding interaction

validate_tuning_model() runs before dataset preparation or teacher loading and again inside the tuning path. It rejects:

  • Meta parameters or buffers, including unresolved meta-backed offload state. Ordinary materialized CPU tensors are not rejected merely for being on CPU.
  • Inference tensors among model parameters.
  • TorchLinear modules in gptq (v1) format whose runtime zero points have not been converted to v2 form (qzero_format() != 2).

The recovery is to save the quantized model, then reload with backend=BACKEND.TORCH. The loader owns the zero-point conversion. The tuner does not silently mutate checkpoint layout or tensor ownership to make backward work. Native gptq_v2/planar modules are not rejected just because a conversion marker remains at its default.

Before enabling autograd, it clears TorchLinear inference weight caches and disables streaming, then restores streaming settings afterwards. This is a training-time runtime adjustment, not permission for serving kernels to retain expanded weight caches.

Embedding requantization and norm tuning can share a call for a training-capable gptq_v2 model. A combined gptq (v1) embedding/norm call is explicitly rejected before either mutation. Use separate passes:

from gptqmodel import BACKEND, GPTQModel, NormConfig, QuantizeEmbed

model.requantize(calibration=calibration, embed_quant_mode=QuantizeEmbed.OUTPUT)
model.save("embedding-requantized")

model = GPTQModel.load("embedding-requantized", backend=BACKEND.TORCH)
model.requantize(calibration=calibration, norm=NormConfig(teacher="path/to/teacher"))
model.save("embedding-and-norm-refined")

Allow memory for both teacher and student plus full-vocabulary logits. Moving a teacher to a different device does not remove the student's autograd or calibration memory requirements. A failed tuning call restores modes/runtime settings, but is not a transactional rollback of successful parameter updates from earlier steps; reload the saved source checkpoint to restart a failed experiment cleanly.

Pitfalls and edge cases

Symptom or assumption Check/action
Rotation constructor succeeds but quantization fails Check model class, selected backend, CUDA/HIP-backed weight-transform requirements, and supported Hadamard dimensions. Config metadata is not the runtime capability check.
Rotated checkpoint produces wrong outputs in another engine Confirm online mlp.down_proj Hadamard handling and retained rotation metadata; do not double-transform activations.
Smoother appears to do nothing Check actual quantizer/packer path, threshold versus effective group size, whether the descriptor is active, and whether fallback was selected. ModulePreProcessor itself only records plans.
MAD runs differ between dictionary and typed configuration Specify k; parser default is 3.0, constructor default is 2.75.
MSE smoother ignores a group threshold Dedicated RTN/fallback MSE branches bypass smooth_block()'s threshold guard.
Row/column smoothing fails during GGUF, FP8, or BitsAndBytes direct packing All three use a helper that rejects post-quantization scale factors. Use a clipping smoother supported by the chosen path instead.
GAR with group_size=-1 or desc_act=True GAR requires a positive group size and mutually exclusive ordering. Explicitly disable GAR for full-row quantization.
FOEM plus gptaq does not combine as expected GPTAQ has processor precedence. Use FOEMConfig(alpha=..., beta=...) alone for combined feedback.
MLX FOEM not selected on Apple Silicon Check every dispatch predicate, including matrix size, MSE disabled, blocksize equal to group size, alpha zero, and supported device/dtype/layout.
Producer calibration rejects a checkpoint It needs an installed encoded NVFP4 Llama stream in eval mode, modern non-headroom recipes, matching save config, and idle boundary quantizers. It is not a generic dense-model calibration API.
Norm tuning raises before loading the teacher Save/reload into Torch, inspect :norm eligibility and selected types, and check meta/inference tensors and zero-point layout.
Norms selected but no gradient reaches them Use an autograd-capable loaded backend; selecting an inference-only W4A kernel is not sufficient.
Norm tuning reports no matching modules LayerNorm is opt-in; inspect norm_module_names(), the definition's tags, and skip suffixes.
Calibration reconstruction error improves but generation does not Local activation RMSE and training KL are not end-to-end accuracy. Evaluate held-out generation/perplexity/task metrics using the actual serving policy.

Source files

Paths below are relative to the repository root:

Source Responsibility
gptqmodel/quantization/config.py Rotation field; every smoother; preprocessor configs/normalization; NormConfig/NormType; FOEMConfig; activation policies; method-specific config dispatch.
gptqmodel/quantization/rotation/rotation.py Norm fusion, basis generation, embedding/attention/MLP/head rotation.
gptqmodel/quantization/rotation/hadamard_utils.py Fixed Hadamard matrices, dimension factorization, backend selection, online transforms and autograd wrapper.
gptqmodel/utils/hadamard.py Vendored JIT extension registration and direct transform wrapper.
gptqmodel_ext/hadamard/hadamard.cpp, gptqmodel_ext/hadamard/fast_hadamard_transform_cuda.cu Native Hadamard operator sources.
gptqmodel/quantization/fallback_smooth.py Weight clipping/rescaling and dedicated MSE range search.
gptqmodel/quantization/rtn.py, gptqmodel/quantization/qqq.py RTN and QQQ smoothing consumers.
gptqmodel/quantization/gptq.py GPTQ fallback smoothing, GAR integration, Hessian inverse and scale handling.
gptqmodel/quantization/gar.py, gar_ref.py Production permutations and loop-based reference.
gptqmodel/quantization/foem.py, mlx_foem.py FOEM accumulation/solve and Metal group-update implementation/dispatch.
gptqmodel/quantization/activation_floatx.py FP8/NVFP4 QDQ, recipes, global-scale math, bounded headroom histogram.
gptqmodel/quantization/activation_calibration.py Topological producer maximum calibration, rollback, reconstruction measurement.
gptqmodel/quantization/dtype.py Floatx dtype registries, capability probes, FP8/FP4/E8M0 decode helpers.
gptqmodel/quantization/norm_tuning.py Norm classification/probing, guardrails, teacher KL objective, FP32 master optimization.
gptqmodel/models/base.py, gptqmodel/models/loader.py, gptqmodel/models/writer.py Lifecycle entry points, norm selection, teacher loading, replacement persistence, online Hadamard setup.
gptqmodel/looper/module_preprocessor.py, gptqmodel/looper/gptq_processor.py, gptqmodel/looper/native_processor.py Plan construction, algorithm dispatch, replay hooks and native activation capture.
gptqmodel/nn_modules/qlinear/gguf.py, fp8.py, bitsandbytes.py Weight-only packing/smoothing consumers.
gptqmodel/nn_modules/qlinear/w4a_nvfp4.py, w4a_activation.py, w4a_boundary.py, w4a_llama_replay.py Concrete NVFP4 backend, encoded carriers, producer boundary quantizers, Llama replay.
README.md Preprocessor usage and experimental norm-tuning guide.
docs/gptq-w4a-gb10.md, docs/gptq-w4a-accuracy-notes.md W4A policy/hardware guide and accuracy interpretation.
tests/test_norm_config.py, test_norm_guards.py, test_norm_lifecycle.py, test_norm_requant.py, test_norm_review_regressions.py under tests/ Norm family, parameterization, guard and lifecycle regression coverage.
tests/models/test_w4a_producer_calibration.py, tests/models/test_w4a_calibration_lifecycle.py, tests/kernels/test_w4a_nvfp4_gb10.py Producer calibration, rollback and floating activation numerical tests.

See also

Documentation home · Quickstart · Troubleshooting · Repository · PyPI

Source baseline: GPT-QModel 7.7.0 / c197691. Maintained by ModelCloud.ai.

Home

Getting started

Quantization methods

Complementary configs

Using a model

Platforms

Internals

Clone this wiki locally