FP8 and NVFP4

FP8 and NVFP4

Documentation home · Quickstart · Lifecycle

On this page

• Overview
• Workflow at a glance
• Concepts
• FP8 usage and configuration
• FP8 weight implementation
• Native FP8 and FP4 checkpoint sources
• GPTQ W4A-FP8 and W4A-NVFP4
• Activation mathematics and kernels
• GSQ refinement
• Pitfalls and edge cases
• Source files
• See also

Overview

GPT-QModel has several distinct floating-point quantization paths. They differ in what is stored in the checkpoint, whether activations are quantized, and which runtime is required. The word “FP8” alone is not enough to choose a configuration or backend.

Path	Saved weights	Activation contract	Configuration/runtime
FP8 weight-only quantization	FP8 weight plus FP32 inverse scales	Ordinary floating inputs; an eligible tensorwise CUDA fast path quantizes inputs internally	FP8Config, METHOD.FP8, FORMAT.FP8, TorchFP8Linear
Native FP8 source checkpoint	Source-format FP8 weights and source scales	Weight-only runtime, or native forward/module-local decoding for a fresh GPTQ/AWQ solve, subject to capability and layout validation	Native floatx source detection and auto-module decoder
Native NVFP4 weight checkpoint	Packed E2M1 weights and block scales	Ordinary floating inputs	TorchFP4Linear / source decoder; not FP8Config(bits=4)
GPTQ W4A-FP8	Native GPTQ INT4 qweight, qzeros, scales, g_idx	E4M3 codes with one FP32 scale per token	activation="w4afp8", BACKEND.GPTQ_W4AFP8
GPTQ W4A-NVFP4	Native GPTQ INT4 weights, plus activation-policy metadata/scalars	Packed E2M1 codes, E4M3 block-16 scales, FP32 global scale	activation="w4a_nvfp4", BACKEND.GPTQ_W4A_NVFP4

The W4A paths do not save FP8 or FP4 weight checkpoints. They execute native GPTQ INT4 weights through floating tensor-core operands at runtime. Conversely, FP8Config is not a switch for the GPTQ W4A activation stream.

This page describes source revision 17d4a422a5163608a86c8d62e7b54e1df89b8402. Hardware and optional Torch/Triton APIs are checked at runtime; format recognition is not a promise of native execution on every device.

Workflow at a glance

Reading the diagram: These lanes are alternatives, not a conversion chain. W4A stores INT4 weights, not FP8/NVFP4 weights. Native-source and hardware support remain layout-specific.

Text version of the workflow

• Weight-only FP8 export: Configure FP8 weights → Pack original weights → Save FP8 checkpoint.
• GPTQ with a W4A policy: Set activation policy → Calibrate and solve → Save INT4 weights → Run encoded activations.
• Native source inference: Inspect source metadata → Map source tensors → Use the source runtime.
• Native FP8 re-quantization: Choose a fresh target → Decode module weights → Calibrate target weights.

Open full-size SVG

Concepts

Method, format family, and actual FP8 dtype

Identifier	Value	Meaning
METHOD.FP8	"fp8"	Weight-only FP8 quantization method
FORMAT.FP8	"fp8"	FP8 checkpoint/backend family
FP8Config.format	Default "float8_e4m3fn"	Actual canonical Torch FP8 dtype name
BACKEND.FP8_TORCH	"fp8_torch"	Torch FP8 weight runtime
BACKEND.GPTQ_W4AFP8	"gptq_w4afp8"	GPTQ INT4 weights with FP8 activations
BACKEND.GPTQ_W4A_NVFP4	"gptq_w4a_nvfp4"	GPTQ INT4 weights with NVFP4 activations

FP8Config._resolve_checkpoint_format() normalizes its dtype-valued format and returns FORMAT.FP8 for backend selection. Thus a serialized format="float8_e5m2" still belongs to the FP8 family; it is not a second quantization method.

Inverse scales versus direct scales

GPT-QModel's FP8 weight format supports only weight_scale_semantics="inverse". For a scale region with maximum absolute weight amax and dtype limit fp8_max, the packer computes:

# Conceptual form of the implementation; all scale arithmetic uses FP32.
scale_inv = fp8_max / max(amax, torch.finfo(torch.float32).tiny)
qweight = clamp(weight * scale_inv, -fp8_max, fp8_max).to(fp8_dtype)
weight_reconstructed = qweight.float() / scale_inv

An all-zero region uses inverse scale 1, not an infinite or zero scale. Inverse-scale storage must not be confused with a source checkpoint's direct weight_scale, where reconstruction multiplies by the scale. The native-source loader normalizes recognized direct scales to inverse scales before constructing TorchFP8Linear.

The W4A activation helpers use their own scale conventions: FP8 token scales and NVFP4 local/global scales are multiplicative decoding scales. Do not pass them as weight_scale_inv merely because they are called scales.

Scale layouts

For a weight matrix shaped (out_features, in_features):

Method	weight_scale_inv shape	Regions
"tensor"	Scalar ()	One scale for the entire matrix
"row"	(out_features,)	One scale for each output row
"block"	(out_features // block_rows, in_features // block_cols)	One scale for each rectangular weight block

Block dimensions follow weight storage order, not activation order: [block_rows, block_cols] means [output_rows, input_columns]. Both dimensions of the matrix must divide exactly by the corresponding block size; the Torch FP8 backend does not automatically pad them.

FP8 usage and configuration

Weight-only quantization

from gptqmodel import GPTQModel
from gptqmodel.quantization import FP8Config

qcfg = FP8Config(
    format="e4m3",
    weight_scale_method="row",
    device="cuda:0",
)
model = GPTQModel.load("/path/to/dense-model", quantize_config=qcfg)
model.quantize()  # No activation calibration dataset is needed for this path.
model.save("/path/to/fp8-model")

FP8 uses the weight-only lifecycle: WeightOnlyLooper and WeightOnlyProcessor directly pack selected original weights. They do not collect activation Hessians or replay calibration forwards. If a calibration dataset is supplied, this path logs that it is ignored. Adapter/EoRA generation, GPTAQ, and FOEM/native activation capture are not supported in this lifecycle.

from gptqmodel.quantization import FP8Config

# Explicit rectangular blocks. All selected matrix dimensions must fit them.
block_qcfg = FP8Config(
    format="e5m2",
    weight_scale_method="block",
    weight_block_size=[128, 128],
    device="cuda:0",
)

# Explicit tensorwise scaling, including eligibility for the CUDA scaled-mm path.
tensor_qcfg = FP8Config(
    format="e4m3",
    weight_scale_method="tensor",
    device="cuda:0",
)

These configurations select different weight-scaling granularity; they do not change the method to GPTQ or enable a W4A stream.

FP8Config fields

FP8Config subclasses PreProcessorConfig, so shared device, dynamic, smoother, and preprocessing controls remain available. The FP8-specific declared fields are:

Field	Declared type	Default	Validation/normalization
bits	int	8	Must equal 8
method	METHOD	METHOD.FP8	Must be fp8
format	Optional[str]	"float8_e4m3fn"	Normalized to an available Torch dtype name
group_size	int	-1	Forced to -1; not used for FP8 weight scaling
desc_act	Optional[bool]	False	Forced to False
sym	bool	True	Forced to True
weight_scale_method	str	"row"	tensor, row, or block
weight_block_size	Optional[Union[List[int], Tuple[int, int]]]	None	Positive two-item block dimensions; stored as a list in normalized config
weight_scale_semantics	str	"inverse"	Only inverse supported

allowed_quant_methods() returns (METHOD.FP8,). quant_linear_init_kwargs() passes format, weight_scale_method, weight_block_size, and weight_scale_semantics to the runtime module. These same normalized settings are added to serialized output. uses_weight_only_lifecycle() returns True.

All FP8 normalization helpers

The format helper has this signature:

def _normalize_fp8_fmt(value: Optional[str]) -> str:

Accepted aliases	Canonical dtype name
e4m3, float8_e4m3, float8_e4m3fn	float8_e4m3fn
e5m2, float8_e5m2	float8_e5m2
e4m3fnuz, float8_e4m3fnuz	float8_e4m3fnuz
e5m2fnuz, float8_e5m2fnuz	float8_e5m2fnuz
e8m0, e8m0fnu, float8_e8m0, float8_e8m0fnu	float8_e8m0fnu
None, empty string, "fp8", or FORMAT.FP8	float8_e4m3fn

Strings are stripped and lowercased. Other FORMAT enum members, unknown aliases, and canonical dtypes missing from the installed PyTorch build raise ValueError.

Configuration recognition is broader than weight packing. float8_e8m0fnu is accepted for dequantization of existing checkpoints, but quantize_fp8_weight() explicitly rejects creating dense-weight E8M0 checkpoints. The native MLX weight encoder supports only the four E4M3/E5M2 variants in the table, not E8M0.

def _normalize_fp8_weight_block_size(value: Optional[Union[List[int], Tuple[int, int]]]) -> Optional[Tuple[int, int]]:

This helper accepts None or a two-item list/tuple, converts each entry with int(), requires both results to be positive, and returns a tuple. It validates the block specification, not divisibility of a particular matrix; divisibility is checked by the runtime/packer.

def _normalize_fp8_weight_scale_method(
    value: Optional[str],
    *,
    weight_block_size: Optional[Tuple[int, int]],
) -> str:

def _normalize_fp8_scale_semantics(value: Optional[str]) -> str:

Helper/input	Result
Scale method None with a block size	"block"
Scale method None without a block size	"row"
Explicit tensor, row, block	Stripped/lowercased and checked
block without a block size	Error
Non-block method with a block size	Error
Unknown method	Error
Scale semantics None	"inverse"
Explicit semantics string	Stripped/lowercased; anything except inverse is an error

Be careful with the distinction between helper defaults and constructor defaults: FP8Config.weight_scale_method defaults to the explicit string "row". Supplying only weight_block_size=[128, 128] to the constructor therefore conflicts with row scaling. Set weight_scale_method="block" explicitly, as in the example. A normalized dictionary whose method is omitted can infer block scaling from a block size.

_normalize_fp8_kwargs(payload) handles dictionary/legacy inputs: it recognizes legacy fmt, normalizes the canonical format, block size, and scale method, and extracts legacy weight-only smoothing settings when no modern smoother/smooth field was supplied. Use format=... in new Python calls rather than assuming the dataclass has a fmt constructor parameter.

def _normalize_fp8_kwargs(payload: Dict[str, Any]) -> Dict[str, Any]:

Dynamic layer settings

from gptqmodel.quantization import FP8Config

qcfg = FP8Config(
    format="e4m3",
    weight_scale_method="row",
    dynamic={
        r".*\.mlp\.down_proj$": {"format": "e5m2"},
        r".*\.self_attn\.q_proj$": {"weight_scale_method": "tensor"},
    },
)

Dynamic entries can select per-module format and scale settings. The FP8 dynamic normalizer requires any bits override to represent 8 and any group_size override to be -1 or None. It canonicalizes a dynamic fmt alias into format, validates local block/method combinations and scale semantics, and serializes a provided block size as a list. clone_weight_only_config_for_module() resolves the module-specific configuration before packing; dynamic exclusion can skip modules entirely. Use actual module names from the model definition.

Load and select a backend

from gptqmodel import GPTQModel, BACKEND

model = GPTQModel.load(
    "/path/to/fp8-model",
    device="cuda:0",
    backend=BACKEND.FP8_TORCH,
)

CPU fallback and Apple Silicon are distinct execution choices:

cpu_model = GPTQModel.load(
    "/path/to/fp8-model", device="cpu", backend=BACKEND.FP8_TORCH,
)
mlx_model = GPTQModel.load(
    "/path/to/fp8-model", device="mps", backend=BACKEND.MLX,
)

These are alternatives; do not keep extra model copies resident unless needed.

FP8 weight implementation

Packing

The public module-local weight helper is:

def quantize_fp8_weight(
    weight: torch.Tensor,
    *,
    format: str = "float8_e4m3fn",
    weight_scale_method: str = "row",
    weight_block_size: Optional[Tuple[int, int]] = None,
) -> tuple[torch.Tensor, torch.Tensor]:

It requires a rank-2 matrix, normalizes format and scale layout, converts work to contiguous CPU FP32, computes inverse scales by region, clamps to the dtype's finite range, casts to FP8, and returns (qweight, scale_inv). Scales are FP32. A block-size mismatch raises an error rather than truncating or padding the matrix.

TorchFP8Linear.pack_original() flattens convolution weights and transposes Transformers Conv1D weights into the matrix convention. It applies the optional weight smoother before quantization; smooth_block_size() returns the input block width for block scaling and -1 otherwise. It ignores GPTQ-style scales, zeros, and g_idx arguments, because these are not the FP8 scale source. Bias is stored in FP16. pack(), pack_block(), and pack_gpu() delegate to this same original-weight packer; pack_gpu() is not an independent GPU FP8 weight-quantization algorithm.

On Darwin arm64 with MLX available, supported E4M3/E5M2 matrices containing at least 60 * 1024 * 1024 elements are routed to the native MLX encoder by pack_original(). Smaller matrices use the Torch helper. The bridge returns contiguous CPU Torch FP8 weights and FP32 inverse scales.

TorchFP8Linear contract

The constructor in gptqmodel/nn_modules/qlinear/fp8.py is:

def __init__(
    self,
    bits: int,
    group_size: int,
    sym: bool,
    desc_act: bool,
    in_features: int,
    out_features: int,
    bias: bool = False,
    pack_dtype: torch.dtype = torch.int32,
    adapter: Adapter = None,
    register_buffers: bool = True,
    format: str = "float8_e4m3fn",
    weight_scale_method: str = "row",
    weight_block_size: Optional[Tuple[int, int]] = None,
    weight_scale_semantics: str = "inverse",
    **kwargs,
):

Capability	Source declaration
Backends/methods/formats	FP8_TORCH; METHOD.FP8; FORMAT.FP8
Bits	8
Devices	CPU, CUDA, ROCm, XPU, MPS
Platforms	All
Activation/model dtypes	FP16, BF16, FP32
Feature divisibility	1 in general; block and fast-path restrictions apply separately
Automatic padding	Disabled
Shards	Supported
Training declaration	SUPPORTS_TRAINING=True
Adapters	LoRA
Pack dtype declarations	int8, int16, int32, int64; these do not change FP8 weight storage

The backend deliberately excludes NPU: the source notes that exposing FP8 dtypes on CPU through torch-npu does not establish CANN support for device FP8 tensors. validate_once() requires a PyTorch build exposing FP8 dtypes.

Persistent buffers are FP8 weight, FP32 weight_scale_inv, and optional bias. dequantize_weight(*, device=None, dtype=None) defaults to FP32 and returns the transposed contiguous matrix (in_features, out_features) for multiplication by flattened inputs.

Forward route	Eligibility and behavior
torch._scaled_mm	CUDA device; tensorwise scales only; _scaled_mm present; FP16/BF16 input; matching width; input/output dimensions divisible by 16; not E8M0; not previously disabled
Dequantize + matmul	General fallback; reconstructs for the current multiplication rather than retaining a dense weight cache
Compiled CPU fallback	Explicit optimize(backend="inductor", mode=None, fullgraph=False) on CPU when Torch compilation is available

The _scaled_mm implementation dynamically quantizes the input using the weight's FP8 dtype and converts inverse weight scales to direct scales for the Torch call. Although scale helpers contain rowwise logic, _can_use_scaled_mm() currently permits only tensorwise weight scales. Do not advertise rowwise/blockwise native scaled-mm execution based on the helpers alone.

If _scaled_mm raises, the module disables that route for subsequent forwards and uses dequantize + matmul. Packing resets that disable flag. Compiled CPU errors log a warning, invalidate compilation, and fall back to eager execution. Device/dtype application and state loading invalidate compiled state. After the product, bias and any adapter are applied and the original leading input dimensions are restored.

For older CUDA devices without native FP8 support, dequantization happens on CPU before transferring the target FP16/BF16/FP32 weight for the operation. E8M0 also forces the validated CPU lookup-table path because the source documents problematic CUDA casting for that format. Being able to store FP8 bytes on a GPU does not prove that its FP8 arithmetic is supported or faster.

Native MLX weight encoding and inference

The encoder in gptqmodel/quantization/mlx_fp8.py has this signature:

def quantize_fp8_weight_mlx(
    weight,
    *,
    format="float8_e4m3fn",
    weight_scale_method="row",
    weight_block_size=None,
):

It accepts nonempty rank-2 finite MLX FP16/BF16/FP32 weights, validates the same scale layout, and returns FP8 storage bytes (mx.uint8) plus inverse scales. Its Metal encoder implements finite saturation, nearest-even rounding, signed-zero behavior for the appropriate formats, and special handling of source FP32 subnormals. It does not return an MLX array with a Torch FP8 dtype.

MLX component	Purpose and restrictions
FP8MlxQuantLinear in qlinear/mlx.py	Torch-side checkpoint holder and transfer contract; Darwin/MPS, no adapters/training, holder dtype declaration FP16
MlxFP8Linear in qlinear/mlx_fp8.py	Native MLX QuantizedLinear(group_size=32, bits=8, mode="mxfp8") plus checkpoint output scale
MlxFP8PackedLinear in the same file	General packed-byte Metal runtime using a 256-value decode table and the original inverse-scale layout

Exact native MXFP8 transfer requires E4M3FN, tensorwise or rowwise scales, input width divisible by 32, positive finite inverse scales with finite reciprocals, and no E4M3 NaN codes. The transfer preserves E4M3 bytes, supplies unit MX block scales, and applies the checkpoint reciprocal scale afterward. The unscaled dot product is computed with FP32 inputs so an intermediate FP16 overflow does not corrupt an otherwise finite scaled result.

Other source-compatible FP8 modules use MlxFP8PackedLinear: tensor, row, and block scales are applied in FP32 while decoding bytes for the current multiplication. Its general Metal route accepts FP16/BF16/FP32 activations and returns their dtype. An optimized prefill route is selected for multiple rows, input width at most 8192 and divisible by 64, and output width at least 8192 and divisible by 32. It uses 16-row tiles; the general route handles the other cases. These are execution choices, not checkpoint conversions to dense weights.

The repository's MLX extra currently declares mlx>=0.32.4 and MLX-LM minima selected by Python version (>=0.31.3 below Python 3.11, >=0.32.2 at Python 3.11+). Use the supported Apple Silicon installation described in MLX / Apple Silicon.

Native FP8 and FP4 checkpoint sources

Source detection and fresh quantization

native_floatx_source_format(config, *, model_local_path=None) in models/loader.py inspects source metadata rather than a model-name suffix:

Source metadata	Detection behavior
config.quantization_config	Inspect a dictionary when present
Optional hf_quant_config.json	Read its quantization object, or the sidecar itself; supports older ModelOpt exports lacking model-config metadata
quant_algo, quant_format, format, quant_method	Collect string algorithm identifiers, lowercased
config_groups[*].format and Quark floatx metadata	Include format declarations from those layouts
Qwen native FP8 exports	Recognize quant_method: "fp8" rather than requiring ModelOpt's quant_algo key

Detected families include fp8, nvfp4, mxfp8, mxfp4, nvfp8, and mixed-mx; MiMo mixed sources have a separate detector and mimo_mixed label. Detection precedence distinguishes mixed MX from single-format sources. These are source labels, not additional FP8Config formats that can all use one ordinary FP8 GEMM.

For a fresh METHOD.GPTQ or METHOD.AWQ quantization, configure_native_floatx_source_quantization() enables module-local decoding:

1. Validate the decode dtype on the target device. Ordinary sources use BF16; MiMo's special path follows its supported FP16/BF16 model dtype and rejects AWQ.
2. Keep GPTQ Hessian staging in FP32, overriding a lower staging dtype if necessary.
3. Probe validated native E4M3 or packed-FP4 linear support for eligible source families.
4. Add AutoModuleDecoderConfig when one is not already present.
5. Enable checkpoint-backed offload_to_disk loading so LazyTurtle preserves original source bytes/scales rather than coercing them into the shell model's floating dtype.

The shell stays on meta; one selected module at a time is decoded for its weight solve. A transient _native_floatx_forward_plan records whether Hessian forwards can use native operands or must decode. This plan is not part of the exported GPTQ/AWQ config. MX and other special layouts can use format-specific decoding even when the ordinary FP8 forward wrapper is inapplicable.

import torch
from gptqmodel import GPTQModel
from gptqmodel.quantization import GPTQConfig

# This is a fresh INT4 GPTQ solve from a native FP8 source checkpoint.
target = GPTQConfig(bits=4, group_size=128, device="cuda:0")
model = GPTQModel.load(
    "/path/to/native-fp8-source",
    quantize_config=target,
    dtype=torch.bfloat16,
)
model.quantize([
    "This example document contains several complete sentences about the kinds "
    "of questions and explanations that the model will encounter during deployment.",
    "Use many real documents from the intended domain for an actual run, keeping "
    "the calibration corpus separate from the data used to evaluate quality.",
])
model.save("/path/to/new-gptq-model")

This does not recover the original pre-FP8 weights. The quantizer receives the source checkpoint's reconstructed weights. Requantization and direct inference of that source are different operations. These strings illustrate the API, not a sufficient calibration corpus; dataset preparation removes samples with at most calibration_data_min_length tokens (default 10).

FP8 forward wrapper and scale conversion

BaseQModel._build_fp8_forward_module() handles compatible linear source modules. It prefers weight_scale_inv; otherwise it reads direct weight_scale or scale and takes reciprocals. For a zero direct scale it uses 1 / torch.finfo(torch.float32).tiny as the inverse-scale substitute. It infers tensorwise, rowwise, or blockwise layout from the normalized scale shape and constructs TorchFP8Linear without repacking the source weights.

It returns no ordinary FP8 wrapper for declared MXFP8/NVFP8 formats, non-linear targets, missing scale metadata, or incompatible layouts. Padded/special source layouts can therefore fall back to module-local dense decoding even when the GPU supports FP8. The per-module checks are necessary in addition to the device probe.

TorchFP4Linear and native NVFP4 weights

gptqmodel/nn_modules/qlinear/fp4.py defines:

def __init__(
    self,
    *,
    in_features: int,
    out_features: int,
    weight: torch.Tensor,
    weight_scale: torch.Tensor,
    weight_block_size: int,
    orig_dtype: torch.dtype,
    bias: Optional[torch.Tensor] = None,
) -> None:

TorchFP4Linear registers packed weight, weight_scale, and optional bias. On CPU with CPU weights, it decodes the live E2M1 buffers through dequantize_f4_e2m1() for the current F.linear() call without retaining a dense weight. On other devices it constructs a TorchAO NVFP4Tensor view and calls F.linear(). That route requires importable TorchAO NVFP4 support and compatible device/runtime APIs; the source path is torchao.prototype.mx_formats.nvfp4_tensor.

This class is a forward wrapper for already-packed weights, not a generic FP4Config quantizer and not the W4A-NVFP4 INT4-weight backend. BitsAndBytes' fp4 format is another distinct representation; see BitsAndBytes.

ModelOpt activation metadata limitation

Direct loading of a ModelOpt checkpoint with activation quantization metadata is rejected by _configure_modelopt_runtime(). The error explicitly distinguishes supported dense-activation weight-only variants such as W8A16/FP8 and W4A16/FP4 from unsupported ModelOpt activation-quantized runtime execution.

A ModelOpt FP8/NVFP4 checkpoint can still be a source for fresh weight-only GPTQ/AWQ quantization when the auto-module decoder is active; its original activation metadata is not used as the runtime policy. This exception does not mean ModelOpt activation quantization and GPT-QModel's own activation stream are interchangeable.

GPTQ W4A-FP8 and W4A-NVFP4

Configure an activation stream

from gptqmodel import GPTQModel
from gptqmodel.quantization import GPTQConfig

qcfg = GPTQConfig(
    bits=4,
    group_size=128,
    sym=True,
    desc_act=False,
    lm_head=False,
    activation="w4afp8",
    device="cuda:0",
)
model = GPTQModel.load("/path/to/supported-llama-model", quantize_config=qcfg)
model.quantize([
    "This example document contains several complete sentences about the kinds "
    "of questions and explanations that the model will encounter during deployment.",
    "Use many real documents from the intended domain for an actual run, keeping "
    "the calibration corpus separate from the data used to evaluate quality.",
])
model.save("/path/to/gptq-w4afp8-model")

Unlike FP8 weight-only packing, this is a calibrated GPTQ solve with activation policy replay and a saved encoded-stream contract. Replace the illustrative strings with a representative corpus whose tokenized samples survive the minimum-length filter. Loading the saved model with AUTO binds the matching W4A backend through backend_for_activation(). Explicitly requesting a different backend raises an error rather than silently discarding the activation policy. An ordinary GPTQ checkpoint without this policy is not automatically assigned W4A execution merely because the backend supports GPTQ packing.

For an explicit FP8-attention / NVFP4-MLP split:

from gptqmodel.quantization import GPTQConfig

qcfg = GPTQConfig(
    bits=4,
    group_size=128,
    sym=True,
    desc_act=False,
    lm_head=False,
    device="cuda:0",
    activation={
        "mode": "w4a_nvfp4",
        "recipe": "least_squares",
        "attention": {"mode": "w4afp8"},
        # Optional promotion of selected MLP layers to FP8:
        "mlp": {"mode": "w4afp8", "layers": [0, 1]},
    },
)

Use layer indices that exist in the target model. The example promotes the first two MLP layers; it is not a universal accuracy prescription. Without an attention override, the installer uses the stream mode for attention too. The default ActivationAttentionConfig.mode="w4afp8" does not imply that the optional object is automatically created for every NVFP4 stream.

Configuration and model restrictions

Constraint	Required value/behavior
Method and checkpoint family	GPTQ; FORMAT.GPTQ or FORMAT.GPTQ_V2
Runtime zero-point convention	REQUIRES_FORMAT_V2=True; GPTQ-v1 serialized zero points are converted by the loader before post-initialization
Weight layout	4-bit, symmetric, group size 128, desc_act=False
Packing	torch.int32
Head	lm_head=False
Dynamic weight-layout overrides	Overrides of bits, sym, group_size, or desc_act are rejected; exclusion remains possible subject to stream completeness
Stream model support	Current installer requires a Llama decoder (LlamaDecoderLayer), not every model with similarly named projections
Selected decoder layer	All seven attention/MLP projections must be W4A modules
Projection outputs for stream transport	Width divisible by 128; stricter than the individual linear class's divisor 32
Reinstallation	A different policy cannot silently replace an installed stream

The seven projections are self_attn.q_proj, k_proj, v_proj, o_proj, and mlp.gate_proj, up_proj, down_proj. Partial conversion of these within a selected layer is rejected. Shared encoded operands must carry consistent scale and rotation contracts. See Model Definitions and Module Tree and Rotation, Smoothing, and Norm Tuning.

Hardware and dependency gates

Requirement	W4A-FP8	W4A-NVFP4
Backend	GPTQ_W4AFP8	GPTQ_W4A_NVFP4
Class	W4AFP8Linear	W4ANVFP4Linear
Minimum compute capability checked	(8, 9) — Ada or newer FP8 tensor cores	(12, 1) — GB10 / SM12.1 floor in this release
Device/platform declaration	CUDA / Linux	CUDA / Linux
Input/model dtypes	FP16, BF16	FP16, BF16
Matrix divisors in class	Input 128, output 32	Input 128, output 32
Optional APIs	Triton; the grouped FP8 GEMM needs the Gluon mma_v2 import (first shipped in Triton 3.6.0)	Triton (the packer is a plain tl.jit kernel — no Gluon); torch.float4_e2m1fn_x2; F.scaled_mm, F.ScalingType, F.SwizzleType
Training/adapters/auto-padding	Not supported	Not supported
Checkpoint shards	Supported	Supported

Do not summarize NVFP4 eligibility as “all Blackwell” or “all SM12x.” The implementation's minimum is 12.1, and API/driver validation still applies. The FP8 lane of a mixed recipe being Ada-compatible does not make its NVFP4 lane Ada-compatible. Older comments in generic backend declarations are less specific than the executable class validation.

The repository currently requires torch>=2.14.1 and torchao>=0.18.0. The W4A Triton helpers import on any Triton build: w4a_triton imports the Gluon pieces lazily, and only the grouped FP8 GEMM is written in Gluon — the token quantizer and the NVFP4 packer are plain tl.jit kernels. On a Triton without mma_v2 (the API first appears in Triton 3.6.0), activation packing still works, but fp8_linear_prepacked raises native_fp8_gemm_error(), which names the required and installed versions. A mixed NVFP4 recipe that steps the attention lane down to FP8 reports the same floor when it runs; pure NVFP4 does not need Gluon. Successful installation of the base package does not replace the per-backend API and capability checks.

ActivationConfig schema

ActivationConfig.from_value(value) returns None for None, returns an existing ActivationConfig unchanged, and constructs an ActivationConfig from a mode string or dictionary. "w4a_nvfp4" supplies recipe="least_squares". The enclosing quantization config stores the validated policy as a canonical dictionary using to_dict().

Field	Default	Meaning
mode	"w4afp8" on the dataclass	w4afp8 or w4a_nvfp4; required in a dictionary
recipe	None; resolves to least_squares for NVFP4	NVFP4 block-scale recipe; forbidden for FP8
attention	None	Optional attention-group policy; only refines an NVFP4 stream
mlp	None	Optional per-layer FP8 MLP promotion; only refines an NVFP4 stream
global_scales	None	Calibrated producer-to-FP32-scale mapping for NVFP4

An attention dictionary permits only mode and recipe. FP8 attention forbids an NVFP4 recipe; NVFP4 attention inherits the stream recipe when omitted. An MLP dictionary requires mode="w4afp8" and a nonempty list/tuple of unique, nonnegative integer layer indices; booleans and duplicates are rejected and indices are sorted. The stream installer separately checks decoder depth. Unknown fields are rejected instead of being ignored.

global_scales requires an NVFP4 stream and cannot be combined with the legacy headroom recipes. It must be a nonempty dictionary with string keys starting with model.layers. and positive finite numeric values; booleans are rejected. Each value is rounded through FP32 and must remain positive and finite after that conversion. The stream installer checks exact producer coverage against the active attention/MLP policy, so an arbitrary label with the right prefix is not sufficient.

The stream installer signature is:

def install_w4a_llama_stream(model: torch.nn.Module, mode: str,
                             recipe: str | None = None,
                             global_scales: dict[str, float] | None = None,
                             attention_mode: str | None = None,
                             attention_recipe: str | None = None,
                             mlp_fp8_layers: tuple[int, ...] | None = None,
                             fused_norms: bool = False) -> None:

This is an internal installation entry point in w4a_llama_stream.py, not a replacement for the saved GPTQConfig.activation policy. Its default attention_mode=None resolves to the stream mode; a mixed policy must request FP8 attention explicitly. fused_norms defaults to False.

Activation mathematics and kernels

FP8 per-token policy

activation_floatx.py defines FP8_E4M3_MAX = 448.0 and:

def fp8_token_qdq(x: torch.Tensor) -> torch.Tensor:

For each last-axis input row it computes amax, uses direct scale amax / 448 (or 1 for an all-zero row), rounds/clamps to torch.float8_e4m3fn, and decodes back to the original dtype. Empty inputs return unchanged; non-finite input values raise an error. This is the Torch quantize/dequantize policy used for replay/reference behavior, not the persistent encoded carrier itself.

w4a_triton.py supplies fp8_pack(), fp8_linear(), and fp8_linear_prepacked(). Packing produces E4M3 codes and one FP32 direct scale per token. The grouped MMA kernel multiplies E4M3 input codes by centered INT4 weight codes represented exactly in E4M3, accumulates each 128-input group in FP32, applies GPTQ group scales, then token scales and bias.

_to_e4m3_rn() emits direct nearest-even cvt.rn.satfinite.e4m3x2.f32 conversion. The source explains that a plain Triton cast on SM8.9 can double-round through truncated FP16; the explicit conversion avoids architecture-dependent code/scale bytes. The launch-device guard selects the operands' CUDA device rather than assuming the process's current device matches a sharded layer.

NVFP4 values and two-level scaling

E2M1 magnitudes are 0, 0.5, 1, 1.5, 2, 3, 4, 6, with a sign bit. The reference uses nearest-even rounding and finite saturation. Each 16-element activation block has an E4M3 local scale; one FP32 global scale is shared by the operand tensor:

decoded_block = e2m1_values * local_e4m3_scale.float() * global_scale

The two principal reference signatures are:

def nvfp4_global_scale(amax: torch.Tensor | float, *, grid_dtype: torch.dtype | None = None,
                       recipe: str = "least_squares") -> torch.Tensor:

def nvfp4_block_qdq(x: torch.Tensor, global_scale: torch.Tensor | float,
                    recipe: str = "least_squares") -> torch.Tensor:

nvfp4_global_scale() requires finite, nonnegative maxima. For a nonzero maximum, it divides by 448 * 6 for NVIDIA/headroom recipes and by 448 * 4 for the other optimized recipes; zero maxima use scale 1. An optional source-dtype grid may be FP16, BF16, or FP32. If rounding to that grid would underflow the scale to zero, the positive FP32 scale is retained.

nvfp4_block_qdq() requires width divisible by 16, finite values, and a positive finite global scale. It returns the original dtype after blockwise rounding. The runtime packer has the stricter width divisor 128 required by GPTQ grouping and scale swizzling.

NVFP4 recipes

Recipe	Local block algorithm	Global/calibration distinction
nvidia	Max-scaled E2M1 using maximum 6	Full E2M1 range
nvidia_headroom	Same local algorithm as nvidia	Legacy calibrated activation-headroom policy
four_six	Evaluate candidates using maxima 4 and 6, keep the lower block SSE	Compatibility recipe retained for older checkpoints
least_squares	Refine both 4/6 assignments with least-squares scales rounded back to real E4M3, then refine the winning assignment	Default recipe
least_squares_headroom	Same local refinement as least_squares	Legacy activation-headroom policy
least_squares_grid	Least-squares refinement plus bounded E4M3-bit-grid search	Search radius 8 around the selected positive finite E4M3 encoding

Legacy strings lsq, lsq_headroom, and lsq_grid normalize to their least_squares* names. These are bounded scale-fitting recipes, not learned-scale QAT. The reference clamps positive local scales to the E4M3 hardware range from 2**-9 through 448 and retains refinements only when true block squared error improves.

NVFP4ActivationHeadroom collects a bounded histogram of maxima of 16-value blocks rather than retaining activation tensors. Its defaults are anchor_percentile=1.0, upper_percentile=99.99, rho=16384.0, num_bins=512, log2_min=-40.0, and log2_max=40.0. It validates percentiles in (0,100] and rho in (0,28672). It tracks the running maximum, rejects non-finite input, and requires nonzero observed blocks before computing a calibrated maximum. If requested headroom does not exceed the calibrated upper percentile it warns and uses the upper value.

Runtime staging and exact INT4 representation

W4AFP8Linear.post_init() reads native GPTQ packing, checks contiguous g_idx groups and symmetric zero point 8, and stages centered INT4 codes as _weight_e4m3. All centered INT4 integers are exactly representable in this E4M3 operand; GPTQ scales remain separate. pack() records logical zero-point format 2; GPTQ-v1 serialization conversion is handled separately and reversed by the loader.

W4ANVFP4Linear cannot represent every centered INT4 integer with one E2M1 value. It builds two exact integer planes:

high = floor((centered + 2.0) / 4.0)
low = centered - 4.0 * high
centered = low + 4.0 * high

The two planes are E2M1-encoded and packed along K, concatenated into _weight_both, and multiplied with unit E4M3 weight scales. A separate _weight_e4m3 operand supports mixed FP8 attention over those same INT4 weights. These are runtime staging buffers, marked non-persistent; the saved weights remain native GPTQ INT4. Do not equate the 4-bit checkpoint size with resident runtime memory: the inspected implementation keeps these staged operands in addition to GPTQ buffers. They are not an additional serialized weight format.

Here persistent=False only excludes the operands from state_dict(); it does not make them per-forward temporaries. In this revision, _weight_e4m3 retains a full-layer INT4-to-E4M3 expansion between calls. This is an existing runtime-storage limitation, not compliance with the repository's packed-only persistent-weight rule. The two packed E2M1 planes are exact alternate operands, not a lossy replacement of the saved INT4 weights.

activation_global_scale_bits stores the FP32 calibration scalar as exact INT32 bits so generic model-dtype casting cannot change it. The property activation_global_scale views those bits as FP32. Post-initialization requires it to be positive and finite. Device/dtype application preserves staged hardware operand bytes rather than casting FP8/FP4 codes into the model precision; packing/loading invalidate staging and require appropriate post_init() rebuilding.

For each 128-input GPTQ group, NVFP4 forward calls F.scaled_mm() with ScalingType.BlockWise1x16 and SwizzleType.SWIZZLE_32_4_4. The accumulation kernel recombines low/high products, multiplies the GPTQ group scale, accumulates in FP32, and applies the activation global scale, optional token rescaling, and bias before output conversion.

Triton packing and encoded transport

File/component	Responsibility
w4a_nvfp4_triton.py	E2M1 packing, recipe-specific scale search, fused scale swizzling, decode at nonlinear consumers, and low/high-plane accumulation
nvfp4_pack_and_swizzle(x, global_scale, *, recipe="least_squares")	CUDA rank-2 input with K divisible by 128; returns packed E2M1 and all per-group swizzled E4M3 scales
nvfp4_decode(...)	Validate packed/scales layout and reconstruct an activation at a consumer requiring ordinary arithmetic
nvfp4_accumulate_group(...)	Fuse two exact weight-plane products with the group/global/token scales
W4AActivation in w4a_activation.py	Scale-aware carrier for mode, codes, scales, shape, model dtype, optional global scale/recipe and rotation state
w4a_llama_replay.py	Calibration replay policy before final encoded-stream installation
w4a_llama_stream.py, w4a_boundary.py	Llama boundary policies and runtime transport through decoder operations

For an input (rows, K), the NVFP4 packer returns FP4 storage shaped (rows, K // 2) and scales shaped (K // 128, padded_rows * 8), where padded_rows rounds rows up to a multiple of 128. Eight block-16 scales belong to each 128-input group. Scale padding/swizzling is a kernel operand layout, not padding of the logical model width.

For rows=0, it returns shapes (0, K // 2) and (K // 128, 0) without a kernel launch. The low-level packer validates rank, CUDA placement, K divisibility, and recipe; unlike the Torch QDQ reference, it does not itself scan the input/global scale for non-finite values. Do not treat successful packing as a finite-value validation step.

An encoded operand is not just a low-precision tensor: losing its token/block/global scales changes the computation. W4A linears validate mode, width, device, and rotation state. Where the installed stream requires encoded input, a plain tensor raises TypeError. Rotated operands cannot be reused across incompatible linear contracts; the runtime may decode, rotate, and repack when needed. Mixed W4A-NVFP4 linears can consume FP8 carriers for the attention lane without changing the checkpoint weights.

Calibrate producer scales after loading

The model wrapper exposes:

def calibrate_activations(self, samples: List[torch.Tensor], *, progress=None) -> Dict[str, Any]:

This is a separate calibration operation for an already-quantized, installed NVFP4 stream, not another GPTQ weight solve. The model must be in evaluation mode. Samples are unpadded, nonempty one-dimensional token-ID tensors; capture must run without KV cache. Calibration data must be disjoint from downstream evaluation data. The implementation updates runtime producer scales and the configuration subsequently used by save().

Producer calibration rejects legacy headroom recipes and requires the installed policy and saved quantization config to match. Shared boundaries and all required NVFP4 producers must be valid; arbitrary partial producer maps are not a substitute for this procedure. See Evaluation for keeping calibration and evaluation separate.

# `model` is an already-quantized model loaded with an NVFP4 activation policy.
# Use unpadded token-ID vectors from a calibration-only corpus.
model.model.eval()
report = model.calibrate_activations(calibration_token_vectors)
model.save("/path/to/calibrated-gptq-nvfp4-model")

The returned dictionary contains:

Report field	Value/meaning
algorithm	"producer_maximum"
recipe	Stream-level NVFP4 recipe
global_scales	Producer-to-scale mapping
boundaries	Per-producer measurements listed below
upstream_activations	"calibrated_encoded_carriers"
weight_updates	0

Each entry in boundaries contains:

Producer report field	Meaning
recipe	Effective attention/MLP recipe for that producer
amax	Maximum observed absolute producer value
global_scale	Frozen positive FP32 global decoding scale
tokens	Number of observed token rows
samples	Number of observations at that producer

The implementation calibrates producers in topological order using maximum calibration and each producer's runtime block-scale recipe. It caches one layer's sample state on CPU, then propagates calibrated carriers into the next layer. On failure it restores prior producer scales and policy/config metadata; the procedure does not update GPTQ weights.

GSQ refinement

FP8Config.gsq refines FP8 weight export on finite E4M3/E5M2 storage — float8_e8m0fnu is rejected — with fixed inverse scales, so learn_scales is unsupported. The default mode is weight-only; gsq_calibration=True instead collects calibration statistics through a dedicated looper processor and requires weight_scale_method "row" or "block" plus calibration data. See GSQ.

Pitfalls and edge cases

Symptom or assumption	Explanation/action
FP8Config(bits=4)	Invalid; FP8 weight config requires 8. Use the appropriate native FP4 source or GPTQ activation policy
FP8 format accepts E8M0, but packing rejects it	Recognition/dequantization support is broader than dense-weight quantization support
Block size supplied with default row scaling	Set weight_scale_method="block" explicitly
FP8 block dimensions do not divide a selected matrix	No automatic padding; choose compatible blocks or module-specific settings
Native tensor-core throughput expected from all FP8_TORCH loads	The scaled-mm gate is tensorwise CUDA only; fallback may decode and matmul
Inverse and direct scales mixed	FP8 weight decoding divides by inverse scales; source direct scales and activation scales multiply
FP8 bytes on an old GPU	Storage support is not native arithmetic support; CPU decoding can be used
W4A-FP8 backend assumed SM12x-only	The executable FP8 gate is SM8.9+; NVFP4 is the stricter SM12.1+ path
SM12.0 assumed sufficient for W4A-NVFP4	The current minimum check is (12,1) plus required Torch FP4 APIs
Generic model assumed supported by W4A	Current full stream requires Llama decoder layers and all seven projections
Individual linear validates output divisor 32, stream still rejects it	Encoded stream transport requires selected projection output widths divisible by 128
Bare w4a_nvfp4 assumed to mean FP8 attention automatically	Request attention={"mode":"w4afp8"} explicitly for the mixed split
W4A checkpoint assumed usable through any GPTQ backend	Saved activation policy binds its matching backend; incompatible explicit backends error
ModelOpt activation-quantized artifact treated as the same stream	Direct execution is rejected; fresh source requantization is a different supported path
W4A forward says weight cache absent	Load/pack native GPTQ tensors and complete post_init(); do not treat staged operands as serialized tensors
Dense CPU FP4 fallback assumed to require TorchAO	CPU decoding uses the local dtype utility; the non-CPU NVFP4Tensor route needs TorchAO
Runtime memory estimated only from serialized INT4 size	Account for the W4A staged E4M3/FP4 operands and activation scales/workspaces

Source files

• gptqmodel/quantization/config.py — FP8 enums/config/normalization; activation schema, validation, and serialization.
• gptqmodel/quantization/dtype.py — FP8/FP4 decoding, block inference, and device/dtype capability maps.
• gptqmodel/looper/weight_only_looper.py, weight_only_processor.py — FP8 weight-only lifecycle.
• gptqmodel/nn_modules/qlinear/fp8.py — Torch FP8 packing, buffers, fast path, fallback, and CPU compilation.
• gptqmodel/nn_modules/qlinear/fp4.py — native packed FP4 forward wrapper.
• gptqmodel/quantization/mlx_fp8.py — native Metal weight encoder.
• gptqmodel/nn_modules/qlinear/mlx_fp8.py, mlx.py — native/general packed FP8 MLX inference and transfer contract.
• gptqmodel/utils/mlx.py — whole-model MLX conversion.
• gptqmodel/quantization/activation_floatx.py — FP8/NVFP4 QDQ references, recipes, global scales, headroom calibration.
• gptqmodel/nn_modules/qlinear/w4a_floatx.py — GPTQ W4A-FP8 backend, validation, and operand staging.
• gptqmodel/nn_modules/qlinear/w4a_triton.py — token FP8 packing and grouped tensor-core products.
• gptqmodel/nn_modules/qlinear/w4a_nvfp4.py — exact two-plane INT4 staging and native block-scaled FP4 forward.
• gptqmodel/nn_modules/qlinear/w4a_nvfp4_triton.py — NVFP4 pack/swizzle/decode/accumulation.
• gptqmodel/nn_modules/qlinear/w4a_activation.py, w4a_boundary.py, w4a_llama_stream.py, w4a_llama_replay.py — scale-aware carriers, supported-model boundaries, and replay.
• gptqmodel/quantization/activation_calibration.py — loaded-stream producer calibration.
• gptqmodel/models/loader.py, base.py — native-source detection, decoder/native plans, forward wrappers, ModelOpt restrictions, and public entry points.
• gptqmodel/utils/backend.py — backend identifiers and activation-to-backend binding.
• gptqmodel/models/writer.py — checkpoint/config serialization.
• requirements.txt, pyproject.toml — current dependencies and MLX extras.

See also

• What Is Quantization?
• QuantizeConfig Reference
• GPTQ
• AWQ
• GSQ
• BitsAndBytes
• Packing Formats and Compatibility
• Kernel Backends Reference
• Loading and Inference
• MLX / Apple Silicon
• Quantization Lifecycle
• Rotation, Smoothing, and Norm Tuning
• Evaluation
• Troubleshooting and FAQ