What Is Quantization

What is Quantization?

Documentation home · Quickstart · Lifecycle

On this page

• Overview
• Workflow at a glance
• Concepts
• Usage and configuration examples
• Method, format, and backend are three different axes
• Implementation internals
• Pitfalls and edge cases
• Source files
• See also

Overview

Quantization represents numerical values with a smaller set of possible values than the original floating-point representation. For an LLM, the most common starting point is to replace FP16/BF16 linear-layer weights with low-bit integer codes plus the metadata needed to reconstruct their approximate values. The model still performs the same language-modeling task; the numerical representation and the implementation of some operations change.

This page separates general quantization concepts from GPT-QModel's configuration and runtime contracts. The implementation descriptions and defaults below refer to GPT-QModel 7.7.0, checked against source revision c1976918cfe3703fbcf6d1ce2440a82b26394375. Older checkpoints, older releases, and external inference engines can have different defaults or supported combinations.

Start with Installation and Quickstart for a complete runnable workflow, or QuantizeConfig Reference for the complete configuration surface.

Why quantize an LLM?

The principal benefits are reduced weight storage and reduced memory traffic. Hardware-specific kernels can also accelerate the matrix multiplications that consume the quantized weights. These benefits are related but not interchangeable:

Goal	What quantization can change	What it does not guarantee
Fit a model in memory	Fewer bits per stored weight	That activations, KV cache, temporary buffers, or unquantized modules also shrink
Reduce checkpoint size	Packed weights replace dense weight tensors	Exactly a 4× smaller whole checkpoint for a 16-to-4-bit recipe
Improve decode latency	Less weight bandwidth; specialized kernels	Speedup on every device, batch size, and sequence length
Improve prefill throughput	Low-precision arithmetic and kernel fusion where supported	That a weight-only kernel uses low-bit activation arithmetic
Preserve quality at lower cost	Calibration and error compensation can limit degradation	Numerically identical logits or unchanged task accuracy

Ignoring metadata, N weights at b bits require N * b / 8 bytes. Reducing a weight tensor from 16 to 4 bits gives a theoretical 4× reduction for that tensor. Scales, zero points, indices, codebooks, padding, biases, and unquantized tensors add storage. Runtime memory additionally includes activations, KV cache, working buffers, and allocator overhead. Quantization itself can require substantially more memory than inference because the source weights and calibration intermediates must be processed.

Workflow at a glance

Reading the diagram: Weight representation is an independent concern; the two activation lanes are alternatives, not sequential stages.

Text version of the workflow

• Weight representation: Dense weights → Quantization method → Compact model state.
• Typical GPTQ or AWQ: Floating-point inputs → Packed-weight compute → Floating-point outputs.
• Explicit GPTQ W4A policy: Floating-point inputs → Low-bit operands → Specialized compute.

Open full-size SVG

Concepts

Post-training quantization versus quantization-aware training

Approach	When quantization is introduced	Typical inputs	Relationship to GPT-QModel
Post-training quantization (PTQ)	After the model has been trained	Trained weights; optionally representative calibration examples	The normal load → quantize → save workflow. GPTQ and AWQ use calibration; RTN, native GGUF, FP8-weight, and BitsAndBytes-weight lifecycles can quantize without it.
Quantization-aware training (QAT)	During training or fine-tuning	Training data, loss, optimizer, and simulated or real quantization	A distinct training procedure, not something enabled by bits=4 or desc_act=True. Training-capable backends and experimental optimization tools are not a guarantee that every backend supports QAT.

PTQ does not necessarily mean “round each weight independently.” GPTQ uses calibration-derived information to compensate quantization error. AWQ performs activation-aware scale/clip search. ParoQuant includes learned-rotation optimization. Conversely, a calibration-free round-to-nearest recipe has fewer steps but can have different quality. For the individual algorithms, see GPTQ, AWQ, and ParoQuant.

There are also post-quantization quality-recovery approaches, including adapters and normalization tuning. They should not be confused with a change in the stored weight datatype; see EoRA and Rotation, Smoothing, and Norm Tuning.

Weight-only versus weight-plus-activation quantization

Weights are learned model parameters. Activations are intermediate tensors produced while processing a prompt or generated token. Weight quantization changes the parameter representation. Activation quantization changes some of the intermediate representation or the operands used inside a compute kernel.

An algorithm can use activations during calibration without quantizing them at inference. In particular:

• Ordinary GPTQ and AWQ recipes commonly keep floating-point model activations.
• desc_act means activation ordering, not activation quantization.
• AWQ's “activation-aware” name describes how weight quantization is chosen, not a promise of INT4 activations.
• A native FP8 weight checkpoint is not automatically W8A8.
• A kernel can accept and return FP16/BF16 tensors while quantizing an internal matrix-multiplication operand to INT8 or FP4.

The method uses_weight_only_lifecycle() has a narrower implementation meaning: it selects a lifecycle that does not need activation calibration. GPTQ is normally called weight-only quantization in the inference-precision sense, but its configuration returns False from this lifecycle query because GPTQ collects calibration activations. Use requires_calibration_dataset() rather than inferring calibration requirements from the phrase “weight-only.”

The base implementation in gptqmodel/quantization/config.py is:

def uses_weight_only_lifecycle(self) -> bool:
    return False

def requires_calibration_dataset(self) -> bool:
    return not self.uses_weight_only_lifecycle()

Calibration-free configs override uses_weight_only_lifecycle() to return True. This describes their weight-quantization lifecycle; optional preprocessing can have additional input requirements.

What W4A16 means

WbAa describes a weight precision of b bits and an activation precision of a bits for the operations being described. It is shorthand, not a complete checkpoint or hardware specification.

Label	Literal interpretation	Important qualification
W4A16	4-bit weights, 16-bit activations	Usually INT4 weights with FP16 or BF16 activations in the ordinary GPTQ/AWQ workflow; not 4-bit activations
W4A8	4-bit weights, 8-bit activations	The 8-bit operand can be INT8 or FP8; the label alone does not say which. QQQ dynamically uses INT8 activation operands, while the experimental GPTQ w4afp8 policy uses FP8.
W8A8	8-bit weights, 8-bit activations	Does not distinguish INT8 from FP8, scale granularity, or accumulation type. FP8Config alone is not a W8A8 switch.
W4A4	4-bit weights, 4-bit activations	Could describe INT4 or floating-point activation codes. GPT-QModel's experimental w4a_nvfp4 policy keeps INT4 GPTQ weights and uses NVFP4 activation transport.

For example, a GPTQ W4A16 checkpoint stores packed 4-bit weight codes and associated scales/zeros. Its linear operation consumes 16-bit floating-point activations. It does not mean that every tensor in the model occupies 4 or 16 bits: embeddings, normalization, biases, output layers, attention state, accumulator precision, and KV-cache policy are separate concerns. The normal configuration has lm_head=False.

For ordinary GPTQ, bits=4 chooses the weight width and the model-loading dtype selects the model floating-point dtype. dtype=torch.float16 and dtype=torch.bfloat16 are both 16-bit choices, but not interchangeable for every backend or device. Neither changes bits into an activation-width field.

Integer and floating-point low-bit representations

Representation	What its codes mean	GPT-QModel connection
INT4	A 4-bit integer code: 16 possible codes. Signedness and zero offset depend on the format.	GPTQ/AWQ commonly use integer codes with group scales and zero points. Four bits are packed into integer storage words; they are not independent FP16 values.
INT8	An 8-bit integer code: 256 possible codes.	Integer weight recipes can use bits=8; QQQ uses dynamic INT8 activation operands internally. A particular kernel can support only a subset of these uses.
FP8	An 8-bit floating-point encoding with sign/exponent/mantissa conventions; not uniformly spaced like an affine integer grid.	FP8Config supports format names such as float8_e4m3fn and float8_e5m2, plus scaled weight storage. The experimental w4afp8 activation policy is separate.
NVFP4	E2M1 4-bit floating-point values with block scaling; more than the 4-bit payload is needed to describe a scaled tensor.	The W4A activation policy uses an E4M3 scale per 16 values plus an FP32 global scale. The GGUF NVFP4 weight layout is a separate serialization contract.
MXFP4	E2M1 4-bit floating-point values with microscaling, using an E8M0 shared scale for a 32-value block.	Supported as a GGUF quantization type. It is not the same layout as GPTQ INT4 or the w4a_nvfp4 activation stream.

FP8's E4M3 and E5M2 names describe exponent and mantissa bit allocations. More exponent bits and more mantissa bits trade dynamic range against local precision. The suffix and scale convention also matter. FP8Config recognizes float8_e4m3fn, float8_e5m2, float8_e4m3fnuz, float8_e5m2fnuz, and float8_e8m0fnu; recognizing a format name does not prove that every backend has native arithmetic for it.

NF4 and BitsAndBytes FP4 are additional 4-bit representations. Do not treat NF4, BitsAndBytes FP4, NVFP4, MXFP4, and affine INT4 as interchangeable because they all have “4” in their names. See BitsAndBytes, FP8 and NVFP4, and GGUF.

Scales, zero points, and affine dequantization

For a conventional unsigned b-bit affine integer representation, let q be the stored code, s > 0 the scale, and z the zero point. A simple quantize/dequantize pair is:

$$q=\operatorname{clip}(\operatorname{round}(w/s)+z,0,2^b-1), \qquad \hat w=s(q-z).$$

The reconstructed weight is approximate because rounding discards information and clipping discards out-of-range magnitude. The zero point is the integer code representing real zero; it is not the minimum real value. For example, with s=0.1, z=8, and stored code q=11, the reconstructed value is 0.3.

This is the unsigned path in gptqmodel/quantization/quantizer.py. Some methods use a signed formulation, for which the code is clamped around zero and dequantization is simply s * q. FP4/FP8 encodings and vector codebooks use different decoding rules; the affine formula is not a universal decoder.

Scheme	How the range is chosen	Zero-point consequence
Symmetric	Positive and negative magnitudes share a range policy, subject to the representable code grid	The signed formulation can use zero point 0; unsigned storage can use a fixed midpoint. In GPTQ's ordinary unsigned path, the midpoint is (maxq + 1) / 2, so 4-bit symmetric weights use zero point 8, not 0. The range finder mirrors the negative bound only when negative values are present.
Asymmetric	Positive and negative bounds can differ	The affine zero point is chosen from the observed range, typically rounded from -w_min / s. It must be applied correctly during decoding.

In GPT-QModel, sym=True requests symmetric quantization; sym=False requests asymmetric quantization where the method/backend supports it. The checkpoint parser treats the legacy zero_point flag as the negation of sym. That flag means “enable an asymmetric zero point,” not “this is the numeric zero-point value.”

Checkpoint layouts can encode zero points differently. In particular, GPTQ/GPTQ_V2 compatibility conversion must be respected. Never infer the mathematical zero point from raw packed qzeros without identifying the format and its conversion path.

Granularity: which values share quantization parameters?

The fewer weights share a scale, the better the representation can adapt to local ranges, at the cost of more metadata and sometimes more complicated kernels.

Granularity	Shared parameters	Typical trade-off
Per-tensor	One scale/zero for an entire tensor	Small metadata cost; a few outliers can set the range for all values
Per-channel	One parameter set per channel, with the axis specified	Better local adaptation; the channel axis must be agreed upon by producer and consumer
Groupwise	Parameter sets for fixed-size groups along an axis	A balance between fidelity, metadata, and implementation complexity
Blockwise	Parameter sets for blocks defined by the representation, potentially two-dimensional	Useful for floating-point microscaling and format-specific block encodings; not necessarily GPTQ's group_size

For a linear-layer weight matrix with shape [out_features, in_features], ordinary GPTQ groups input-feature columns within each output channel. With group_size=128, each output channel has a scale/zero per group of 128 input weights. With group_size=-1, the group spans the input dimension: there is still per-output-channel quantization, not one scale for the entire matrix. Reordering can require a g_idx mapping from original input positions to groups.

For fresh GPTQ quantization with group_size=-1, explicitly set act_group_aware=False. Config normalization accepts -1, but GPTQ._validate_act_group_aware_shape() requires a positive group size when group-aware ordering is enabled. Ordinary GPTQ enables that ordering by default:

from gptqmodel.quantization import GPTQConfig

per_channel_config = GPTQConfig(group_size=-1, act_group_aware=False)

Smaller groups increase parameter metadata relative to weight payload. A full group of 128 INT4 weights contains 512 payload bits before its scale/zero and other overhead. Changing the group size changes both quality and kernel compatibility; it is not a free accuracy knob.

For native FP8 weights, use weight_scale_method="tensor", "row", or "block". "block" requires weight_block_size. FP8 does not reuse GPTQ's group_size mechanism; its configuration keeps group_size=-1. GGUF similarly defines its own block layout. VPTQ's codebook grouping is method-specific.

Static versus dynamic activation scales

Scale policy	When scales are determined	Practical consequence
Static	From calibration, then stored/reused at inference	Avoids re-estimating that scale on every input; calibration coverage and outliers matter
Dynamic	From the current activation tensor, token, row, or block	Adapts to current inputs, with runtime scale-computation/packing cost

Static/dynamic describes a scale policy, not whether the model weights change. A recipe can combine static and dynamic components. GPT-QModel's w4afp8 transport carries one dynamic FP32 activation scale per token. w4a_nvfp4 carries E2M1 codes with per-16 E4M3 block scales and an FP32 global scale; the global scale can be calibrated, while block packing is performed for runtime activations. QQQ dynamically quantizes activation operands inside the linear operation.

Neither static_groups in GPTQ nor the dynamic per-module configuration dictionary is a general activation-scale toggle. They concern weight grouping and module configuration overrides, respectively.

Calibration data

Calibration examples are representative inputs used to observe the model's internal behavior. They are not generated outputs and do not require supervised labels for the ordinary GPTQ/AWQ workflow. GPTQ uses activation-derived Hessian information; AWQ uses activations for scale and clipping choices. Methods with optimization stages can have additional calibration requirements.

Good calibration covers the intended languages, domains, prompt lengths, and interaction patterns. A large amount of unrelated text does not automatically replace relevant coverage. For chat models, consistent prompt formatting and the correct tokenizer matter. For multimodal models, use the model's processor and required non-text inputs; a list of unrelated text is not a universal multimodal recipe.

Use a finite collection. The ordinary normalization helper materializes iterable datasets into a list, so passing an unbounded stream is not safe. A streaming dataset can still be used upstream to select a bounded subset, as shown in Installation and Quickstart.

The default model.quantize() batch size is 1; batch size changes memory/speed, not the total number of calibration examples. The default minimum-length setting is 10, and normalization removes samples of length less than or equal to 10 tokens. Inspect filtering messages rather than assuming every supplied sample contributed to calibration. Keep evaluation data separate from calibration data.

Measuring error and model quality

Measure	Definition or interpretation	What it answers
Weight MSE	Mean squared difference between original and reconstructed weights	How close reconstructed values are to the originals in a local squared-error sense
Output/reconstruction error	Compare a module's output before and after quantization on fixed inputs	How local weight error affects the operation, including activation distribution
Perplexity	Exponentiated average negative log likelihood under a specified tokenization and evaluation protocol	How well a language model predicts held-out tokens; lower is better under the same protocol
Task metrics	Accuracy or another task-specific score	Whether the compressed model still performs the target workload

For N weights and T evaluated tokens, the two scalar metrics above are:

$$\operatorname{MSE}=\frac{1}{N}\sum_i(w_i-\hat w_i)^2, \qquad \operatorname{PPL}=\exp\left(-\frac{1}{T}\sum_t\log p(x_t\mid x_{<t})\right).$$

These are general metric definitions, not promises that a particular quantize() log field is perplexity or ordinary MSE. In particular, GPT-QModel's mse configuration controls shrink-grid search. A number such as mse=2.4 is a legacy shorthand for MSEConfig(norm=2.4) and minimizes an absolute-error power objective, not ordinary squared error. MSEConfig.norm defaults to 0.0, disabling that shrink-grid search. It does not disable GPTQ's entire error-compensation algorithm.

A lower weight MSE does not guarantee lower perplexity or better task accuracy. Compare the original model and quantized checkpoint using the same tokenizer, rendered prompts, token IDs, sequence/chunk policy, dtype, and generation settings. A successful generation is a save/load smoke check, not an accuracy evaluation. See Evaluation.

Usage and configuration examples

Core GPTQ fields and their actual defaults

These are defaults for ordinary GPTQConfig(), also produced by a default QuantizeConfig() factory call in this revision. They are not universal defaults for every configuration subclass.

Field	Declared / resolved default	Concept and behavior
bits	4	Weight code width, not activation width. GPTQ config choices are 2, 3, 4, 5, 6, 7, 8. Five-, six-, and seven-bit GPTQ selects the planar gptq_p layout.
group_size	128	Number of input weights per output-channel group; -1 spans the input dimension and requires act_group_aware=False for the GPTQ solve. Config normalization checks positivity or -1; backend validation imposes narrower supported sets.
sym	True	Symmetric weight quantization; it does not imply raw packed zero codes equal zero.
desc_act	Declared None, resolved to False by GPTQConfig.default_desc_act()	Enables Hessian-diagonal-based descending activation ordering when explicitly true. Does not quantize activations.
act_group_aware	Declared None, resolved to True for GPTQ	Group-aware activation ordering. Explicit desc_act=True disables this default unless the caller explicitly enabled both, which is rejected.
static_groups	False	Precomputes quantizers for original column groups before the solve; preserves original group membership with desc_act. Not static activation scaling.
method	METHOD.GPTQ ("gptq")	Quantization method family.
format	FORMAT.GPTQ ("gptq")	Checkpoint layout.
activation	None	No explicit low-bit W4A activation stream.
pack_dtype	torch.int32	Container dtype for packed codes, not the weight's effective numerical precision.
true_sequential	True	Sequential calibration/quantization behavior.
lm_head	False	Does not include the output head in ordinary quantization by default.

The public factory is a dataclass with metaclass dispatch, not a hand-written constructor with one universal signature. Its source declaration is:

@dataclass
class QuantizeConfig(BaseQuantizeConfig, metaclass=QuantizeConfigMeta):
    """Backward-compatible quantization config factory.

    Direct construction dispatches to a concrete method-specific config class.
    """

Prefer the concrete class when you know the method:

from gptqmodel import QuantizeConfig
from gptqmodel.quantization import GPTQConfig

# Equivalent ordinary GPTQ starting points in this revision.
config = QuantizeConfig(bits=4, group_size=128, sym=True)
gptq_config = GPTQConfig(bits=4, group_size=128, sym=True)

# Explicit activation ordering, not activation precision.
ordered_config = GPTQConfig(
    bits=4,
    group_size=128,
    desc_act=True,
    act_group_aware=False,
)

The actual GPTQ solve orders columns using the Hessian diagonal when desc_act is enabled and records/restores group indexing as needed. Changing this field in an already packed checkpoint is not a conversion procedure.

Explicit low-bit activation policies

An explicit policy is different from ordinary GPTQ:

import torch
from gptqmodel.quantization import GPTQConfig

fp8_activation_config = GPTQConfig(
    bits=4,
    group_size=128,
    sym=True,
    desc_act=False,
    pack_dtype=torch.int32,
    activation="w4afp8",
)

These experimental policies require GPTQ/GPTQ_V2 INT4, symmetric groups of 128, desc_act=False, INT32 packing, and lm_head=False. Weight-layout overrides in dynamic are restricted. The saved weights remain packed GPTQ integers; the policy changes the activation contract and binds a matching backend.

Policy	Runtime selection	Requirements in this revision
w4afp8	BACKEND.GPTQ_W4AFP8	Supported Llama decoder integration; Linux CUDA, SM89+, Triton FP8 support, FP16/BF16 model inputs; input width divisible by 128 and output width by 32
w4a_nvfp4	BACKEND.GPTQ_W4A_NVFP4	Supported Llama decoder integration; Linux CUDA, documented GB10/SM121 path (validator floor: compute capability 12.1), Triton packing, native Torch FP4 and block-scaled GEMM APIs; the same packed-weight/shape restrictions

This example is a configuration illustration, not a replacement for the complete experimental workflow. It does not imply that all attention operations or the KV cache become FP8/FP4. See FP8 and NVFP4.

Method, format, and backend are three different axes

Axis	Question it answers	Examples
Method	How are quantization parameters/codes chosen?	GPTQ, AWQ, QQQ, ParoQuant, VPTQ; RTN as a calibration-free lifecycle
Format	How are the resulting codes and metadata serialized?	gptq, gptq_v2, gptq_p, AWQ gemm, gguf, exl3
Backend	What implementation executes or packs the representation?	gptq_torch, gptq_marlin, gptq_exllama_v2, awq_torch, vllm, mlx

The actual METHOD values are gptq, gguf, fp8, bitsandbytes, qqq, awq, exl3, paroquant, and vptq. There is no METHOD.RTN: RTNConfig selects the weight-only lifecycle and exports a compatible layout.

The actual FORMAT family values are gptq, gptq_v2, gptq_p, gguf, fp8, bitsandbytes, marlin, bitblas, qqq, exl3, vptq, gemm, gemv, gemv_fast, llm-awq, and paroquant. Concrete FP8/BitsAndBytes/GGUF configs can expose a more specific public format string, such as float8_e4m3fn or nf4, which is resolved to its family internally.

A GPTQ-format checkpoint can run through Marlin without being saved in legacy Marlin format. Fresh quantization with FORMAT.MARLIN is rejected; use GPTQ output and select a compatible Marlin inference backend. Likewise, selecting BACKEND.VLLM does not convert an unsupported checkpoint into a format supported by vLLM.

BACKEND.AUTO selects among compatible local implementations; it cannot install missing dependencies or make unsupported bit widths, layouts, devices, and dtypes valid. Consult Packing Formats and Compatibility, Inference Backends, and Kernel Backends Reference.

Implementation internals

Packed integers are containers, not expanded weights

For eight 4-bit codes q0 through q7, each in [0, 15], a 32-bit word can hold all eight codes:

$$p=\bigvee_{i=0}^{7}(q_i\ll4i),\qquad q_i=(p\gg4i)\mathbin{\&}15.$$

The implementation must mask after shifting, including when the container is a signed INT32 tensor. The sign of the container is not the signedness of each 4-bit code. Packing eight codes into an INT32 word does not turn the weights into 32-bit-precision values.

In the ordinary GPTQ representation, the relevant tensors include:

Tensor	Role
qweight	Packed weight codes
scales	Scale values for groups/output channels
qzeros	Packed zero-point metadata, interpreted according to checkpoint format
g_idx	Input-position-to-group mapping, including layouts that need reordered groups
bias	Optional floating-point bias

The Torch decoder reconstructs the equivalent of scales[g_idx] * (codes - zeros[g_idx]) for ordinary affine GPTQ. Optimized kernels can fuse decoding with matrix multiplication rather than materialize a whole dense matrix. Three-bit continuous packing and the five-/six-/seven-bit planar formats have different packing rules; “32 divided by bits” is not a universal layout specification.

Implementation	Responsibility
gptqmodel/quantization/config.py	Factory dispatch, method/format enums, validation, effective defaults, and activation policy
gptqmodel/quantization/quantizer.py	Range selection, scale/zero calculation, quantize/dequantize reconstruction, shrink-grid objective
gptqmodel/quantization/gptq.py	Calibration-derived Hessian handling, error compensation, ordering, and group indices
gptqmodel/nn_modules/qlinear/__init__.py	Quantized-linear contracts and integer packing implementations
gptqmodel/nn_modules/qlinear/torch.py	Native Torch decoding and inference paths
gptqmodel/nn_modules/qlinear/gguf.py	GGUF-specific encoding/decoding, including distinct MXFP4 and NVFP4 layouts
gptqmodel/utils/backend.py	Backend identifiers, legacy-name normalization, activation-policy backend binding
gptqmodel/utils/calibration.py	Text/token/chat dataset normalization, length filtering, sorting, concatenation, batching

Pitfalls and edge cases

• Do not read desc_act as “activation bits.” It changes weight-quantization ordering; bits still describes weights.
• Do not copy old defaults. Here GPTQ resolves desc_act=False and act_group_aware=True. Other subclasses and releases differ.
• Do not equate group_size=-1 with per-tensor. For ordinary GPTQ linear weights it spans input columns per output channel; fresh GPTQ quantization also requires act_group_aware=False for that grouping.
• Do not equate sym=True with no zero metadata. Unsigned symmetric codes can have a nonzero midpoint.
• Do not equate a 4-bit payload with 4 total bits per parameter. Metadata, codebooks, padding, and unquantized modules count too.
• Do not relabel packed tensors to convert formats. Changing format, sym, group_size, or desc_act in JSON without the corresponding tensor transformation can silently corrupt decoding.
• Do not compare perplexities across different tokenization/chunk protocols. Establish an unquantized baseline first.
• Do not assume narrower is faster. Hardware support, shape, compute intensity, packing overhead, and batch size decide the actual performance.
• Do not assume the native loader accepts arbitrary W8A8 checkpoints. The ordinary auto-loader explicitly rejects several external activation-quantized metadata schemes. The supported experimental GPTQ activation policy is its own path.

Source files

Paths are relative to the verified source revision:

• README.md — platform matrix, method/format/backend overview, and precision examples.
• docs/quantization-guide.md — weight/activation distinction and end-to-end recipe map.
• docs/gptq-w4a-gb10.md — activation transport and scaling contracts.
• gptqmodel/quantization/config.py — BaseQuantizeConfig, QuantizeConfigMeta, GPTQConfig, MSEConfig, ActivationConfig, METHOD, and FORMAT.
• gptqmodel/quantization/quantizer.py — Quantizer.find_params, quantize, and shrink-grid search.
• gptqmodel/quantization/gptq.py — ordering, Hessian solve, and group-index construction.
• gptqmodel/nn_modules/qlinear/__init__.py and gptqmodel/nn_modules/qlinear/torch.py — packing and ordinary integer decoding.
• gptqmodel/nn_modules/qlinear/gguf.py — blockwise MXFP4/NVFP4 weight decoding.
• gptqmodel/nn_modules/qlinear/w4a_floatx.py, gptqmodel/nn_modules/qlinear/w4a_nvfp4.py, gptqmodel/nn_modules/qlinear/w4a_triton.py, and gptqmodel/nn_modules/qlinear/w4a_nvfp4_triton.py — W4A hardware/runtime contracts and packing.
• gptqmodel/utils/backend.py and gptqmodel/utils/calibration.py — runtime selection and input normalization.
• gptqmodel/models/auto.py — quantized-checkpoint detection and unsupported activation-scheme guards.
• gptqmodel/version.py — version used for this page.

See also

• Installation and Quickstart
• Quantization Lifecycle
• QuantizeConfig Reference
• GPTQ, AWQ, QQQ, and VPTQ
• FP8 and NVFP4, GGUF, and EXL3
• Packing Formats and Compatibility
• Inference Backends
• Evaluation