Skip to content
Qubitium edited this page Nov 18, 2026 · 64 revisions

AWQ 🎯

🏠 Home · 🧭 Choose a method · 🚀 Quickstart · 🧰 Help

On this page

Runtime tuning below uses KernelConfig: pass kernel=... through the quantization/load API. Kernel-policy fields are not environment variables; build/cache and diagnostic controls remain separate.

AWQ uses representative inputs to find channel scales that make low-bit weight rounding less harmful. For model authors targeting AWQ-compatible inference: begin with a 4-bit GEMM checkpoint, verify its quality, and choose a runtime that matches your device.

Overview

🧭 Is AWQ a good fit?

Choose this when… Consider another path when…
Your deployment reads AWQ checkpoints, and you can collect representative calibration inputs You need a calibration-free baseline → RTN
You want activation-aware weight scale/clip search while keeping ordinary floating-point activation interfaces You specifically need dynamic INT8 activation operands → QQQ, or compare MACaQ for MXFP4 W4A4
You can validate the model's scaling groups and the target backend's layout/shape constraints Your runtime expects GPTQ layouts → GPTQ, or compare the method guide

Your route: 🧭 Choose the method · 🚀 Quantize, save, and load · 🔬 Why scaling works, controls, and kernel reference.

AWQ is GPT-QModel's activation-aware weight-quantization method. It uses calibration activations to search for channel scales that reduce the reconstruction error of low-bit weights, folds those scales into adjacent operations, optionally clips weight groups, and packs the resulting weights for inference. Activations are not converted to INT4 by this procedure: a 4-bit AWQ checkpoint is a weight-only quantized model whose computation dtype depends on the selected inference kernel.

Three choices must be kept separate:

Choice API/configuration Meaning
Quantization method AWQConfig, METHOD.AWQ ("awq") How calibration and weight quantization are performed.
Checkpoint format FORMAT.GEMM, FORMAT.GEMV, FORMAT.GEMV_FAST, FORMAT.LLM_AWQ How integer weights, scales, and zero points are stored.
Inference backend For example, BACKEND.MARLIN, BACKEND.AWQ_GEMM_TRITON, BACKEND.AWQ_TORCH Which implementation consumes a compatible packed checkpoint. Selecting Marlin inference does not require quantizing into a Marlin checkpoint.

Here, weight-only describes inference storage: activations remain floating point. It does not mean calibration-free quantization. AWQConfig requires calibration; an RTN configuration exporting an AWQ layout instead uses the separate weight-only lifecycle and does not run AWQ's activation-aware scale/clip search.

The normal starting point is AWQConfig(bits=4, group_size=128, format=FORMAT.GEMM) with BACKEND.AUTO when loading the saved model. Use an explicit backend when a deployment requires a particular implementation, and check its bit width, group size, device, dtype, and shape requirements below.

This page describes public source revision 2665486b8. Requirements and kernel dispatch are version-dependent; use the source and installation requirements matching your installed GPT-QModel version.

Usage

✅ Before you start

Install GPT-QModel for your platform and choose a supported model whose weights you can access. The first recipe also uses the datasets package. You need representative calibration text and memory for source weights, captured features, reference outputs, and search workspace; the final INT4 checkpoint size is not the quantization-memory requirement.

Start with FORMAT.GEMM unless your deployment requires a different AWQ layout. Native NVIDIA CUDA kernels and AMD ROCm paths are separate choices: compatible ROCm GEMM paths include the declared Triton implementation or portable Torch; native CUDA GEMV/fast extensions do not become ROCm kernels through hipify. Check the backend reference before choosing a specific implementation.

Quantize and save

from datasets import load_dataset

from gptqmodel import CalibrationConfig, GPTQModel
from gptqmodel.quantization import AWQConfig, FORMAT, MSEConfig

model_id = "meta-llama/Llama-3.2-1B-Instruct"

# Choose representative, nonempty text for the model and task.
dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")
calibration = [text for text in dataset["text"] if len(text.strip()) > 100][:128]

qcfg = AWQConfig(
    bits=4,
    group_size=128,
    sym=False,  # Explicit choice; the inherited default is True.
    format=FORMAT.GEMM,
    scale_search_chunked_activations=True,
    mse=MSEConfig(clip_grid_budget="256M"),
)

model = GPTQModel.load(model_id, quantize_config=qcfg)
model.quantize(CalibrationConfig(dataset=calibration, batch_size=1))
model.save("./llama-awq")

The model ID is illustrative and requires access to that model. The 128 examples and character filter are example dataset choices, not AWQ API defaults or a guarantee of calibration quality. GPTQModel.load() defaults to trust_remote_code=False; enable remote code only when you have reviewed and trust it. See Installation and quickstart and Quantization lifecycle for dataset preparation and the general workflow.

For native AWQ checkpoints with expanded per-expert packed weights, use GPTQModel.load() or import gptqmodel before loading through Transformers. Importing installs a bridge around Transformers' AWQ pre/post weight-loading hooks, rebuilding the serialized expert modules before assigning tensors. Without the bridge, the built-in linear-only replacement can leave fused experts randomly initialized despite a completed load. Non-native AWQ checkpoints retain Transformers' original path. See the native loading bridge.

Mixed-width modules

AWQ config validation accepts 2, 3, 4, or 8 bits, globally or through per-module dynamic rules. For mixed widths, use GEMM packing and the portable Torch backend:

from gptqmodel import CalibrationConfig, GPTQModel
from gptqmodel.quantization import AWQConfig, FORMAT
from gptqmodel.utils.backend import BACKEND

qcfg = AWQConfig(
    bits=4, group_size=128, format=FORMAT.GEMM,
    dynamic={r"^model\.layers\.0\.mlp\.down_proj$": {"bits": 3}},
)
model = GPTQModel.load("/path/to/dense-model", quantize_config=qcfg)
model.quantize(CalibrationConfig(dataset=calibration), backend=BACKEND.AWQ_TORCH)
model.save_quantized("/path/to/mixed-awq")
# Reload with the same portable runtime, not an INT4-only GEMM/GEMV kernel.
model = GPTQModel.load("/path/to/mixed-awq", backend=BACKEND.AWQ_TORCH)

Use representative calibration as in the preceding example. Scale search evaluates each member of a shared scaling group at its own width, group_size, and sym; clipping, pseudo-quantization, staging, optional local refinement, and packing use the same resolved settings. sym=False is a real affine override, not an exclusion; None inherits the global symmetry. Missing settings inherit globally, and negative rules still exclude modules. Divisibility and backend restrictions still apply.

AWQConfig(modules_to_not_convert=["visual"]) excludes that module-path component and its descendants, not similarly named siblings. Assigning a replacement list after construction refreshes the derived -: rules and exported exclusions. Producer regexes with a re: prefix are accepted without that marker. See dynamic configuration for ordering and per-method validation.

GEMM modules with non-4-bit widths or group sizes outside AwqGEMMLinear.SUPPORTS_GROUP_SIZE route to AwqTorchLinear during packing. The native CUDA GEMM/GEMV kernels only unpack INT4; the portable class implements 2–8-bit layouts, but the public config contract is narrower (2/3/4/8). Local gsq and staged gsq_training retain their separate 4-bit validation constraints. For budget-selected widths, see format-aware RCO allocation.

Load a checkpoint and generate

from gptqmodel import GPTQModel
from gptqmodel.utils.backend import BACKEND

model = GPTQModel.load("./llama-awq", backend=BACKEND.AUTO)
output = model.generate("Explain weight-only quantization.", max_new_tokens=64)
print(output)

For an explicit compatible GEMM-format implementation:

# NVIDIA Ampere or newer, subject to all Marlin validation checks.
model = GPTQModel.load("./llama-awq", backend=BACKEND.MARLIN)

# Portable GEMM-format fallback; this can temporarily dequantize weights.
model = GPTQModel.load("./llama-awq", backend=BACKEND.AWQ_TORCH)

BACKEND.AWQ_TORCH selects a reader for the saved layout. AwqTorchLinear reads GEMM, AwqGEMVTorchLinear reads GEMV on CPU, and the separate GEMV_FAST and LLM_AWQ readers support CPU, CUDA, and ROCm. Use Loading and inference for device placement and generation details.

Choose a saved layout

from gptqmodel.quantization import AWQConfig, FORMAT

gemm_config = AWQConfig(format=FORMAT.GEMM)
gemv_config = AWQConfig(format=FORMAT.GEMV)
fast_config = AWQConfig(format=FORMAT.GEMV_FAST)
llm_awq_config = AWQConfig(format=FORMAT.LLM_AWQ)

Choose before quantization. In BaseQModel.quantize(), GEMV_FAST and LLM_AWQ automatically set pack_dtype=torch.int16; the default for GEMM/GEMV is torch.int32. GEMV, GEMV_FAST, and LLM_AWQ require FP16 under AUTO. The loader resolves an inferred dtype to FP16 before materializing the entire source model, including non-quantized modules, so calibration and finalization agree. An explicit incompatible dtype raises before calibration; choose dtype=torch.float16 or GEMM with a BF16-capable backend. The same early conflict check applies when loading quantized checkpoints, using the explicit backend's declared dtype support when one is selected. Do not select a fast layout solely because of its name: format compatibility can be more restrictive than GEMM's range of inference backends.

Workflow at a glance

Scale search versus sparse-coverage fallback. AWQ chooses a per-module route inside a calibrated layer pass.

Reading the diagram: These are alternative module paths within AWQ, not calibration-free RTN. Clipping defaults on; fallback does not invoke the generic RTN processor.

Text version of the workflow
  • Normal calibration coverage: Capture scaling groups → Search 20 ratios → Fold scales → Quantize and pack.
  • Fallback selected: Check coverage → Skip scale and clip → Quantize and pack.

Open full-size SVG

Per-call fallback settings are normalized against AWQConfig.fallback rather than replacing inherited preferences with an unset value. AWQ/Paro post-initialization also receives its batch size explicitly; AWQ_BATCH_SIZE is no longer a supported environment channel. Set CalibrationConfig.batch_size and pass that object to quantize().

For MoE calibration with fallback=None, experts that receive no calibration input are pruned from AWQ's pending layer state, restored to their original dense source representation, and recorded as dynamic exclusions. The covered modules still quantize and pack; an unseen expert no longer prevents the entire layer from finishing. Scaling groups missing an excluded member are skipped so they cannot fold into that dense expert. This coverage policy is separate from replay/group errors, for which fallback=None still raises. See MoE coverage policy.

Concepts

Why activation-aware scaling helps

Think of channel scaling as moving numerical range between an input channel and its weights. Before rounding, the two changes cancel; after rounding, some scale choices preserve the observed outputs better. AWQ searches those choices, then folds the winning scale into supported neighboring operations.

For a linear layer, use row-token inputs X ∈ ℝ^(T×K), weights W ∈ ℝ^(N×K), and a positive diagonal channel-scale matrix D = diag(s) ∈ ℝ^(K×K). Here T is the number of calibration rows, K the input width, and N the output width. Then X Wᵀ = (X D⁻¹)(W D)ᵀ. A shape-explicit version of the reconstruction objective is:

$$\min_{s>0}\;\frac{1}{TN}\left\|XW^T-(XD^{-1})Q(WD)^T\right\|_F^2.$$

Q rounds/clamps weights and reconstructs them on the candidate quantization grid. The implementation searches a finite candidate family and can inspect a larger module rather than just this one linear; it is not an exact optimization over every positive scale vector.

The scale-search docstring uses the equivalent channel-by-token shorthand (a different orientation from the shape-explicit formula above):

L(s) = || Q(W * s) (s^-1 * X) - W * X ||

Q here is pseudo-quantization: round and clamp to integer codes, then reconstruct floating-point weights. The implementation measures mean-squared output reconstruction error. It does not keep selected channels permanently in full precision, nor does it choose scales from weight magnitudes alone.

For each model-defined scaling group, the processor computes:

  • x_mean: mean absolute calibration activation per input channel, using FP32 accumulation.
  • w_mean: mean normalized absolute weight magnitude per input channel across the group's linears. Weight groups are normalized by their maximum magnitude plus 1e-6 before accumulation; the implementation streams the linears instead of concatenating all their weights.

With the default processor setting duo_scaling=True, a candidate is:

s = x_mean^r / (w_mean^(1-r) + 1e-4)

With duo_scaling=False, it is s = x_mean^r. Both branches clamp the candidate scales to at least 1e-4. The current search tries 20 ratios, r = i / 20 for i = 0..19, thus 0.00, 0.05, ..., 0.95. Each candidate is normalized by sqrt(s.max() * s.min()) and sanitized for non-finite values. The candidate weights are pseudo-quantized, the inspected module is replayed, and the lowest reconstruction loss wins. Original weights are restored between candidates and after the search.

When a previous-subset weight-scale hint is available, _compute_best_scale() first multiplies w_mean by that hint. Candidate products are rebuilt from pristine master weights rather than repeatedly multiplying and dividing the previous candidate. Selection uses strict <, so equal finite losses retain the first minimum.

Group quantization and zero points

group_size partitions the input-feature dimension of each weight row. Positive values require compatible dimensions; -1 denotes one group across the row. The quantizer's basic arithmetic is:

Mode Scale and integer range Reconstruction
Asymmetric, sym=False scale = clamp(max-min, min=1e-5) / (2^bits-1); zero = clamp(-round(min/scale), 0, 2^bits-1) (clamp(round(w/scale)+zero, 0, 2^bits-1)-zero)*scale
Symmetric, sym=True scale = clamp(max(abs(w)), min=1e-5) / (2^(bits-1)-1); signed codes in [-2^(bits-1), 2^(bits-1)-1] Signed code times scale. Packing shifts codes to unsigned and supplies zero = 2^(bits-1) without changing the reconstructed value.

Consequently, a symmetric AWQ checkpoint still has affine zero-point data. sym=True does not mean that its packed zero-point tensor can be discarded. In the GEMM representation, dequantization is (integer_weight - integer_zero) * scale, not GPTQ's legacy on-disk zero-minus-one convention.

Clipping is a second search

After scales have been applied, AWQ can narrow a group's weight range before the final pseudo-quantization. _compute_best_clip() evaluates calibration-weighted group outputs, not just the distance between original and quantized weights. It computes an original group product, evaluates clipped and quantized group products, averages squared error over sampled tokens, and selects a bound for each output-channel/group pair.

The current defaults and boundaries are important:

Item Implementation
Clipping enabled AWQProcessor.apply_clip = True.
Search signature defaults n_grid=20, max_shrink=0.5, n_sample_token=512.
Candidate count int(max_shrink * n_grid), which is 10 by default.
Actual default range multipliers 1 - i/20 for i=0..9: 1.00, 0.95, ..., 0.55. The endpoint 0.50 is not evaluated.
Initial bound Maximum absolute weight within each group; clipping is symmetric about zero, including when the quantizer uses asymmetric zero points.
Token sampling Flatten activations, then take every max(1, token_count // n_sample_token)th row. This is a stride target, not a strict cap of exactly 512 rows.
Output-channel batching 256 when the output-channel count is divisible by 256; otherwise 64, capped at the channel count. Full tiles reuse one scratch set throughout the search call; a partial final tile gets its own exact-row-count buffers so reshaping remains a view rather than a copy with stale candidate values. Clipping adds no 64/256-channel divisibility requirement beyond the packer's own alignment rules.
Excluded names Any linear name containing q_, k_, query, key, or Wqkv; the code avoids these because q/k attention products make precise clipping difficult. This is substring matching, not a semantic attention classifier.
Ties and invalid candidates First minimum wins; NaN errors are mapped to infinity for selection.

Candidate, token, and product work is chunked. For FP16/BF16 weights, shrink factors are FP32 and their multiplication order is preserved before casting products back to weight dtype. These details are part of the numerical behavior, not merely performance tuning. The clipping helpers require a 2-D weight tensor and raise a shape-specific ValueError for non-2-D input rather than relying on an assertion.

Configuration reference

AWQConfig and its inheritance

AWQConfig is a dataclass derived from PreProcessorConfig, which derives from BaseQuantizeConfig. Its AWQ-specific declarations are:

@dataclass
class AWQConfig(PreProcessorConfig):
    method: METHOD = field(default=METHOD.AWQ)
    format: FORMAT = field(default=FORMAT.GEMM)
    scale_search_chunked_activations: bool = field(
        default=True,
        metadata={
            "help": "Stream and chunk AWQ scale-search activations to reduce peak memory during reference and reconstruction forwards."
        },
    )
AWQ-specific contract Value/behavior
method METHOD.AWQ; allowed_quant_methods() returns (METHOD.AWQ,).
format Defaults to FORMAT.GEMM.
scale_search_chunked_activations True; generate reference outputs in replay chunks on the inspection device and score candidate outputs chunk by chunk. The scorer retains all reference chunks on that device; this is not a GPU-residency cap. False restores the legacy eager reference/candidate path. Serialized into quantization metadata.
default_desc_act() False. The inherited constructor field starts as None and resolves to AWQ's method default.
supported_export_formats() AWQ_EXPORT_FORMATS: GEMM, GEMV, GEMV_FAST, MARLIN, BITBLAS, LLM_AWQ. Membership does not imply that fresh quantization supports every format.
Output compatibility fields Writes zero_point = not sym, version = format, and format = format, in addition to common method/quantization serialization.

Selected inherited public fields are listed below; use QuantizeConfig Reference for the full surface. Defaults are constructor declarations unless a resolved default is explicitly stated. Not every shared option changes AWQ's search algorithm; in particular, GPTQ Hessian/damping controls are not AWQ controls.

Field Default AWQ/shared behavior
bits 4 Config accepts 2, 3, 4, or 8. Native GEMM/GEMV kernels require INT4; non-4-bit GEMM uses portable Torch. Kernel/group/shape validation still applies.
dynamic None Per-module override/exclusion map. Effective bits, group_size, and sym are used throughout scale/clip search, staging, refinement, and packing; missing settings inherit globally. Other inherited fields are not a promise of arbitrary per-module AWQ search overrides.
group_size 128 Input-feature grouping; -1 means row-wide grouping. Check the selected kernel and actual layer widths.
desc_act None, resolved to False for AWQ AWQ does not perform GPTQ activation-order permutations. Some kernels accept True as compatibility metadata; this does not add GPTQ's g_idx ordering to AWQ.
sym True Symmetric versus asymmetric quantization. Set False explicitly when asymmetric AWQ is desired.
true_sequential True Shared sequential module/layer lifecycle control.
lm_head False Whether to include the output head in quantization.
activation None Shared GPTQ W4A configuration, not an AWQ option. Non-None activation policies are rejected with METHOD.AWQ in this revision.
meta None Quantization metadata.
device None Quantization device selection; normalized by the common lifecycle.
telemetry TelemetryConfig() Diagnostics; its device flag defaults to False. Not an AWQ numerical parameter.
mse MSEConfig() Typed configuration for shrink-grid/clip scratch budgets. Details below.
pack_dtype torch.int32 Packed word dtype; fast AWQ formats override it to torch.int16 during quantization.
pack_impl "auto" Shared packing implementation choice.
adapter None Adapter configuration; kernel adapter support must also match.
offload_to_disk True Offload completed modules during quantization.
offload_to_disk_path None Optional directory for disk offload.
rotation None Shared rotation option; declared choices are "hadamard", "random". Separate from AWQ's diagonal scaling.
fallback Fallback() Default object uses strategy=FallbackStrategy.RTN, threshold="0.5%", smooth=None. AWQ uses coverage/fallback decisions and directly pseudo-quantizes fallback modules without its normal scale search.
compute_device_filter None Optional fn(devices: List) -> List filter for compute-intensive work.
calibration_data_device None Calibration-storage device; supports explicit devices and "balanced" round-robin storage.
auto_forward_data_parallel True May clone forwarding work across GPUs; can increase VRAM use and setup cost.
dense_vram_strategy VramStrategy.EXCLUSIVE ("exclusive") Dense-pool placement; alternative enum value is BALANCED ("balanced").
dense_vram_strategy_devices None Optional device pool for dense modules.
moe_vram_strategy VramStrategy.EXCLUSIVE Expert-pool placement; expert families stay co-located.
moe_vram_strategy_devices None Optional expert device pool.
gc_mode GcMode.INTERVAL ("interval") Shared GC policy; alternative ON_STAGE_END ("on_stage_end").
wait_for_submodule_finalizers False Wait for packing/offload finalizers before the next layer, potentially reducing VRAM pressure.
moe None Optional MoEConfig for routing/batching. See Multimodal and MoE models.
preprocessors New empty list per instance Ordered shared preprocessing configs.
smoother None Accepts SmootherConfig, a smooth-method object, dictionary, or string; normalized into the preprocessor list.
smooth None Backward-compatible smoother input alias, hidden from repr; the resolved smooth method is also exposed here after normalization. Prefer smoother.

The internal _offload_temp_dir field is not an initializer argument. method and format are inherited fields overridden by the AWQ declarations above. See QuantizeConfig reference for common normalization, dynamic rules, placement, and preprocessing APIs.

MSEConfig, SmoothMSE, AWQ scales, and clipping

These similarly named mechanisms are not interchangeable:

Mechanism Configuration/default Relationship to AWQ
AWQ activation-aware scale search Processor duo_scaling=True; fixed 20-ratio search Native AWQ algorithm, implemented by _search_best_scale() and _compute_best_scale(). Does not read SmoothMSE.steps or SmoothMSE.maxshrink.
Generic MSE shrink-grid objective MSEConfig.norm=0.0 Zero disables the general quantizer's shrink-grid search. It does not disable AWQ's scale search or clip search; those compute their own reconstruction MSE.
Generic shrink-grid workspace MSEConfig.grid_budget=None Used for the GPTQ/QQQ shrink-grid workspace; not the AWQ clip-grid budget.
AWQ clip-grid workspace MSEConfig.clip_grid_budget=None Read by _compute_best_clip() to size independent scratch buffers/chunks. Does not set clip strength or turn clipping on/off.
Generic smoother shrink search SmoothMSE(steps=32, maxshrink=0.8, group_size_threshold=128) A typed shared smoothing method. AWQConfig can normalize/serialize it through its parent config, but AWQ's scale, clip, and pseudo-quantization routines do not consume it as their search policy.
AWQ clipping policy Processor apply_clip=True, clip method defaults n_grid=20, max_shrink=0.5 Independent of the generic SmoothMSE and mse.norm controls.
AWQ chunk budget Processor max_chunk_memory=1024 * 1024 * 1024 Used by activation statistics, reconstruction loss, and replay chunking. Not the same setting as clip_grid_budget.

PreProcessorConfig normalizes smoother (or, when absent, the legacy smooth alias), keeps a single smoother entry after other preprocessors, and exposes its method through resolve_smooth_method(). ModulePreProcessor records an ordered per-module preprocessor plan. This configuration plumbing must not be mistaken for an AWQ implementation of every generic smoother: the SmoothMSE branches that perform generic shrink optimization are in the GPTQ, QQQ, and RTN quantizers, not AWQ's scale/clip search.

There are no AWQConfig(apply_clip=...), AWQConfig(duo_scaling=...), AWQConfig(max_shrink=...), or AWQConfig(n_grid=...) fields in this implementation. Those are processor attributes or internal method parameters. Do not copy similarly named options from another AWQ library into this dataclass. Changing them requires processor-level integration, not an undocumented keyword to GPTQModel.quantize().

The source's smoother constructor is:

def __init__(self, steps: int = 32, maxshrink: float = 0.8, group_size_threshold: int = 128):
    ...

The memory-budget config fields are:

norm: float = 0.0
grid_budget: Optional[Union[int, str]] = None
clip_grid_budget: Optional[Union[int, str]] = None

Budgets accept integer byte counts and binary-unit strings such as "256M" and "1G". A None budget resolves to 16 MiB on CPU and 256 MiB on other devices. These are chunk-sizing targets for individual scratch workspaces, not limits on total model, activation, process, or device memory. Several scratch buffers can coexist, and minimum chunk sizes can exceed a very small requested target.

A zero clip_grid_budget selects single-candidate chunks and one-token product tiles; it does not turn clipping off. Negative byte budgets are rejected.

When the active allocator has a finite max_split_size_mb, clip search additionally caps its scratch target below that split limit, leaving 2 MiB of headroom (with a 1 MiB floor). With a caller-set 256 MiB split cap, this means at most 254 MiB per target. This keeps rounded allocator segments reusable between projections instead of growing reserved VRAM for each near-cap allocation. Without a split cap the requested budget is unchanged. The adjustment changes chunk sizes, not the candidates or reduction axes; it is automatic and adds no AWQ option. See allocator-aware scratch sizing.

from gptqmodel.quantization import AWQConfig, MSEConfig

qcfg = AWQConfig(
    scale_search_chunked_activations=True,
    mse=MSEConfig(clip_grid_budget="64M"),
)

This changes AWQ clip workspace sizing. It does not disable clipping, change the 20-ratio scale search, or set a different clipping grid.

Implementation and looper integration

Public entry points and the AWQ processor

The public lifecycle is GPTQModel.load(..., quantize_config=AWQConfig(...)), model.quantize(CalibrationConfig(dataset=calibration), ...), then model.save(...). The loader entry point is defined in gptqmodel/models/auto.py:

@classmethod
def load(
    cls,
    model_id_or_path: Optional[str],
    quantize_config: Optional[QuantizeConfig | Dict] = None,
    device_map: Optional[Union[str, Dict[str, Union[str, int]]]] = None,
    device: Optional[Union[str, torch.device]] = None,
    backend: Union[str, BACKEND] = BACKEND.AUTO,
    profile: Union[str, int, PROFILE] = PROFILE.AUTO,
    trust_remote_code: bool = False,
    allow_remote_adapter: bool = False,
    adapter_revision: Optional[str] = None,
    adapter_sha256: Optional[str] = None,
    **kwargs,
):
    ...

The quantization entry point is defined in gptqmodel/models/base.py:

def quantize(
    self,
    calibration: Optional[CalibrationConfig] = None,
    *,
    tokenizer: Optional[PreTrainedTokenizerBase] = None,
    backend: Optional[Backend] = BACKEND.AUTO,
    adapter: Adapter = None,
    embed_quant_config: Optional[Union[QuantizeEmbedConfig, QuantizeEmbed]] = None,
    embed_quant_mode: Optional[QuantizeEmbed] = None,
    checkpoint: Optional["CheckpointConfig"] = None,
    fuse: Optional[bool] = None,
) -> Dict[str, List[Dict[str, str]]]:
    ...

The AWQ-specific processor signature in gptqmodel/looper/awq_processor.py is:

def __init__(
    self,
    tokenizer,
    qcfg: QuantizeConfig,
    calibration: CalibrationConfig,
    gptq_model,
    model,
    forward: Optional[ForwardConfig] = None,
    calculate_w_wq_diff: bool = False,
):
    ...

The model forwards the effective CalibrationConfig.batch_size explicitly to post-initialization and schedules shared preprocessors before AWQProcessor. Its execution configuration requires forwards by default, enables activation capture, requests replay after processing, and permits early stopping of subset forwards:

ForwardConfig(
    required=True,
    replay_after_process=True,
    subset_early_stop=True,
    activation_capture=True,
)

Per-layer lifecycle

Stage Implementation and state
Register modules preprocess() honors dynamic exclusions, records modules in _AWQLayerState, normalizes fallback preferences, and initializes capture buckets. Locks guard shared layer state and finalization.
Capture inputs Forward hooks collect input activations. Padding is removed when the active keep mask can be aligned to input rows. Model-specific MoE policies can capture a shared root feature rather than only routed expert inputs.
Aggregate calibration Every module defaults to bounded token_rows capture, including dense modules with variable-length inputs. MoE roots additionally capture the shared pre-router input. Logs distinguish raw tokens, retained tokens, batches, aggregation mode, and the feature actually used for scaling.
Wait for scheduled work process() tracks processed subsets and pending modules. _quantize_layer() runs after all scheduled subsets for the layer are ready, avoiding partially collected features.
Resolve scaling groups gptq_model.awq_get_modules_for_scaling() uses model module-tree information to identify the preceding operation, linears sharing a scale, captured inputs, module to inspect, and forward kwargs.
Search and apply scales _search_best_scale() gathers statistics/reference outputs; _compute_best_scale() tests candidates. apply_scale() folds the winning scale into the graph and adjusts captured downstream features.
Search and apply clips Unless apply_clip is false, eligible non-fallback linears are passed to _search_best_clip() and apply_clip().
Pseudo-quantize apply_quant() writes reconstructed weights, saves q_scales and q_zeros, optionally records the original-minus-quantized difference for adapter work, and stages packing state.
Pack submodule_finalize() calls pack_module(), which resolves the selected kernel, creates the quantized module, packs its weights/scales/zeros, and removes temporary state. The dense fake-quantized wq is consumed for an adapter difference or dropped after packing; it is not kept until cyclic garbage collection.
Finalize finalize() marks the model quantized and sets the method to METHOD.AWQ.

AWQProcessor selects AwqTorchLinear for GEMM modules with non-4-bit widths or group sizes outside the native GEMM declaration before consulting the model's selected qlinear_kernel. Otherwise it can reuse that kernel; its internal format selector maps GEMM to AwqGEMMLinear (or AwqTorchLinear on NPU), GEMV to AwqGEMVLinear, GEMV_FAST to AwqGEMVFastLinear, and LLM_AWQ to LLMAwqLinear. The normal public lifecycle chooses a compatible packing backend first, including portable GEMM packing on non-CUDA hosts. The processor's internal fallback selector alone is not the entire public backend-selection policy.

Calibration is established at processor construction. set_calibration_dataset() deliberately raises NotImplementedError, and verify_calibration_dataset() rejects missing calibration. Continuation methods save/restore AWQ loss, sample-count, and batch-size state; this does not make an ordinary inference checkpoint equivalent to a complete resumable quantization checkpoint. See Looper processors and Quantization lifecycle.

Replay, positional inputs, and memory

Chunked scale search is enabled by default. It prepares the inspected module's device placement and rotary scaffolding once and generates reference outputs in replay batches directly on the inspection device. The full reference remains there as detached chunks throughout the ratio grid; it no longer takes an intermediate GPU→CPU→GPU round trip. _prepare_module_forward() also moves the full replay input to that device. Candidate outputs and their loss comparisons are processed chunk by chunk, but this is not streaming of the full input/reference from CPU.

Weight statistics stream across linears, and activation sums use chunked FP32 reductions; computing inp.abs() still materializes an activation-sized tensor. Large original-weight master copies may be staged on the host when their combined size exceeds max_chunk_memory. Replay sizing now uses all elements per leading-dimension row: x.numel() // x.shape[0], not just hidden width. For [batch, sequence, hidden], the sequence length therefore counts toward the estimate; input, output, and CPU-reference copies contribute a factor of three, with at least two bytes per element. This is still a heuristic, not an exact bound on module intermediates or total scale-search memory. Disabling scale_search_chunked_activations restores the eager path that assembles full reference/candidate outputs.

The replay machinery sanitizes kwargs against each module's actual forward signature, removes use_cache, aligns masks and positional metadata to the replay input, and handles rotary-device placement. Typed rotary selection prefers the attention's rope_layer_type, then layer_type / attention_type, including nested attention modules; declared rotary labels are validated before the call. This avoids passing DeepSeek V4's architecture label sliding_attention to a rotary indexed by main / compress. Mapping-shaped position_embeddings retain their keys and rebuild a cos/sin pair per captured rope type; replay chunks slice the mapping too. When no valid selector can be resolved, matching-length captured embeddings are retained and stale ones are dropped. See rotary replay selection.

If those attention/module attributes provide no valid selector, replay also checks model.config.layer_types[current_layer_index], still validating against the rotary's declared labels. This supports Laguna's config-only full_attention/sliding_attention labels without guessing from its boolean attention flags. MPT's definition supplies an inert zero residual when an MLP replay requires residual and no tensor residual was captured; the same additive term cancels from reference/candidate reconstruction loss. This fixes the missing-argument failure without adding a user option. See MPT replay and rotary resolution.

Packed activation rows cannot automatically reuse an arbitrary original attention mask or position-ID tensor. Token-row provenance lets replay reconstruct document/position structure from captured masks (including block-diagonal additive masks where applicable); incompatible positional metadata is rebuilt or dropped. Captured input_ids are selected in the same order as the retained activation rows, including when no attention mask was captured. With padded batches, replay maps each retained-row index back to the original batch and sequence position before reading the corresponding ID. This keeps padding and excluded attention sinks from shifting the IDs. DeepSeek-V4 hash_moe uses those IDs to route the replayed tokens. If replay receives an ID tensor with the wrong token count, AWQ raises a scale-group error and applies the configured fallback rather than passing mismatched IDs to the router. This bounded replay mask can itself require quadratic storage in the retained row count. A model definition can override the default aggregation policy; returning None restores legacy batch-concat/latest-capture behavior for that module. Models can opt into preserving explicit position embeddings through their AWQ policy.

For modules whose captures match the cached batch lengths, AWQ plans deterministic per-batch quotas and keeps the sampled rows as each batch arrives. The default budget starts at min(total_rows, max(largest_batch_rows, non_empty_batch_count)). AWQ caps both that budget and a definition's max_tokens at the model's declared position limit when present. Each nonempty batch contributes when the planned quota permits. Packed rows form one replay sequence, so a batch of individually valid sequences can otherwise exceed a fixed causal buffer, as on GPT-Neo.

When a planned quota keeps one row from a batch, AWQ selects the middle of the eligible rows (zero-based index row_count // 2). It no longer selects row zero from every batch, which could leave scale search dominated by repeated BOS activations on calibration sets with many short documents. This also applies to selected batches when the budget is smaller than the number of nonempty batches. Multi-row quotas still sample evenly from the first through the last eligible row.

With CalibrationConfig(exclude_attention_sinks=True), AWQ deducts the marked BOS rows from the batch length used to plan quotas. The same rule applies when the model supplies no attention mask, so an unmasked batch with one BOS row can still contribute its remaining rows.

When captures do not match the planned lengths, as with dynamically routed experts, AWQ uses a bounded priority reservoir per module. For a stream handled entirely by the reservoir, each captured row has the same chance of surviving regardless of its batch's position: AWQ assigns random priorities seeded by capture slot and keeps the lowest priorities up to the budget. Each module updates its buffer in place without rebuilding it on every batch or holding up other experts' merges. If a module switches from planned capture to the reservoir, previously sampled rows seed the buffer; rows already omitted by the plan cannot be recovered. The retained rows carry their original batch and row references for replay, and raw token counts still drive coverage checks. Capture and aggregation source

Sibling projections can share a host capture under torch.inference_mode(), including when they use the same keep mask. The retention budget applies to saved activations between captures. The current batch, transient sampling buffers, and replay masks also use memory. Broadcast or matrix-shape errors in scale-group replay use the configured group fallback (fallback=None raises). Unrelated runtime errors still propagate.

Packed replay preserves the captured attention-mask form: floating additive grids treat every nonnegative entry as attended, including an all-zero unpadded grid; integer square 0/1 grids use positive entries as keep (including LLaDA2's multiplicative masks). Boolean grids retain whether True means keep or mask, and planar keep masks keep their rank. Packed integer grids retain their integer dtype and block separation. A planar mask cannot express cross-document block separation; causal handling remains in that attention implementation. Multimodal rotary discovery selects the text decoder's rotary module, not a vision/audio tower, and supports nested language-model wrappers. Folds do not cross residual/nonlinear boundaries such as LongCat post-attention norms or Nemotron-H MLP activations. These are correctness fixes, not new AWQ configuration flags.

Recurrent/convolution mixers whose registered builder is create_recurrent_attention_mask (such as linear_attention and conv) receive a planar [1, retained_tokens] keep mask even when the original unpadded capture used None. They never receive the attention-only [1, 1, T, T] grid that previously failed scale-search replay on hybrid Mamba/conv layers. Replay resolves block_type and nested mixer metadata as well as attention metadata. Packed rows retain source order, but recurrent state is not reset between documents; the planar mask does not provide attention-style block isolation. See layer-type replay.

Nemotron-H and Nemotron Omni share the same decoder declaration. Their dense, shared-expert, and routed-expert up_proj entries are tagged :nosrc: the expert computation is down_proj(relu2(up_proj(x))), so folding a scale from up_proj through squared ReLU would change the layer output. The block input norm remains the scale source for the routed/shared expert group, including compensation for its retained router. Omni's shared/routed projections are now declared for quantization rather than silently left in BF16. See Nemotron decoder coverage.

For mixed attention stacks, token-row replay selects the captured mask for the current layer's resolved attention type, rather than reusing layer 0's sliding-window mask for a full-attention layer. Retained rows' original positions also survive signature sanitization through private replay metadata, so modules accepting only position_embeddings rebuild rotary embeddings from real positions instead of a new contiguous range. See calibration aggregation.

Bloom replay requires a captured, sequence-aligned attention mask. When retained-row positions are available, its model hook rebuilds ALiBi from those positions and the model's head slopes, even if a stale captured ALiBi has the right shape. Without usable positions, it requires aligned captured ALiBi and reports expected/actual shapes on mismatch; missing context never becomes an all-visible mask. The hook supplies a fixed additive residual when absent and slices ALiBi's batch-times-heads axis correctly for replay chunks. Unsupported context uses the configured group fallback, or raises with fallback=None. When the live block sets apply_residual_connection_post_layernorm=True, Bloom disables scaling groups and plain-quantizes its supported linears even with fallback=None, because folding that norm would alter the residual path. See Bloom replay safety.

Scaling groups follow actual data flow

Before calibration, AWQ asks the selected packing kernel whether each included Linear can use its configured group size, checking both input and output widths where that kernel requires it. If needed, it tries supported positive sizes no larger than the request, then per-channel -1. If none validates, it leaves the setting unchanged for later validation rather than claiming the shape is supported. Exact per-module dynamic overrides preserve resolved settings so search, packing, save, and reload agree. Excluded modules and an unquantized LM head are not retuned. See the pinned processor.

Offset-gain norm folds (such as Gemma's 1 + weight) renormalize scales by their peak when it exceeds one, avoiding an unrepresentable BF16 1 + weight round trip. Captured features and non-quantized consumers use that same effective scale. Ordinary multiplicative-gain norms keep their prior behavior.

Tree order alone does not establish a valid fold source. Sandwich post-norms (:postnorm) and leaves marked :nosrc cannot supply the following consumer's scaling group. This prevents folding across residual additions, non-homogeneous activations, gated paths, or incompatible widths. Weightless/input-gated norms, Conv1D and HyperConnection sources are skipped when they cannot absorb the fold.

A two-dimensional projection that the module plan leaves unquantized (:!/:?) cannot be a fold source: dividing its output rows has no compensation step. That scaling group is dropped and its consumer is quantized plainly. For example, Jamba's excluded mamba.dt_proj remains unchanged instead of absorbing a scale intended for mamba.x_proj; matching widths and tree order do not prove that one supplies the other's input. Gain norms and quantized projections remain eligible subject to the other source/coverage checks. See the scaling-source contract.

An absent declaration blocks fallback to a stale producer when it is a missing predecessor in the same branch after that branch has already emitted quantized modules. On remote Ling 2.0 layers without the declared MLA q_b_proj/kv_b_proj pair, attention.dense therefore does not fold its input scale into input_layernorm; that unsupported group remains unscaled and is quantized plainly. Absent sibling branches under different module-plan parents, such as mamba versus self_attn or conv versus self_attn, do not discard a valid norm source. Shared :in=<tag> metadata also identifies siblings, including Qwen3.5-style layers that omit self_attn while their live linear_attn reads the tagged input. These checks survive a run of absent declarations. See the group builder.

When live sibling branches share one norm but only some members lie inside the replay boundary, AWQ scores the covered members and folds the same scale into every declared member reading that shared input. Falcon-H1's attention/Mamba branches therefore keep a valid norm fold without replaying the full decoder on already-normalized input or dropping the group. For Cohere2-MoE, the router reads the same input_layernorm output from outside the attention replay boundary: it is not replayed there, but the final fold still compensates it. Fold-only siblings must still bind correctly and match the source width. Non-foldable producers, such as Conv1d, weightless norms, or gated norms without a plain foldable gain, are skipped before scale search. This does not relax replay-boundary checks for arbitrary unrelated modules.

One gain norm has one merged scaling group: routed/shared experts and parallel attention/MLP branches using that norm are scaled together. Shared :in=<tag> metadata resolves parallel branches to the same source. Candidate search leaves non-quantized consumers such as the router and shared_expert_gate unchanged: the candidate linears already divide out the trial scale, so scaling the router would change the objective's routing. Final folding compensates all declared consumers together. DeepSeek MLA producer/consumer projections use separate sequential subsets.

AWQ captures the shared pre-router input of a declared layer-level :moe root from ModulePlan.moe_roots(), even when the definition has no moe_lifecycle_hooks. This keeps routed-expert gate/up scaling active for definitions such as Granite-MoE and gpt-oss. Capture deduplicates roots already covered by a lifecycle hook; nested roots retain the existing fallback. If a declared root still captures no activations, the builder warns and skips its gate/up scaling group, leaving those projections on plain quantization rather than silently dropping the group as a width mismatch. This capture fix does not add routing-bypass support; see scaling groups.

Before scoring, a group must resolve to live modules with compatible widths, and its inspection boundary must contain every quantized target and declared non-quantized consumer. Reference replay must execute all of them. A group spanning separate attention/MLP roots is rejected if the retained inspection module covers only one branch; the processor does not split a shared norm into independent folds or guess a wider replay input. Duplicate sources and overlapping/tied targets across groups raise before filtering or mutation. Candidate weights are restored in finally, including on quantizer, replay, or loss failures.

For models with a declared expert index and resolvable scalar expert count, reference and candidate search re-enter the configured ExpertsRoutingOverride used during capture, then restore the original routing state. All-expert capture therefore does not get scored under native top-k by mistake. Dense groups still reject an unexecuted target; routed-expert groups whose search input comes from outside the member list can score only the members that replay actually executes, while folding the resulting scale into all members. No executed target still rejects the group. With MoEConfig(routing=ExpertsRoutingBypass()), supported bypass replay separately feeds every target expert the shared block input before using the original routed forward for the loss output. It restores the original MoE forward even if search fails. The replay boundary must contain the owning MoE block and expose iterable experts through its lifecycle hooks; unsupported scaffolding still falls through to the coverage guard. Neither mode changes inference-time routing.

Bypass replay preserves the MoE block's own forward signature. Blocks that accept only hidden states, such as OLMoE's sparse MoE block, no longer receive layer-level attention_mask or position_ids just because the bypass wrapper accepts arbitrary keywords. Blocks that genuinely accept **kwargs still receive them. See MoE routing overrides.

Architecture-specific repairs also disable unsafe sources or fix projection staging:

Definitions Current AWQ behavior
cohere_compass, granitemoe_swa Shared norm consumers are grouped together; incomplete cross-root replay uses fallback rather than a partial objective.
hunyuan_v1_moe Includes the nested router weight mlp.gate.wg in final compensation.
falcon_mamba, moss, qwen Disable sources crossing state-space, parallel/residual, or non-homogeneous activation boundaries.
granitemoehybrid mamba.in_proj:nosrc blocks folds across convolution/SSM operations; mamba.norm:! supplies out_proj. post_attention_layernorm:!:nosrc prevents an incomplete MoE-only fold that misses shared_mlp; those projections quantize plainly. Mamba state owners and the router remain dense.
hymba The input_layernorm → mamba.in_proj group scores the input projection directly when it is the norm's only consumer. The search uses the captured input rows without needing the mixer block's cross-layer K/V or packed mask. Other block replay preserves captured public kwargs such as attention_mask. Routed experts take their gate/up input scale from pre_moe_layernorm; mamba.in_proj:nosrc protects the downstream convolution and state-space path. conv1d, x_proj.0, and dt_proj.0 stay dense.
internlm2, minicpm Input and output projections use separate sequential stages; parallel Q/K/V and gate/up inputs remain grouped.
gemma3n, gemma3n_text Disable folds across AltUp, Laurel, value normalization, and residual/activation paths; retain explicit weight ownership.
axk2, glm_moe_dsa, longcat_flash Disable ambiguous attention sources and declare nested norm/indexer/router owners.
intern_s2_preview, minicpmv4_6 Restore shared :in=x tags for attention/DeltaNet input branches and gate/up projections, so each input norm folds once with compensation for in_proj_b/in_proj_a.
minicpmv, minicpmo Q/K/V share stage 0; o_proj and MLP down_proj have separate downstream stages instead of folding parallel projections into one another.
diffusion_gemma v_proj:nosrc blocks folds through scale-free value normalization. The router remains unquantized but is not compensated as a consumer of an expert norm it does not read.
ricursive The input norm and sigmoid g_proj are not fold sources; lerped v_proj occupies its own stage. A width-compatible v_proj → o_proj fold remains allowed.
qwen4_exp, qwen4_exp_text o_proj, in_proj_a, and out_proj cannot supply folds across gated hyper-connections/DeltaNet boundaries; shared and routed MLP experts use plain quantization together.
inkling_mm_model Disable the norm fold with unhandled native 3-D shared-expert consumers; keep those shared weights dense.

If no usable group exists, AWQ still pseudo-quantizes and packs the layer with scales_list=[]; it does not finalize an unquantized layer or leave missing q_zeros. See the source contracts for scaling groups and calibration aggregation.

Assertions raised by a replayed module become scale-group failures and follow the configured fallback policy. On a balanced MoE pool, norm-anchored groups can span expert devices during scale search and folding. See Hymba KV-reuse replay.

This is why new architectures need model-specific grouping and capture policies, not just a list of all nn.Linear modules. See Model definitions and module tree.

Loss and fallback semantics

Strategy-only fallback dictionaries retain the configured parent threshold, or the standalone Fallback default "0.5%"; they no longer silently raise it to "1%". This matters for sparsely routed experts near the cutoff.

The search methods are internal AWQProcessor APIs; their source signatures expose the difference between the fixed scale-ratio grid and the clip method's parameterized grid:

def _compute_best_scale(
    self,
    x: torch.Tensor,
    w_mean: torch.Tensor,
    x_mean: torch.Tensor,
    module2inspect: torch.nn.Module,
    linears2scale: List[nn.Linear],
    fp16_output: torch.Tensor,
    kwargs: Dict={},
    replay_batch_size: Optional[int] = None,
    prepared_forward: Optional[Tuple[torch.Tensor, Dict]] = None,
    bits: Optional[int | List[int]] = None,
    group_sizes: Optional[int | List[int]] = None,
):
    ...
def _compute_best_clip(
    self,
    w: torch.Tensor,
    input_feat: torch.Tensor,
    n_grid=20,
    max_shrink=0.5,
    n_sample_token=512,
    bits: Optional[int] = None,
    group_size: Optional[int] = None,
):
    ...

_compute_loss() flattens reference and candidate outputs, compares them in chunks, subtracts in the output dtype, casts the differences to FP32, and sums their squares in FP32 with an FP64 running total. It divides by the number of output elements. The chunked replay path uses the same reconstruction objective and avoids assembling one contiguous reference tensor, but retains its complete reference as a list of device-resident chunks. These paths need not have identical floating-point reduction order for different chunk boundaries.

Asymmetric pseudo-quantization includes zero in each group's fitted range, preventing constant positive or negative groups from collapsing to zero. Range selection repairs low-precision overflow, bounds endpoint codes whose reconstruction would overflow, and preserves the source dtype. GEMV_FAST/LLM_AWQ staging additionally checks the scaled zero offset so packed low-precision decode remains finite and zero-valued weights are not shifted by a rounded offset. These are affine fitting/representation fixes, not extra user flags or a guarantee for non-finite source weights.

When a group lacks sufficient calibration coverage, configured fallback decisions can exclude it from normal AWQ scaling/clipping. AWQ's fallback implementation calls apply_quant(..., scales_list=[]): direct pseudo-quantization rather than a call to the separate generic RTN processor. Do not assume that Fallback.smooth or a generic RTN SmoothMSE policy will be applied by this AWQ fallback path. For token-row aggregation, fallback coverage uses raw routed token counts rather than the deliberately bounded retained sample; other aggregation modes use retained rows when available.

Routed-expert gate/up groups with a shared MoE-root search input split coverage per member: covered experts keep AWQ scale search even if a sibling has too few or no routed samples. Every member still receives the compensating fold, and only starved quantized members take plain quantization afterward. A dynamic-excluded expert stays dense but receives the fold required to preserve the shared norm's transform. If every member is individually starved but aggregate group coverage passes the threshold, search uses the executed members; otherwise the whole group falls back. Dense groups retain the all-member coverage requirement. Empty captured features are not divided by the folded scale. See the coverage implementation.

Gate/up groups prefer a nonempty captured MoE-root input even when an earlier sibling has already selected that root. This keeps partial-coverage search active for covered routed experts. In hybrid models such as ERNIE 4.5 MoE, the dense prefix can have an empty MoE-root entry because it never runs the expert hook. Its norm-to-gate/up group uses the dense block's own captured input in that case. This does not substitute dense features for a routed-expert group whose required capture is missing.

With fallback disabled, empty tensors present in the input-feature map trigger a RuntimeError. This is not a universal guarantee that every invalid scaling group raises: missing feature-map entries and predecessor/linear dimension mismatches can be filtered out. If no usable scaling group remains, the processor still pseudo-quantizes and packs with an empty scale list; normal activation-aware scale search has not run. Inspect warnings and actual quantized modules when adding an architecture or diagnosing incomplete quantization.

Clip search separately skips a projection whose captured feature tensor is empty and logs the missing samples, avoiding an empty-tensor reshape. Direct _compute_best_clip() calls with empty input raise ValueError. Subset capture waits for every member to execute in each calibration batch, fixing missing sibling features when runtime order differs from the module declaration, as in Falcon-H1 and Ling hybrid attention. The pending-member set and stop flags reset before every batch, so completion of the first batch cannot truncate capture in later batches. This does not establish valid coverage for an expert that never receives routed tokens.

Constructed groups rejected by replay-boundary, target-execution, or explicit unsupported-replay checks use configured plain quantization with a warning; fallback=None makes those group errors raise. Conflicting folds always raise. Definitions that intentionally suppress groups with :nosrc or a model override still take the existing plain-quantization path independently of that setting. Unrelated forward exceptions propagate rather than being treated as safe fallback. Matching widths, plan coverage, and execution of declared targets alone cannot prove producer identity or discover undeclared consumers.

If every scale-search candidate produces NaN/infinite reconstruction loss, _AWQScaleGroupError reports the inspected module, input width, and candidate count. That error follows the same configured plain-quantization fallback, or raises with fallback=None, instead of aborting with an uninformative generic exception.

The displayed per-module loss is the associated scale-search reconstruction loss. Modules without an applicable scale-search result are logged as "not applicable"; it is not a perplexity, model accuracy score, or universal GPTQ-comparable loss. Retained-token logging also must not be confused with the number of original calibration examples. Evaluate the saved model separately; see Evaluation.

The quantization/awq/ package

Scale and clip searches live in AWQProcessor; the AWQ package provides transformations, packing, calibration utilities, and Triton execution components.

Path under gptqmodel/quantization/awq/ Main symbols Role
quantize/scale.py apply_scale, apply_clip, scale_ln_fcs, scale_fc_fc, scale_fc_fcs, scale_gelu_fc Fold searched scales into supported predecessor/linear patterns; clamp weights using searched clip bounds.
modules/act.py ScaledActivation Wrap an activation and divide its output by channel scales.
modules/triton/gemm.py awq_dequantize_triton, awq_gemm_triton, JIT kernels GEMM-format INT4 unpack/dequantization and fused matrix multiplication.
modules/triton/scheduler.py AwqTritonPlan, select_awq_triton_plan, candidate_plans, rule/cache helpers Deterministic, device/dtype/exact-shape-aware dispatch policy; no benchmark or kernel compilation during plan selection.
utils/packing_utils.py pack_awq, pack_awq_columns, unpack_awq, reverse_awq_order, pack_exllama, unpack_reorder_pack, dequantize_gemm Integer layout conversions and Torch reconstruction.
utils/calib_data.py get_calib_dataset Standalone AWQ calibration-block helper, distinct from the public model's dataset-preparation lifecycle.
utils/module.py get_named_linears, get_op_by_name, set_op_by_name, get_op_name, append_str_prefix, exclude_layers_to_not_quantize Resolve modules and names, enumerate linears, and filter exclusions.
utils/utils.py get_module_by_name_suffix, simple_dispatch_model, set_module_name, compute_memory_used_pct, get_lowest_memory_device_index Module lookup/replacement, Accelerate hooks/device-map dispatch, CUDA memory helpers.
Package __init__.py files Package boundaries These do not replace AWQProcessor with a separate public quantization engine.

apply_scale(module, scales_list, input_feat_dict=None) dispatches according to predecessor type. Norm-to-linear folding resolves the actual gain parameterization with a live probe: plain gain uses weight / scale; offset gain uses (1 + weight) / scale - 1 in FP32 before restoring storage dtype. This is not a fixed Gemma class list or a source-text heuristic. If gain semantics cannot be resolved, the group is logged and skipped without mutating norm weights, consumer weights, or captured features. Linear-to-linear folding divides the appropriate trailing predecessor weights/bias and multiplies one or several downstream linears. Activation-to-linear folding uses a scaled activation wrapper. When features are supplied, the same input-channel scale is divided out of captured downstream inputs. apply_clip(module, clip_list) reshapes weights into clip groups, clamps, and restores the module's prior device placement after working on its execution device.

The predecessor is the last width-matching gain norm feeding the block, excluding definition leaves tagged :postnorm. In particular, LFM3 post-operator/post-FFN residual norms must not receive the input scale. If the remaining fallback is itself a declared post-norm, AWQ skips that block. See normalization selection.

The standalone calibration helper has this source signature:

def get_calib_dataset(
    data: Union[str, List[str], List[List[int]]] = "pileval",
    tokenizer=None,
    n_samples=128,
    max_seq_len=512,
    split="train",
    text_column="text",
):
    ...

"pileval" resolves to mit-han-lab/pile-val-backup, validation split; other dataset strings use the requested split. String datasets are shuffled with seed 42. It accepts lists of text or token-ID lists, skips empty or overlong examples, concatenates accepted tokens, and returns only full max_seq_len blocks, dropping the final partial block. n_samples counts accepted input examples, not returned blocks. Supply nonempty data with enough accepted tokens: the helper is not an empty-dataset recovery mechanism. Its defaults are not defaults for model.quantize().

Saved formats and packing

Let K = in_features, N = out_features, G be the resolved group size, and Z = calculate_zeros_width(K, G). Shapes below describe the usual 4-bit stored layouts before backend-specific runtime repacking. These layouts are not interchangeable.

Transformers Conv1D projections, such as GPT-2's, store weights as [in_features, out_features]. AWQ quantizes and packs them as [out_features, in_features] so group sizes apply to input channels. The temporary quantized weights are transposed back when updating the source layer; the saved AWQ format is unchanged. See the packing helper.

Format enum/value Weights Scale/zero storage Key differences
FORMAT.GEMM, "gemm" qweight: INT32 [K, N/8] scales: floating [K/G, N]; qzeros: INT32 [K/G, N/8] Weights and zero points pack output columns using classic AWQ INT4 interleaving. Broadest AWQ runtime compatibility.
FORMAT.GEMV, "gemv" qweight: INT32 [N, K/8] scales: FP16 [N, Z*8]; qzeros: INT32 [N, Z] Input-axis packing and padded group storage; not GEMM with a different dispatch flag.
FORMAT.GEMV_FAST, "gemv_fast" qweight: INT16 [N/4, K] for the fast packer scales and qzeros: FP16 [Z*8, N] Permuted/interleaved fast layout. Despite its name, qzeros holds the floating additive offset -scale * integer_zero, not packed integer zero codes.
FORMAT.LLM_AWQ, "llm-awq" Same fast INT16 layout FP16 scales and scaled_zeros LLMAwqLinear subclasses the fast implementation and changes the zero-buffer name/format identity.
FORMAT.MARLIN, "marlin" Backend/legacy compatibility format Loader/runtime-specific handling Advertised by the AWQ format registry, but fresh BaseQModel.quantize() rejects FORMAT.MARLIN as deprecated. Use GEMM plus Marlin inference instead.
FORMAT.BITBLAS, "bitblas" BitBLAS representation BitBLAS quantization state Supported format/backend route; do not treat it as a synonym for raw GEMM tensors. The public quantization lifecycle can select the BitBLAS packer.

For INT4 GEMM, packing uses:

AWQ_ORDER = [0, 2, 4, 6, 1, 3, 5, 7]
AWQ_REVERSE_ORDER = [0, 4, 1, 5, 2, 6, 3, 7]

pack_awq() preserves this established INT4 layout and rejects INT4 dimensions not divisible by eight. For other 2–8-bit widths it uses a continuous little-endian bit stream, including codes crossing 32-bit word boundaries; reverse_awq_order() is a no-op outside INT4. Support for these helper layouts does not make every optimized inference kernel support those widths.

Eight-bit packed readers likewise use natural code order for both weights and zeros, not the legacy INT4 interleave. This corrects existing 8-bit portable AWQ reads without changing the INT4 format or enabling native INT4-only kernels for 8-bit weights.

The lower-level pack_awq_columns() packs complete words and ignores an incomplete trailing group. Callers must enforce shape compatibility; it is not an automatic-padding API. GEMV_FAST's pack_intweight() uses interleave=4, kstride=64, and additional input/output permutations. The current native and CPU fast readers enforce K % 64 == 0 and N % 128 == 0.

calculate_zeros_width() pads the group axis in blocks of eight, with additional multipliers for group sizes 64, 32, and 16. The fast formats' padded scales/offsets must be preserved. Do not infer scale shapes from K/G alone or change only version in JSON to convert formats.

Native AWQ packing validates the actual stored-scale cast before replacing module buffers. GEMM preserves FP16/BF16 source scales and otherwise stores FP16; GEMV and GEMV_FAST/LLM_AWQ store FP16. AllSpark uses its existing scale-buffer dtype, or the compute dtype if no buffer exists. A nonfinite cast result or nonzero scale that casts to zero raises ValueError instead of silently changing packed codes. For FP16 storage, 65520 overflows and 2**-25 rounds to zero; 65504 and 2**-24 remain representable. GEMV_FAST also validates its scaled zero offset before installing buffers. Inspect the scale range and choose a backend/storage dtype that can represent it; do not independently clamp scales after computing codes. Portable Torch's source-precision contract is separate. See the pack-time validation source.

The non-quantized bias does not follow the scale-storage dtype. Native GEMM, GEMV, and GEMV_FAST/LLM_AWQ packers clone it at the source layer's precision, preserving finite FP32 values that would overflow/underflow in FP16. Their loader declarations, plus Triton GEMM's, keep a saved bias from being narrowed to the requested model dtype. Native GEMM adds the stored bias to its FP32 product before the output cast; GEMV-family runtime preparation moves/contiguates bias without narrowing it, but their kernel product is still FP16, so this is not a guarantee of FP32 product precision or cancellation after an overflowing GEMV product. Scale/offset validation above is unchanged, and already-corrupted checkpoint biases require regeneration from valid source weights. See the bias-storage contract.

AWQConfig writes both version and format for AWQ compatibility and derives zero_point from not sym. Config loading also recognizes LLM-AWQ metadata conventions. For serialized fields, model shards, and external checkpoint compatibility, see packing formats and compatibility.

AWQ inference kernels

Selection and compatibility

select_quant_linear() in gptqmodel/utils/importer.py normalizes backend aliases using the quantization method, validates the method/format pair, and filters candidate classes by their declared contracts and runtime availability. AUTO uses the registry's per-format priorities and validation; it is not a startup benchmark. Multi-device selection validates candidates on each exact device ordinal. Trainable auto-selection additionally filters for training support.

Use the canonical method-specific backend names below. Compatibility aliases such as gemm, gemm_triton, gemv, gemv_fast, torch, torch_awq, torch_fused_awq, torch_int8_awq, bitblas_awq, marlin, and machete are normalized for AWQ; the filename spelling is not necessarily the canonical backend string. In particular, torch_aten_kernel_awq.py corresponds to AWQ_TORCH_ATEN / "awq_torch_aten".

The following table lists AWQ kernel modules' declared contracts. -1 means one group across the input dimension, subject to backend validation of the resolved width. Dtype entries are class validation dtypes; runtime casting and native-extension caveats follow the table. All rows refer to AWQ, not similarly named GPTQ kernels.

Module / class Canonical backend/string Input formats Bits; group sizes Device/platform and declared dtype
allspark_awq.py / AllSparkAwqLinear ALLSPARK / allspark GEMM 8; -1 Explicit W8A16, Linux, FP16/BF16; CUDA capability 8.0–12.0 or ROCm gfx1200/gfx1201 only; K/N divisible by 16, int32 words, no shards/training/adapters. CUDA and HIP use different repacking layouts.
gemm_awq.py / AwqGEMMLinear AWQ_GEMM / awq_gemm GEMM 4; -1,16,32,64,128 CUDA; all platforms; FP16/BF16. Native CUDA extension required.
gemm_awq_triton.py / AwqGEMMTritonLinear AWQ_GEMM_TRITON / awq_gemm_triton GEMM 4; -1,32,64,128 CUDA/ROCm; Linux/Windows; FP16. Requires K % 32 == 0, N % 8 == 0.
gemv_awq.py / AwqGEMVLinear AWQ_GEMV / awq_gemv GEMV 4; -1,64,128 CUDA only, all platforms; FP16. Requires N % 64 == 0; native runtime caveat below.
gemv_awq.py / AwqGEMVTritonLinear AWQ_GEMV_TRITON / awq_gemv_triton GEMV 4; -1,64,128 Explicit-only (AUTO priority 0), CUDA/ROCm, Linux; FP16 and INT32 words. Requires N % 8 == 0, no automatic padding; normalized group must be 64 or 128.
gemv_fast_awq.py / AwqGEMVFastLinear, LLMAwqLinear AWQ_GEMV_FAST / awq_gemv_fast GEMV_FAST; LLM_AWQ through subclass 4; 128 CUDA only, all platforms; FP16; INT16 qweight; K % 64 == 0, N % 128 == 0.
gemv_fast_awq.py / AwqGEMVFastTorchLinear, LLMAwqTorchLinear AWQ_TORCH / awq_torch GEMV_FAST; LLM_AWQ through separate reader 4; 128 CPU, CUDA, ROCm; all platforms; FP16; INT16 qweight; K % 64 == 0, N % 128 == 0; no automatic padding/training. AUTO fallback after regular candidates.
marlin_awq.py / AwqMarlinLinear MARLIN / marlin GEMM, MARLIN 4 or 8; -1,32,64,128 NVIDIA CUDA, Linux, compute capability >= 8.0; FP16/BF16. Not ROCm.
machete_awq.py / AwqMacheteLinear MACHETE / machete GEMM, MARLIN at class level 4 or 8; -1,64,128 CUDA, Linux, Hopper SM90 only; FP16/BF16; K % 64 == 0, N % 128 == 0; desc_act=False.
bitblas_awq.py / AWQBitBlasKernel (AwqBitBLASLinear alias) AWQ_BITBLAS / awq_bitblas GEMM, BITBLAS 4; -1,32,64,128 CUDA, Linux/Windows; FP16/BF16; K and N divisible by 16.
torch_awq.py / AwqTorchLinear AWQ_TORCH / awq_torch GEMM 2–8; -1,16,32,64,128 All device/platform families; FP16/BF16; N % 8 == 0. Torch operation support on the device still matters.
torch_fused_awq.py / TorchFusedAwqLinear AWQ_TORCH_FUSED / awq_torch_fused GEMM 4; -1,16,32,64,128 CPU/XPU, all platforms; FP32/FP16/BF16 inherited contract, with fused-op dtype/device restrictions.
torch_aten_kernel_awq.py / TorchAtenAwqLinear AWQ_TORCH_ATEN / awq_torch_aten GEMM 4; 32,64,128,256 CPU, all platforms; FP16/BF16; local ATen CPU int4pack ops required.
torch_cpu_avx512_awq.py / TorchCpuAvx512AwqLinear AWQ_TORCH_CPU_AVX512 / awq_torch_cpu_avx512 GEMM 4; 16,32,64,128,256,512 CPU, Linux, AVX512-BF16; BF16 only; K % 2 == 0, N % 32 == 0, complete groups.
torch_int8_awq.py / TorchInt8AwqLinear AWQ_TORCH_INT8 / awq_torch_int8 Retired No usable runtime Validation fails and construction raises NotImplementedError; persistent INT4-to-INT8 expansion is disabled.
mlx_awq.py / MlxAWQLinear, MlxAWQGroup16Linear MLX / mlx Packed-transfer holders for GEMM, GEMV, GEMV_FAST, LLM_AWQ, MARLIN, BITBLAS Depends on source holder/transfer path Apple Silicon, macOS arm64, Metal; mlx and mlx_lm required. These are MLX runtime modules, not interchangeable Torch qlinear classes.

The file list is not the entire registry: exllamav2_awq.py also supplies AwqExllamaV2Linear (AWQ_EXLLAMA_V2 / awq_exllama_v2) for 4-bit GEMM with groups -1,32,64,128, CUDA on Linux/Windows, K/N divisible by 32, and FP16/BF16/FP32 activation/output dtypes. It unpacks/reorders/re-packs AWQ codes and retains FP16 scales, but no longer unconditionally narrows activations to FP16. Native GEMM accumulates in FP32 and returns the input activation dtype. Group 16 is deliberately disabled; ROCm is not declared supported. It requires ExLlamaV2 runtime initialization and scratch space.

swordfish.py additionally defines AwqSwordfishLinear under SWORDFISH / swordfish, with its own validation/repacking policy. Consult Kernel backends reference for that implementation and the complete registry, rather than assuming all AWQ backend names are aliases for native GEMM.

Native GEMM

During quantization, packing-kernel resolution checks each module's actual input/output widths at preprocessing and packing. When the selected kernel cannot pack a GEMM shape that AwqTorchLinear accepts, that module uses the portable packer and logs a warning, preserving the GEMM checkpoint layout. For example, Mamba's 8192 -> 288 projection can avoid the native GEMM output-width restriction. If the portable packer also rejects the shape, the original kernel's module-specific error remains. This does not remove format/group constraints; QQQ instead rejects unsupported tiles early because its portable reader shares them.

AwqGEMMLinear packs the GEMM layout using the shared AWQ column packer. Its autograd function saves packed weights/zeros/scales and input metadata, not a full activation tensor. Empty input returns an appropriately shaped output without launching the native kernel.

The wrapper temporarily dequantizes for 640 or more flattened input rows when fp32_accum=True, and always for more than 1024 rows. That dense-current-call branch uses FP32 output/reduction and adds bias before the final activation-dtype cast, then releases the dense weight. Other calls use the native GEMM with split_k_iters=8; biased native calls request FP32 output before adding bias. These thresholds use all leading input dimensions flattened into rows, not one sequence-length field. Input-gradient backward also temporarily dequantizes; packed integer weights do not become trainable floating-point parameters.

Evaluation mode does not disable autograd: native GEMM keeps the graph when input or bias gradients are required and returns bias gradients from an FP32 reduction, including zero gradients for empty rows. Native and Triton GEMM also avoid creating inference tensors when an adapter is attached. Built-in LoRA composition returns a new sum rather than modifying the kernel output; see adapter runtime loading.

KernelConfig.fp32_accum defaults to enabled and controls the native GEMM accumulation choice. Disabling it is a numerical/performance choice, not a different checkpoint format.

The native CUDA GEMM fused split-K epilogue uses 64-bit output indexing for result tensors whose element count exceeds the signed 32-bit range.

Triton GEMM and dispatch policy

Triton requires version >= 2.0.0; when Python's GIL is disabled, validation requires >= 3.4.0. The qlinear's declared dtype is FP16, although its forward wrapper can cast other input tensors to FP16 and restore output dtype. Low-level Triton GEMM requires contiguous rank-2 FP16 input/scales, INT32 weights/zeros, matching devices, and consistent packed shapes. Group 16 is not supported.

Biased fused GEMM requests FP32 output, adds bias before a single FP16 conversion, and keeps this wide epilogue even with fp32_accum=False. The dense path uses the shared wide matmul helper. Backward preserves input and bias gradients, including empty-row bias gradients. See numeric epilogues and input gradients.

The Triton qlinear preserves those input/bias gradients in .eval() too; only ordinary no-gradient, adapter-free evaluation uses its internal inference-mode context. Native and Triton AWQ GEMM restore the leading batch dimensions without a full-output copy when none of the custom forward's inputs need gradients. Tracked batched results still materialize the reshaped output for in-place accumulation and backward.

AwqTritonPlan records path, tile sizes, split-K, and optional warps/stages. Defaults are tiles (32,32,32), split-K 8, and implicit launch warps/stages. Historical dispatch uses fused execution for M <= 128 and temporary dequantization plus dense matmul for M > 128.

KernelConfig.awq_triton_schedule selects "auto" (default) or "legacy". Auto can use exact measured device/shape/dtype rules; separate built-in tables are gated on the source's precise RTX 4090 and Radeon AI PRO R9700 (gfx1201) profiles and FP16 input/compute/output with FP32 accumulation. The HIP profile includes board name, gfx architecture, compute-unit count, and memory size; a different board or dtype does not inherit its rules. Unlisted cases retain legacy behavior; there is no general promise that one GPU family's threshold applies to another. Runtime measured rules can be registered with register_awq_triton_rule(), and explicit plans are validated before launch. Training deliberately keeps legacy scheduling.

The qlinear keeps the schedule mode separate from an explicit launch plan. Without a direct mode override, it snapshots self.kernel.awq_triton_schedule from its own module policy; the explicit plan stays unset. This honors per-load AwqTritonConfig(schedule="legacy") (or "auto") even when the process default differs, rather than consulting the process default later. For direct qlinear construction, schedule_mode (alias awq_triton_schedule_mode) takes "auto"/"legacy" and overrides that module mode, while schedule (alias awq_triton_schedule) takes an AwqTritonPlan or plan dictionary. A mode string is never an explicit plan. See the qlinear wrapper.

The scheduler has bounded plan/warmup caches (256 entries) and separate helpers to clear plans, warmup state, or registered rules. CUDA Graph capture performs cache lookup only and requires a successful eager prewarm for the exact plan/request; discovery, compilation, and benchmarking do not occur during capture. candidate_plans() describes an offline search space rather than performing online autotuning.

The low-level execution entry point is:

def awq_gemm_triton(
    input: torch.Tensor,
    qweight: torch.Tensor,
    scales: torch.Tensor,
    qzeros: torch.Tensor,
    split_k_iters: int,
    block_size_m: int = 32,
    block_size_n: int = 32,
    block_size_k: int = 32,
    fp32_accum: bool = FP32_ACCUM,
    output_dtype: torch.dtype | None = None,
    num_warps: int | None = None,
    num_stages: int | None = None,
) -> torch.Tensor:
    ...

FP32 accumulation defaults to enabled here too. Split-K 1 overwrites an uninitialized output buffer; multi-split execution starts with a zeroed accumulation buffer and uses atomic additions. output_dtype controls the final cast, not an alternative packed representation.

GEMV, GEMV_FAST, and LLM-AWQ

Native GEMV flattens leading input dimensions and chooses awq_gemv_forward() for up to eight rows or awq_gemmv2_forward() for more than eight. Computation/scales are FP16; the wrapper restores the original input dtype where it casts inputs.

Its validator resolves group_size=-1 to K and then accepts only normalized groups 64 or 128. Thus -1 is not general row-wide support for arbitrary input widths in this native backend. The class-wide N % 64 == 0 requirement protects the GEMMV2 path even for requests currently using at most eight rows.

GEMV_FAST uses the decode kernel only for rank-3 input with batch_size < 8 and n_tokens == 1. Other shapes, including flattened rank-2 input, use the prefill entry point. Its fast packer requires K % 64 == 0 and N % 4 == 0 for its reshapes; current class validation is stricter, requiring N % 128 == 0 and group size 128. Runtime preparation makes weights, scales, offsets, and bias contiguous and resident on the execution device. LLMAwqLinear uses that same execution logic with the scaled_zeros buffer.

On CUDA, GEMMV2 and the GEMV_FAST/LLM-AWQ prefill kernel use 64-bit activation and output offsets so large flattened batches can address elements beyond the signed 32-bit range.

Portable GEMV: BACKEND.AWQ_GEMV_TRITON (string "awq_gemv_triton"; legacy GEMV_TRITON / "gemv_triton") reads the same INT32 GEMV checkpoint without the native CUDA extension. It is an explicit-only Linux backend for CUDA/ROCm with FP16 work inputs/scales, output width divisible by 8, no automatic padding, and normalized group size 64 or 128 (-1 is accepted only when K resolves to one of those). It checks packed-buffer co-location, requests FP32 output when biased, adds bias before casting to the input dtype, then applies an optional LoRA adapter. When input or bias gradients are needed, it selects temporary differentiable dequantization/matmul rather than the inference-only fused kernel. Ada/RDNA4 launch tables target decode (M == 1); wider batches are not promised a speedup. It does not read GEMV_FAST or LLM_AWQ layouts.

CPU GEMV: BACKEND.AWQ_TORCH selects AwqGEMVTorchLinear for FORMAT.GEMV. CPU-only AUTO uses it when no ordinary AUTO candidate matches, and GEMV quantization can finish with a runnable CPU in-session model before save(). It declares 4-bit INT32 packing, FP16 activations, group sizes -1, 64, or 128, output width divisible by 64, no automatic padding, and no training. It reconstructs weights only for the current FP32 matmul, adds bias before the final output cast, and retains only packed GEMV state. Accelerator AUTO priority is unchanged. See the CPU GEMV reader.

Portable GEMV_FAST / LLM_AWQ: AWQ_TORCH selects AwqGEMVFastTorchLinear or LLMAwqTorchLinear on CPU, CUDA, and ROCm. Both require 4-bit INT16 packing, FP16, group size 128, K % 64 == 0, and N % 128 == 0. They read the original interleaved weights and FP16 offsets (qzeros or scaled_zeros) and reconstruct weights for the current FP32 matmul and bias addition. The readers have priority zero and serve as AUTO fallbacks where no regular candidate qualifies. NVIDIA AUTO continues to prefer the native kernel when compatible. CPU and ROCm quantization use AWQ_TORCH to pack these formats, then keep the portable reader for in-session inference and subsequent reloads on those devices. See the portable readers and packer selection.

The low-level awq_gemv_triton() helper also accepts non-power-of-two groups that are positive multiples of eight and divide K: its power-of-two K tile uses the group's largest power-of-two factor, keeping every tile within a scale/zero boundary (for example, group 96 uses tile 32). This helper-level correction does not expand the qlinear's declared group-size choices above.

The Triton GEMV kernel launches flattened activation rows on grid axis 0, so a prefill beyond 65,535 rows can run without hitting the secondary-axis limit on CUDA and ROCm. Activation and output row offsets use 64-bit arithmetic for tensors beyond the signed 32-bit element range. See the Triton GEMV implementation.

from gptqmodel import BACKEND, GPTQModel

# A compatible four-bit AWQ GEMV checkpoint, not a GEMM/GEMV_FAST checkpoint.
model = GPTQModel.load(
    "/models/awq-gemv", backend=BACKEND.AWQ_GEMV_TRITON, device="cuda:0"
)

ROCm backend selection: Use AUTO or AWQ_TORCH for compatible GEMV_FAST and LLM_AWQ checkpoints. For ordinary GEMV, explicitly select AWQ_GEMV_TRITON. GEMM checkpoints can use AUTO, AWQ_GEMM_TRITON, or portable Torch. The native AWQ_GEMV and AWQ_GEMV_FAST backends require NVIDIA CUDA. The Triton GEMM requires Triton >= 2.0.0 (>= 3.4.0 with the GIL disabled).

The GEMV refusal follows the target device, not the installed PyTorch build. In a ROCm process, set device="cpu" to load a compatible GEMV checkpoint with AUTO or AWQ_TORCH. An unresolved target still requires the explicit Triton GEMV backend.

Marlin and Machete

Marlin loads AWQ affine integer codes and repacks/permutates weights, scales, and zero points into its runtime layout. The registered bias keeps its checkpoint dtype and order through load, post-initialization, and repeated FP16/BF16 forwards. Each launch derives a temporary FP32, padded/permuted bias operand and adds it to the native FP32 accumulator before the output cast, preserving cancellation that would be lost by narrowing a wide bias to FP16 first. This applies to GPTQ Marlin too; see Marlin bias storage and the source contract. AWQ uses empty activation-order indices. Workspace and indexing buffers are runtime-only and must be initialized before forward. NVIDIA compute capability >= 8.0 is checked on the target device, and ROCm is rejected.

Explicit MARLIN can enable runtime tile padding for supported 4-bit tail shapes. Auto selection deliberately leaves tile-misaligned shapes to another backend. Output width must still be divisible by 32 // bits; padding cannot repair a partially packed integer word. Non-4-bit tile-misaligned shapes are rejected. Thus permissive generic divisibility metadata does not remove these checks.

Machete removes AWQ interleaving, packs/prepacks integer weights for its matrix product, and converts zero points to additive -scale * zero values. Its static runtime validator is Hopper-only, exactly compute capability (9,0), not “Hopper or newer.” It also checks opt-in shared memory. Blackwell is not accepted by this validator. Machete and Marlin classes may advertise MARLIN compatibility, but the high-level loader restricts legacy MARLIN checkpoints to their matching Marlin route (or supported MLX transfer); class metadata alone does not override loader policy.

AWQ Machete accepts only group_size=-1, 64, or 128. Group size 32 is rejected during selection because the native mainloop requires groups aligned to its 64-column K tile; AUTO can choose another compatible backend instead of failing on the first Machete forward.

BitBLAS

AWQBitBlasKernel builds an unsigned affine BitBLAS quantization config even for symmetric AWQ, because stored AWQ codes retain zero points. It can pack from pseudo-quantized weights/scales/zeros or repack a raw AWQ GEMM module by unpacking codes and reversing AWQ interleaving.

The optional dependency in this revision is bitblas==0.1.0.post1; the runtime minimum is 0.1.0.post1. BF16 is an interface dtype, but the shared implementation computes low-bit BF16 modules in FP16 and casts at the module boundary because that BitBLAS version's bundled CUTLASS cannot compile its generated BF16 dequantization path. Target detection also normalizes unsupported targets; do not infer native coverage of every newer NVIDIA architecture from a CUDA device declaration. See Inference backends for installation and backend selection.

Portable Torch and fused CPU/XPU paths

AwqTorchLinear uses dequantize_gemm() followed by a wide reduction, keeping packed weights as persistent state and constructing dense weights only for the current computation. It accepts 2–8-bit GEMM packing and preserves input dtype at output. FP16/BF16 inputs ordinarily decode in that dtype; other input dtypes are cast to FP16 internally and restored at output. Before decoding, it checks whether stored scales and the largest legal code product remain finite and nonzero-representable. Unsafe ranges select FP32 decode, or FP64 if FP32 is also insufficient, then a wide product/bias epilogue before the final output cast. Thus a wide or tiny checkpoint scale does not become inf or zero before a representable dot product. Its current forward() remains the dense reference path and does not invoke its CPU-AVX512 helper methods. Select AWQ_TORCH_CPU_AVX512 explicitly, or use AUTO to choose a compatible packed kernel, for that optimized BF16 path. The portable implementation does not promise the latency or peak workspace of a fused packed kernel.

Compiled and CUDA-graph calls use precomputed scale-range checks to choose the same decode dtype as eager execution, rather than promoting every call to FP64. Initialization, checkpoint loads, device/dtype moves, and cache clears refresh those checks outside graph execution. After an in-place scale edit, call clear_weight_cache() before compiling or capturing again. See the decode implementation.

The range decision follows scale identity/version, activation dtype, and bit width, and resets with clear_weight_cache(). After unversioned inference-tensor edits, clear the cache explicitly. An uncached decision during compilation or CUDA graph capture uses wide decode conservatively. See Torch AWQ decode range.

AwqTorchLinear declares eager CUDA-graph capture support. A cold eager call waits for the shift-table copy before publishing it, so a concurrent forward on another stream cannot read unfinished offsets. During capture, a newly built table stays local to the graph; compiled calls use no shared cache. See the capture-safe extraction code and kernel capabilities.

dequantize_gemm() now reconstructs packed weights in group-aligned input-row tiles instead of expanding group scales/zeros and column reorder indices across the full matrix. The temporary decode workspace is bounded by a tile; the dense weight still exists only for the current forward, never as a persistent cache. The result matches the prior arithmetic for bits 2, 3, 4, and 8, and ragged input-feature tails retain their last group's scale while returning exactly in_features rows. A missing packed zero-point tensor raises ValueError; group_size=-1 treats the whole row as one group. See the AWQ dequantization flow and measured memory. These are portable Torch decoding changes, not new configuration flags or automatic selection of a fused backend.

Stored scales and bias retain their source precision through pack(), post_init(), and repeated forwards with different activation dtypes. The model loader restores Torch AWQ's checkpoint scales and bias after Accelerate's model-dtype cast. Forward converts scales only for the current computation and moves bias without narrowing it; a wider or incomparable-dtype bias is added to the FP32 reduction before the final output cast. An FP16 call therefore cannot permanently zero a tiny BF16 scale used by a later BF16 call, and a stored FP32 bias is not overflowed or rounded before cancellation. These guarantees concern the portable Torch implementation, not the separate operand contracts of native AWQ GEMM/GEMV kernels. See the runtime precision flow.

Groups larger than the decode tile budget are split across bounded row tiles instead of forcing a full-group temporary, including whole-row group_size=-1. With autograd enabled and trainable scales, partial groups use out-of-place integer differences so later tile updates cannot invalidate saved tensors needed for scale gradients. No-grad execution can still use fused output writes. These bounds cover decode workspace, not the call-local dense output matrix.

The shared dequantization helper also supports FP32 computation for Paro's dense fallback. It multiplies codes by temporary FP32 scales directly, avoiding a half-precision intermediate that could underflow or overflow. Both callers retain dense weights only for the consuming operation.

TorchFusedAwqLinear reverses AWQ packing into the CPU/XPU fused representation. CPU uses ATen int4 packing, and XPU uses _weight_int4pack_mm_with_scales_and_zeros. When fused execution is unavailable, it falls back to temporary weight reconstruction and matrix multiplication. After a training forward, switching to eval() lets the next forward select fused INT4 again when the fused operators are available and the input needs no gradients. AWQ has no GPTQ g_idx activation reorder to apply. Fused AWQ implementation

TorchAtenAwqLinear requires local aten::_convert_weight_to_int4pack_for_cpu and aten::_weight_int4pack_mm_for_cpu availability. On its first eligible inference call it transforms packed weights and builds separate affine metadata for the FP16/BF16 activation contract, preserving the checkpoint scales buffer. Once transformed, it requires one consistent activation dtype per layer; that contract is keyed by _cpu_transform_dtype, not the checkpoint scale dtype. TorchFusedAwqLinear's CPU transform also keeps checkpoint scales separate from runtime metadata. Both request an FP32 product for biased FP16/BF16 execution so bias is added before the final output cast. A non-fused/training path uses temporary dequantization. Availability of a symbol in one PyTorch version does not guarantee it exists in another build.

Native AwqGEMMLinear also preserves loaded scales and bias through post_init() and forwards: it creates contiguous activation-dtype operands for the current call instead of replacing registered buffers. Alternating FP16/BF16 calls therefore does not permanently narrow the checkpoint state. This is a storage guarantee, not a promise that every native AWQ backend shares the same compute or bias-epilogue contract. See AWQ runtime precision.

TorchCpuAvx512AwqLinear requires AVX512-BF16 plus the shared CPU extension's packing/GEMM ops. It keeps scales and bias in FP32, consumes BF16 activations, and uses FP32 accumulation. For FP16 execution, use a compatible ATen path instead of forcing this BF16-only backend. It is trainable: train(True) switches forward() from the no-grad fused kernel to a dequantized-matmul training fallback, with gradients intact; the AWQ FP32 bias is added in place so the sum rounds once to the activation dtype, matching the ATen path. The cached fused-vs-dequantized dispatch resets on mode changes, so an eval forward does not pin later training forwards to the fused kernel. Its transformed representation stays quantized; it does not retain an expanded INT8 or floating weight cache.

After the AVX512 inference transform, module moves such as .to(torch.bfloat16), .half(), or .float() preserve the exact FP32 scale/bias operands instead of rounding them through the requested lower precision. This prevents scale drift and the native absmax must be float32 failure on a later BF16 inference call; it does not add FP16 activation support. clear_weight_cache() restores captured checkpoint tensors when available, or widens affine operands to FP32 if the layer remains transformed without a snapshot. Before inference or projection fusion transforms the layer, it captures a non-persistent checkpoint snapshot including bias. After the runtime layout has a verified bit-exact inverse, the layer can release the checkpoint qweight copy and reconstruct it on demand with Torch operations. See checkpoint saving and projection fusion.

TorchInt8AwqLinear is retired: validation returns failure and construction raises because persistent INT4-to-INT8 expansion is disabled. Select AUTO or a compatible packed backend; use AWQ_TORCH for GEMM-format portable fallback. Changing backend aliases cannot reactivate the retired implementation.

MLX on Apple Silicon

MLX inference is selected with BACKEND.MLX, not an AWQ_MLX enum. Validation requires macOS (Darwin), arm64, importable mlx.core/mlx_lm, and available Metal. The optional dependency group in this revision declares mlx>=0.32.4, with mlx_lm>=0.31.3 for Python < 3.11 and mlx_lm>=0.32.2 for Python >= 3.11. Install the extras matching the release:

pip install 'gptqmodel[mlx]'

The checkpoint-holder classes in qlinear/mlx.py validate source tensor layouts before conversion. AwqMlxQuantLinear accepts GEMM, bits 2–8, groups -1,16,32,64,128, FP16 checkpoint scales, and no activation ordering; separate holders cover GEMV, GEMV_FAST, LLM_AWQ, MARLIN, and BITBLAS. Fast holders use INT16 qweights and require K % 64 == 0, N % 8 == 0; GEMV holders require K % 32 == 0, N % 64 == 0. Holder forward() is not a Torch inference implementation: conversion must replace holders with MLX modules.

utils/mlx.py first attempts packed transfer without requantization. GEMV, fast, LLM_AWQ, MARLIN, and BITBLAS routes require transferable packed state instead of silently falling back to generic conversion. A generic GEMM conversion fallback can dequantize/requantize when exact packed transfer is unavailable; this should not be described as a universally lossless conversion.

At runtime, MlxAWQLinear wraps an affine quantized linear and casts its output to the input activation dtype. MlxAWQGroup16Linear has exact group-16 metadata handling and packed Metal execution: one-row decode uses its packed kernel; small BF16 prefill (1 < rows <= 16, N <= 2048) uses a tiled packed path; other prefill uses the parent group-16 implementation. It supports storage widths 2,3,4,5,6,8 in its direct constructor; source 7-bit storage is mapped to 8-bit MLX codes by transfer policy. Group-16 transfer requires input width divisible by 32. FP16 and BF16 are separate runtime paths; checkpoint-holder FP16 validation does not imply BF16 activations should be downcast permanently. See MLX and Apple Silicon.

Native AWQ runtime helpers

gptqmodel/utils/awq.py exposes the shared native runtime through the central extension API. Its extension is named gptqmodel_awq_ops, uses the gptqmodel_awq namespace, and declares python_abi_dependent=False. Required registered operations include GEMM, FP32-reduction GEMM, GEMMV2, GEMV, fast prefill/decode, and dequantization.

Helper Purpose
awq_runtime_available() -> bool Ask the central extension API whether the AWQ runtime is available.
awq_runtime_error() -> str Empty string when available; otherwise the extension error or "CUDA AWQ runtime unavailable.".
prewarm_awq_extension() -> bool Explicitly request AWQ extension loading/building before a latency-sensitive operation.
clear_awq_extension_cache() -> None Clear the loader's AWQ cache through the shared singleton; not an instruction to delete all on-disk build artifacts.

The operation wrappers have these source signatures:

def awq_gemm_forward(input, qweight, scales, qzeros, split_k_iters, fp32_accum: bool):
    ...
def awq_dequantize_weights(qweight, scales, qzeros, split_k_iters, thx, thy, dbg):
    ...
def awq_gemmv2_forward(input, qweight, scales, qzeros, group_size, split_k_iters):
    ...
def awq_gemv_forward(input, qweight, scales, qzeros, group_size):
    ...
def awq_fast_gemm_forward_prefill(input, qweight, scales, qzeros):
    ...
def awq_fast_gemv_forward_decode(input, qweight, scales, qzeros, m, n, k, group_size):
    ...

These are internal packed-tensor interfaces, not a calibration API. Their argument order is significant: qlinear callers arrange their stored buffers to match these wrappers.

The JIT sources are gptqmodel_ext/awq/torch_bind.cpp, quantization/gemm_cuda_gen.cu, quantization/gemv_cuda.cu, gemm_fast_cuda_entry.cu, and gemv_fast_cuda_entry.cu under that extension directory. Include-path discovery checks for cusparse.h. The loader sets requires_cuda=True; on NVIDIA, building these kernels requires CUDA-enabled Torch and a suitable CUDA development toolchain, not just runtime libraries. The shared loader has ROCm-aware build handling, but that does not make this extension's CUDA-specific kernels portable. See the ROCm caveat above.

Control Role
GPTQMODEL_AWQ_BUILD_ROOT Override the AWQ JIT build directory.
GPTQMODEL_AWQ_FORCE_REBUILD Force-rebuild control passed to the shared loader.
GPTQMODEL_EXT_VERBOSE Verbose extension build output.
KernelConfig.fp32_accum Native/Triton GEMM numerical accumulation policy; defaults enabled.
KernelConfig.awq_triton_schedule Triton schedule mode, default auto; alternative legacy.

awq_gemm_pack_backend(device_type, is_rocm, weight_only), defined in utils/backend.py, returns AWQ_TORCH for weight-only lifecycle exports, ROCm, or any non-CUDA device; otherwise it returns AWQ_GEMM. Both pack the same AWQ GEMM layout. The public model lifecycle uses this to avoid requiring CUDA headers just to write a GEMM checkpoint on another device. NPU AWQ quantization explicitly requires GEMM format.

The source revision declares Python >= 3.10, torch>=2.13.0,<2.14.0 on Python <3.11 or >=2.15.0 on Python >=3.11, and transformers>=5.20.2 on Python <3.11 or >=5.22.2 on Python >=3.11. These are revision-specific package requirements, not minimum versions for the AWQ algorithm itself. Optional kernel requirements remain additional constraints; follow Installation and quickstart for the installed release.

GSQ refinement

AWQConfig.gsq refines scalar AWQ export after the ordinary solve on 4-bit GEMM, GEMV, GEMV_FAST, or LLM_AWQ formats; non-GEMM formats additionally require group_size of -1, 32, 64, or 128. AWQConfig.gsq_training selects the staged dense-Llama/Qwen3 block-training recipe with the AWQ initializer, exporting 4-bit GEMM with AWQ's scaled weights, integer zero points, and packer. gsq and gsq_training are mutually exclusive on one config. See GSQ.

Pitfalls and troubleshooting

Symptom or assumption What to check
“AWQ defaults to asymmetric quantization.” This implementation inherits sym=True. Choose sym=False explicitly if needed; serialized zero_point is its inverse.
“mse.norm=0 disables AWQ clipping.” It does not. AWQ clipping is controlled by the processor's apply_clip, and the clip scratch budget is mse.clip_grid_budget.
“SmoothMSE(steps=...) changes AWQ's 20-ratio search.” It does not; generic smoother normalization is separate from AWQ's native scale/clip searches.
Unexpected quantization memory use Distinguish processor chunk estimates, clip scratch budgets, the full replay input and all reference chunks moved to the scoring device, model weights, and multi-GPU cloning/offload memory. Chunked search reduces some intermediate allocations but none of these settings is a total-memory cap.
Clip assertion on a narrow projection _compute_best_clip() batches output channels in 256/64 and asserts divisibility. Generic kernel metadata does not remove calibration-time shape requirements.
Bad or missing MoE calibration coverage Inspect raw/retained token counts, selected scale feature, routing/capture policy, and fallback logs. A successful direct fallback is not evidence that normal AWQ scale search ran.
Native extension missing or build fails Inspect awq_runtime_error(), CUDA/toolchain/header availability, build root, and verbose output. GEMM-format portable Torch packing/inference does not require that native extension.
ROCm is listed but native GEMV fails Runtime extension support is stricter than GEMV class metadata. Prefer a validated GEMM-format ROCm path rather than assuming CUDA native sources work on HIP.
Triton rejects group 16 Supported groups are -1,32,64,128; select another compatible backend or quantize with a compatible group size.
CUDA Graph capture requests eager prewarm Run the exact shape/device/dtype/request eagerly before capture; a different shape or plan's warmup is not sufficient.
Explicit Marlin tails fail Check packed output-word alignment, bit width, tile-padding policy, and target capability. Auto intentionally does not enable the explicit 4-bit padding route.
Machete fails on Blackwell This revision accepts exact SM90 and has no Blackwell image in its Machete validator. Use another backend.
CPU ATen dtype error after initial inference The transformed layer's scales/zeros match the first fused activation dtype. Keep that dtype consistent.
AVX512 path rejects FP16 It requires BF16 activations and AVX512-BF16, not merely an arbitrary AVX512-capable CPU.
AWQ_TORCH_INT8 fails immediately It is retired; use a packed implementation or portable GEMM fallback.
Renaming JSON format fields breaks load GEMM, GEMV, and fast layouts differ in axes, permutations, word dtype, padding, and zero representation. Metadata editing is not conversion.
A GEMM checkpoint is called “Marlin AWQ.” Usually this describes the inference backend. Inspect actual format/version and the loaded kernel, not the model repository name.
Comparing logged loss to perplexity Scale reconstruction MSE is a local search metric, not a task-level quality measure. Evaluate full-model behavior separately.

On ROCm, apply_clip() performs a best-effort release of unused allocator cache at each layer boundary after clip-search scratch has been freed. It runs with or without a caller-set split cap; the ROCm default no longer sets one. It does not change clipping math or release live tensors. NVIDIA CUDA retains its existing allocator cache. ROCm layer-cache cleanup source.

On Linux with glibc, the layer stage also returns unused host heap pages after layer completion. This helps large AWQ runs release freed calibration and packing buffers. Layer finalization and host memory

Source files

Research background: AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. The search grids, config names, and packing behavior documented here follow GPT-QModel rather than another library's AWQ API.

Paths below are relative to the verified source tree.

Area Key paths
Configuration and serialization gptqmodel/quantization/config.py, gptqmodel/quantization/__init__.py
AWQ lifecycle, scale/clip search, MSE gptqmodel/looper/awq_processor.py
Shared preprocessor planning gptqmodel/looper/module_preprocessor.py
AWQ graph transforms gptqmodel/quantization/awq/quantize/scale.py, gptqmodel/quantization/awq/modules/act.py
Packing and calibration helpers gptqmodel/quantization/awq/utils/packing_utils.py, calib_data.py, module.py, utils.py in that directory
Triton implementation/policy gptqmodel/quantization/awq/modules/triton/gemm.py, gemv.py, scheduler.py in that directory
Public model integration gptqmodel/models/auto.py, gptqmodel/models/base.py, gptqmodel/models/loader.py, gptqmodel/models/writer.py
Backend registry/validation gptqmodel/utils/backend.py, gptqmodel/utils/importer.py, gptqmodel/nn_modules/qlinear/__init__.py
Native AWQ runtime gptqmodel/utils/awq.py, gptqmodel/utils/cpp.py, gptqmodel/extension.py, gptqmodel_ext/awq/
Native GEMM/GEMV/fast gptqmodel/nn_modules/qlinear/gemm_awq.py, gemv_awq.py, gemv_fast_awq.py in that directory; gptqmodel/utils/gemv.py
Triton qlinear gptqmodel/nn_modules/qlinear/gemm_awq_triton.py, gemv_awq.py in that directory
Marlin gptqmodel/nn_modules/qlinear/marlin_awq.py, gptqmodel/utils/marlin.py
Machete gptqmodel/nn_modules/qlinear/machete_awq.py, gptqmodel/utils/machete.py
BitBLAS gptqmodel/nn_modules/qlinear/bitblas_awq.py, gptqmodel/nn_modules/qlinear/bitblas.py
Portable/fused Torch gptqmodel/nn_modules/qlinear/torch_awq.py, torch_fused_awq.py, torch_fused.py in that directory
CPU ATen/AVX512 gptqmodel/nn_modules/qlinear/torch_aten_kernel_awq.py, torch_aten_kernel.py, torch_cpu_avx512_awq.py, torch_cpu_avx512.py in that directory; gptqmodel/utils/cpu_features.py
Retired INT8 backend gptqmodel/nn_modules/qlinear/torch_int8_awq.py
Additional AWQ runtimes gptqmodel/nn_modules/qlinear/exllamav2_awq.py, gptqmodel/nn_modules/qlinear/swordfish.py
MLX runtime/transfer gptqmodel/nn_modules/qlinear/mlx_awq.py, mlx.py, mlx_group16.py in that directory; gptqmodel/utils/mlx.py, gptqmodel/utils/mlx_packing.py
Generic SmoothMSE consumers gptqmodel/quantization/gptq.py, gptqmodel/quantization/qqq.py, gptqmodel/quantization/rtn.py
Dependency requirements pyproject.toml, requirements.txt

See also

Next: reload your checkpoint, measure quality, or compare methods and dtypes.

Clone this wiki locally