Skip to content

Looper Processors

Jump to bottom
Qubitium edited this page Oct 27, 2026 · 3 revisions

Looper Processors

Documentation home · Quickstart · Lifecycle

On this page

Overview

Processors implement the algorithm-specific part of quantization: calibration hooks, statistics or activation capture, numerical conversion, reconstruction for sequential replay, packed-module installation, and adapter generation. The looper owns scheduling; the processor owns the work and state that scheduling invokes.

This reference covers every direct LoopProcessor subclass in gptqmodel/looper/ at source revision c1976918cfe3703fbcf6d1ce2440a82b26394375, including the planning-only ModulePreProcessor. For the full call graph, stages, device pools, offload policy, and resume transactions, read Quantization Lifecycle.

Workflow at a glance

Repeat layers and subsets. ModuleLooper schedules work while each processor defines its operations.

Reading the diagram: These are nested scopes, not consecutive lanes: repeat subsets inside each processor and layer. Native capture merges subsets; processor flags can skip capture or replay.

Text version of the workflow
  • Outer scope: each layer: Stage layer → Run processor subsets → Propagate outputs → Finalize layer work.
  • Inner scope: each subset: Preprocess targets → Capture if required → Process targets → Replay if required.

Open full-size SVG

Concepts: solve, replay, pack, finalize

These operations are deliberately separate:

Operation Typical owner Result
Plan/source annotation ModulePreProcessor.preprocess() Ordered decoder/padding/smoothing state, not quantized weights
Capture pre_process_fwd_hook() and optional capture extensions Hessian, bounded/raw activations, native inputs, or EoRA statistics
Solve/optimize process() or weight-only quantize_module() Reconstructed floating-point weights and method-specific packing payload
Replay Looper / ForwardExecutor Next-layer calibration inputs reflecting the reconstructed model
Pack/replace submodule_finalize() Quantized runtime module installed at the original dotted name
Whole-run finalization finalize() Model flags, adapter results, resource cleanup

A reconstructed dense weight temporarily installed for calibration replay is not the final inference storage. Likewise, an algorithm loss printed during solving is not a model evaluation score. Packed outputs vary by kernel and format; do not treat every processor as a producer of the GPTQ qweight/qzeros/scales/g_idx layout.

Processor selection

Selection is implemented in BaseQModel._quantize_with_calibration(), not in a plugin registry inside ModuleLooper. The model wrapper constructs a list and passes it into ModuleLooper(model, processors=...).

Config / condition Primary processor Additional processors
Calibration METHOD.GPTQ / default calibration branch GPTQProcessor Optional native capture, source preprocessor, EoRA
METHOD.AWQ AWQProcessor Optional source preprocessor and EoRA
METHOD.QQQ QQQProcessor Optional source preprocessor and EoRA
METHOD.VPTQ VPTQProcessor Optional source preprocessor and EoRA
METHOD.PARO ParoQuantProcessor Optional source preprocessor and EoRA
METHOD.EXL3 EXL3Processor Public EXL3Config has no preprocessors, smoother, or smooth constructor fields; adapter/EoRA, GPTAQ, and FOEM rejected
qcfg.preprocessors nonempty No change to primary method ModulePreProcessor before the primary processor
qcfg.gptaq is not None Numerical task chosen inside GPTQProcessor on the GPTQ branch NativeProcessor inserted at the beginning
qcfg.foem is not None and qcfg.foem.alpha > 0 FOEM task chosen inside GPTQProcessor on the GPTQ branch NativeProcessor inserted at the beginning
qcfg.adapter is a Lora No change to primary method EoraProcessor appended; primary processor receives calculate_w_wq_diff=True
uses_weight_only_lifecycle() == True WeightOnlyProcessor Separate WeightOnlyLooper; optional local planning preprocessor, not the activation chain
BaseQModel._eora_generate(...) DequantizeProcessor followed by EoraProcessor Optional ModulePreProcessor before them

Thus a representative extended GPTQ chain is NativeProcessor -> ModulePreProcessor -> GPTQProcessor -> EoraProcessor, when the corresponding supported settings apply. The native insertion is conditional: FOEM with nonpositive alpha does not trigger that capture insertion. An ordinary GPTQ run has just the primary processor unless additional features are configured.

AWQ and ParoQuant construction also sets AWQ_BATCH_SIZE to the public batch_size and passes the model wrapper and underlying model into their constructors. EoRA receives its explicitly supplied adapter calibration dataset or, if absent, the main calibration data. The weight-only branch ignores calibration and rejects LoRA/EoRA, GPTAQ, and FOEM.

LoopProcessor base contract

Construction and shared state

The base constructor in gptqmodel/looper/loop_processor.py is:

def __init__(
        self,
        tokenizer, qcfg: QuantizeConfig,
        calibration,
        prepare_dataset_func: Optional[Callable] = None,
        calibration_concat_size: Optional[int] = None,
        calibration_sort: Optional[str] = None,
        calibration_concat_separator: Optional[str] = None,
        batch_size: int = 1,
        execution_config: Optional[ExecutionConfig] = None,
):
Attribute / facility Purpose
qcfg Run-level config; per-module clones can apply dynamic overrides without globally rewriting it
qcfg_dynamic Optional latest effective per-module config, used by GPTQ-family naming
tasks Thread-safe mapping of per-module solver/capture tasks
inputs_cache Thread-safe wrapper around InputCache
calibration_dataset Prepared calibration batches until initial cache capture/release
num_batches, total_calibration_tokens Prepared-batch count and token accounting, using masks when available
lock and specialized locks Protect algorithm state, results, progress, input-cache replacement, timing, and diagnostics
_results Processor-level mapping indexed by stable full module names
log, durations, module_names, memory lists Completed-work diagnostics
Thread-local batch/mask state Current capture batch and active padding mask without cross-thread contamination
pb, layer_count Progress handle and layer-count metadata supplied by orchestration

When calibration is supplied, the base requires a nonempty sized input and a prepare_dataset_func. It delegates concatenation, sorting, batching, and separator handling to that function. Fewer than 256 supplied samples and prepared average sequence length below 256 generate warnings. Without supplied calibration it initializes zero batches and an empty calibration list; no-forward processors override verification as appropriate.

Missing data and explicit empty data take different branches: calibration=[] is rejected by the supplied-data length check, but calibration=None creates calibration_dataset=[]. Several concrete verifiers reject only None, not an empty list, so their successful return does not prove that activation capture has usable batches. The base forwards the public calibration_data_min_length through prepare_dataset_func (a functools.partial binding), so the ordinary path honors the argument; multimodal prepare_dataset overrides receive it via **kwargs and may apply different filtering. See Quantization Lifecycle for the complete preparation chain.

result_save(key, value), result_get(key, default=None), result_pop(key, default=None), and results() provide locked result access; results() returns a mapping copy. The base is an override contract with no-op methods, not an abc.ABC implementation that automatically rejects missing overrides.

ExecutionConfig

Actual source defaults:

@dataclass
class ExecutionConfig:
    require_fwd: bool = True
    fwd_replay_after_process: bool = True
    fwd_all_modules_in_single_pass: bool = False
    subset_forward_early_stop: bool = False
    enable_activation_capture: bool = False
    capture_layer_forward_context: bool = False
Field Effect
require_fwd Needs activation forward/capture; false permits a quantization-only subset pass
fwd_replay_after_process Authoritative next-layer outputs come from a replay after processing, not the pre-solve subset forward
fwd_all_modules_in_single_pass Merges a layer's capture subsets; cached batches are still forwarded individually
subset_forward_early_stop Allows capture to stop once the last required subset hook fires
enable_activation_capture Enables capture-only module/hook participation, including model-tree capture markers
capture_layer_forward_context Preserves pristine layer inputs/kwargs/outputs for a processor that needs them despite later replay

Effective constructor policies, assuming the default require_fwd=True where it is an argument:

Processor Forward Replay after process Single capture pass Subset early stop Activation capture Original layer IO
GPTQProcessor Yes Yes No Yes No No
AWQProcessor Yes Yes No Yes Yes No
QQQProcessor Yes Yes No No No No
VPTQProcessor Yes Yes No No No No
ParoQuantProcessor Yes Yes No Yes Yes When normalized opt_scope != "module"
EXL3Processor Yes Yes No Yes No No
EoraProcessor Yes Yes No No No No
NativeProcessor Yes No Yes No No No
DequantizeProcessor No No No No No No
WeightOnlyProcessor No No No No No No
ModulePreProcessor No No No No No No

Hook order and signatures

The main work signature is shared by calibration processors:

def process(
        self,
        module: NamedModule,
        device: torch.device = None,
        subset: Optional[Dict[str, NamedModule]] = None,
        previous_subset: Optional[Dict[str, NamedModule]] = None,
        subset_index: Optional[int] = None,
        subset_total: Optional[int] = None,
):
Hook / method Contract
verify_calibration_dataset(processor_index) Validate own data and return whether initial capture should use it; a later processor returning false can reuse an earlier cache
receive_input_cache(input_cache) Install shared/reused cache through the synchronized wrapper
preprocess(module, **kwargs) Create task, clone effective config, or annotate state before skip evaluation/capture
is_skipped(module) Decide whether this preprocessed candidate should enter processing
pre_process_fwd_hook(name) Return a module forward hook consuming input/output tensors for statistics
process(...) Solve, optimize, or transfer captured state; can run on a device worker thread
receive_layer_inputs(layer_inputs) Install authoritative outputs as inputs for the next stage/layer
receive_layer_forward_context(...) Optional pristine layer IO; ParoQuant uses it for grouped optimization
submodule_finalize(module, model, **kwargs) After consumers finish: install packed module, preserve adapter, or clean transient state
finalize(model, **kwargs) After all layers and worker drains: final model flags/results and shared resource cleanup
name() Label for logs and returned per-processor results; called during base initialization

Submodule and whole-run finalization traverse processors in reverse order. EoRA must save its adapter before the base quantizer consumes/restores wq and packs the original quantized base. Native input buffers must remain alive until their downstream solver finishes. Do not pack a module immediately in a new process() implementation if later processors still require its dense state.

Optional capture extensions include begin_shared_input_capture(model, subset_names, is_lm_head_module=False), end_shared_input_capture(subset_names), and register_moe_root_capture_hook(moe_block, moe_block_name, handles). Default implementations do nothing. GPTQ uses explicit verified shared-input groups; AWQ can capture a model-specific pre-router MoE-root input. Coverage is queried with has_captured_input_ids(name) on participating processors; the name is historical and need not mean literal IDs are stored.

NamedModule state and concurrency

NamedModule provides the live module plus relative name, dotted full_name, layer_index, original module_dtype, shared state, and synchronized delegation. Algorithm task dictionaries often use relative names; packed replacement and persisted results use full names. Keep that distinction when adding a processor.

stream_state_payload_to_cpu(...) stages payload tensors through the transfer machinery. A finalizer using those tensors must call stream_sync() before reading/cloning them. Several packers clone the CPU state after synchronization so the packed consumer does not retain a borrowed staging buffer. Module-tree replacement uses parent-module locks; result/task locks alone do not protect graph mutation.

Logs and temporary files

There is no base tmp_dir API allocating one scratch directory per processor. Distinguish these storage mechanisms:

Storage Ownership / behavior
Processor log file logs/<processor>_log_<random>_time_<timestamp>.log, relative to the process working directory; the constructor creates logs/
Processor results In-memory locked result map; EoRA later attaches it to model.lora_results
Quantization offload directory qcfg.offload_to_disk_path; if omitted with offload enabled, config owns a shared temporary directory with gptqmodel_ prefix
Checkpoint attempt directory checkpoint_session() creates attempt-* under the checkpoint root and redirects offload there
Packed inference export Written by the model save workflow, not by the base processor's temporary-log writer

log_new_row(stat) prints the dynamic terminal table and submits _async_log_writer to the serial CPU worker. That writer appends an indented JSON object plus newline for each record; the whole file is not a single JSON array. Default columns include process/layer/module, feature dimensions, dtype/size, loss, samples, damping, time, forward time, memory, and dynamic configuration, with extra keys added as needed. Field availability and loss semantics vary by processor.

GPTQMODEL_LOG_QUANT_STATS enables an additional plain-text stats line when its value is not "", "0", "false", or "False". Memory collection combines available accelerator and CPU diagnostics; unavailable vendor telemetry is not itself a numerical quantization failure. At the documented revision, accelerator memory reporting uses the active device.

Base finalize() closes diagnostic handles and deletes the input cache/results references. Automatic temporary-log deletion is commented out: logs remain. NativeProcessor.finalize() is a specialized cache cleanup and does not call the base implementation. release_calibration_dataset() separately removes the retained prepared dataset after initial capture.

Processor input/output reference

Class / name() Consumes Intermediate state Final output
GPTQProcessor: gptq, gptaq, or foem Dense source, Hessian/statistics; optional native inputs and activation calibration q_scales, q_zeros, q_g_idx; optional wq, w_wq_diff, activation scale Selected GPTQ-compatible packed linear; activation metadata where applicable
AWQProcessor: awq Captured activations, model scaling groups, dense source and replay kwargs Host activations; q_scales, q_zeros, wq; optional weight error AWQ-format packed linear; scaled nonquantized neighboring weights remain in the model
QQQProcessor: qqq Activation batches/Hessian and dense source Scales/zeros/group indices plus q_scales_extra; optional weight error QQQLinear or QQQTorchLinear with QQQ packed weights/scales
VPTQProcessor: vptq Calibration-built Hessian, dense source vptq_result; optional independent wq and error VPTQLinear codebooks, packed indices, optional residual/outlier data, scale/bias/permutation
ParoQuantProcessor: paroquant Captured activations; grouped scopes also pristine layer IO/module pack_weight, scales/zeros, pairs, theta, channel_scales ParoLinear with packed scalar weights plus learned rotation/scaling metadata
EXL3Processor: exl3 GPTQ capture Hessian, source weight, effective EXL3 config EXL3 tensor dictionary and optional bias ExllamaV3Linear with trellis/transform/codebook-related tensors
EoraProcessor: eora Activation statistics, preceding base wq and w_wq_diff, Lora config Temporary adapter; corrected dense replay weight CPU Lora factors in results / model.lora_results; no new packed base linear
NativeProcessor: native Unquantized module inputs native_inp No packed module; inputs consumed by later solver and then removed
DequantizeProcessor: de-quantize Mapping of existing TorchLinear modules plus original dense source wq, float32 w_wq_diff No packed replacement; provides EoRA inputs
WeightOnlyProcessor: weight_only_rtn, weight_only_gguf, weight_only_fp8, weight_only_bitsandbytes Dense source only RTN scales/zeros/indices or direct-packing source Selected method-specific packed module
ModulePreProcessor: module-preprocessor Config preprocessors and source dimensions Decoder/padding/smoothing plan annotations No packed replacement

GPTQProcessor

Source: gptqmodel/looper/gptq_processor.py. See GPTQ.

Capture and numerical task selection

preprocess() calls clone_gptq_config_for_module(...); a full dynamic exclusion returns no config/task. The clone applies the supported per-module overrides, including quantization bits/symmetry/group size, MSE configuration, activation ordering, damping, static groups, and fallback. It constructs GPTAQ if the effective config has gptaq, otherwise FOEM if it has foem, otherwise GPTQ. The ordinary GPTQ task receives normalized fallback and expected total calibration tokens. Its quantizer is configured per-channel.

The forward hook feeds activations into the task. GPTQ preserves the batch keep-mask semantics so padded [B, S, H] batches are masked in the accumulator rather than incorrectly counted as real tokens. Explicit model-tree shared-input groups can share Hessian capture; the plan is enabled only for verified definitions and eligible groups, not just because two linears have equal dimensions. Followers adopt captured state after leader collection.

process() verifies placement consistency between module weight, assigned target device, task weight/Hessian, and current accelerator-thread context. It calls g.quantize(), which returns reconstructed weights, scales, zeros, group indices, duration, loss, effective damping, and sample count. Packing metadata is streamed to CPU; task buffers are freed and module.weight.data becomes the reconstructed weight for sequential replay.

With calculate_w_wq_diff=True, it computes the difference in float32 and preserves an independent CPU copy of the reconstructed weight under wq. That copy must not alias a parameter that subsequent lazy rematerialization might overwrite before EoRA runs.

Activation-quantized replay

For w4afp8, the processor enables FP8 input QDQ replay after solving the target. For w4a_nvfp4, it uses the observed/calibrated activation global scale and selected recipe, writes activation_global_scale, and enables NVFP4 replay. Existing stream-replay pre-hooks prevent a second QDQ of the same input. These replay helpers currently require torch.nn.Linear targets. Missing required headroom calibration or nonfinite observed activation maxima raise errors; they are not silently replaced by a valid-looking scale.

Packing and naming

submodule_finalize() synchronizes staged transfers, restores the saved base reconstruction if EoRA temporarily changed the weight, consumes CPU q_scales/q_zeros/q_g_idx, creates the selected quantized module, and calls pack_module(...). Applicable activation-scale state is installed on the packed module. Original dense-weight registration and consumed temporary state are removed.

name() returns gptaq for an effective GPTAQ config, foem for FOEM, otherwise gptq. This is a reporting label; finalize() marks the model quantized and sets model.quantize_config.method = METHOD.GPTQ before shared cleanup. verify_calibration_dataset() rejects calibration_dataset is None; it does not test for zero prepared batches. has_captured_input_ids() reflects the task's forward counter, not just successful task construction.

AWQProcessor

Source: gptqmodel/looper/awq_processor.py. See AWQ.

AWQ uses captured activation features and model-defined scaling groups rather than converting each hooked module immediately. Construction requires both gptq_model and the underlying model; it resolves the quantized kernel from the wrapper or format mapping. Constructor defaults include apply_clip=True, duo_scaling=True, and max_chunk_memory = 1024 * 1024 * 1024 for chunked internal search computations. These are processor implementation attributes, not additional public quantize() parameters.

preprocess() registers each eligible module in _AWQLayerState, initializes its feature bucket, and records pending modules. Capture detaches input features to CPU, excludes matched padding rows, and retains batch metadata. Model policy can request MoE-root feature capture and bounded token-row aggregation; aggregation keeps deterministic rows from contributing batches instead of assuming a single latest batch represents the layer. Replay kwargs are aligned to those features.

process() merges the subset mapping, records completed subset indices and pending modules, and computes/caches previous-subset weight-scale context. It triggers _quantize_layer() only once the registered layer work is ready. A process() return for an early module therefore need not mean its module has already been solved independently.

The layer quantizer obtains model scaling groups, searches scales and optional clipping, applies scaling to participating modules, and calls apply_quant(). A source decoder can supply quant_source_module rather than the live wrapper. The pseudo-quantized dense weight becomes the replay weight; q_scales and q_zeros are staged to CPU, wq is recorded, and optional EoRA error is recorded. TP padding is trimmed back to the original column/group extent.

submodule_finalize() delegates to pack_module(): synchronize, restore base wq when needed, consume scales/zeros, create the AWQ kernel for the resolved format, and call the common packer with q_g_idx=None. Scaling can also mutate nonquantized neighbors such as norms; a packed weight alone is not the complete transformed model state.

AWQ logs feature-specific sample/token/batch/aggregation metadata, including separate activation and scaling-feature counts. If a module has no matching scaling-loss result, the displayed loss is "not applicable". Do not compare that column as though it had GPTQ's loss definition. Some AWQ/QQQ calibration errors retain the legacy GPTQProcessor wording; use the selected processor and call stack to identify the actual algorithm.

QQQProcessor

Source: gptqmodel/looper/qqq_processor.py. See QQQ.

preprocess() deep-copies the config, applies supported dynamic choices, constructs QQQ, supplies normalized fallback and expected samples, and configures its quantizer. Its hook calls add_batch(inp[0].data, out.data). Unlike GPTQ/AWQ/EXL3, it leaves subset early-stop at the base default False.

There is an important override boundary: the QQQ task receives the per-module clone, but tmp.quantizer.configure(...) reads bits, sym, mse.norm, and group_size from the processor-wide self.qcfg. Do not infer that every cloned override also changes this separate quantizer configuration at this revision.

process() calls the QQQ solver and frees its task in finally. Along with reconstructed weights it receives ordinary scale/zero/group-index metadata and q_scales_extra. Scales, zeros, and group indices are staged on CPU; optional EoRA state includes an independent CPU reconstructed-weight snapshot. The live module receives reconstructed weights for replay.

submodule_finalize() selects the concrete QQQ kernel: Darwin, CPU, and NPU use QQQTorchLinear / BACKEND.QQQ_TORCH; other supported accelerator paths use QQQLinear / BACKEND.QQQ. It creates that class and passes q_scales_extra into the common packer. QQQ's final representation includes packed B, s_channel, and group scales s_group when applicable, plus bias; it is not a generic GPTQ buffer schema. Per-channel packing can omit the optional group-scale buffer.

Whole-run finalization sets METHOD.QQQ and model.quantized=True. Dataset replacement through set_calibration_dataset() is unsupported; initial calibration is required and task fwd_counter supplies the coverage probe.

VPTQProcessor

Source: gptqmodel/looper/vptq_processor.py. See VPTQ.

This path constructs Hessians from calibration activations; it does not require a precomputed hessian_path. preprocess() skips full dynamic exclusions, clones the config, constructs VPTQ, and supplies fallback and expected sample count. The hook feeds input/output batches to the task's accumulator.

Unlike the GPTQ clone helper, this processor's preprocess() does not resolve individual dynamic field overrides into the clone: its explicit dynamic handling is the whole-module exclusion check. Do not assume the GPTQ override contract applies unchanged to VPTQ.

process() receives a VPTQQuantResult, duration, and loss, frees task buffers in finally, and restores the reconstructed weight into the live module's native layout for replay. It moves result tensors to CPU, including centroids, indices, res_centroids, res_indices, weight_scale, weight_bias, perm, and quantized_weight, and stores the object under vptq_result. Optional EoRA state uses float32 error and an independent CPU wq snapshot.

The finalizer creates VPTQLinear using BACKEND.VPTQ_TORCH, method-specific initializer arguments, and original module dtype; then calls:

qmodule.pack(linear=layers[module.full_name], vptq_result=result)

The runtime loader fills codebook embeddings, packed code indices, optional residual/outlier data, scale/bias, and permutation data from the result. The source linear contributes its bias. VPTQLinear.pack() rejects a missing result; codebook and index validation occurs while loading it. A reconstructed dense tensor is a replay artifact, not a substitute for those codebooks/indices.

Finalization sets METHOD.VPTQ, FORMAT.VPTQ, and the model's quantized flag. Dataset replacement is unsupported. VPTQ is not in the quantization checkpoint adapter's supported-chain allowlist at this revision.

ParoQuantProcessor

Source: gptqmodel/looper/paroquant_processor.py. See ParoQuant.

ParoQuant is a standalone capture/optimize/export/pack workflow, not an AWQ subclass. It accepts the wrapper and underlying model, maintains _ParoQuantLayerState, and defers solving until all registered layer subsets finish capture. Its execution config enables activation capture/early-stop and requests pristine layer IO when the normalized optimization scope is not "module".

Inputs depend on scope:

Scope Required optimization context
module Each module's captured activation features
compute_block Group/module context plus preserved layer inputs, kwargs, and target outputs
layer Layer-level optimization context, preserved IO, and pristine source/replay state

The processor receives layer context and pristine source modules through explicit hooks. Grouped replay loaders can stream CPU-owned batches to device shards and rebuild autograd-safe tensors from inference-mode capture. It does not rely on ordinary inference-mode tensors becoming trainable merely because an optimizer is invoked.

process() records pending/completed subset state and triggers _quantize_layer() once ready. The implementation resolves optimization groups, optimizes modules/groups, installs reconstructed replay weights, logs results, and releases layer feature/replay caches after completion. Missing features with fallback disabled raise an error; grouped optimization without preserved layer inputs/outputs raises an error.

The pack payload contains pack_weight, q_scales, q_zeros, pairs, theta, and channel_scales; optional EoRA state is separate. pack_weight is the weight in the packing coordinate system, not necessarily the dense replay reconstruction. The finalizer synchronizes and clones that payload, restores pack_weight on CPU, creates the resolved Paro kernel, calls the common packer with no group-index tensor, verifies the replacement is a ParoLinear, copies rotation/scaling buffers, and invokes post_init().

The looper prewarms the rotation runtime and restricts ParoQuant quantization to a single device at this revision. It also drains ParoQuant finalizers synchronously to avoid retaining optimization state across layers. finalize() sets METHOD.PARO; the label is paroquant. Continuation extends the base state with clean_group_layer_inputs, allowing the supported checkpoint chain to preserve its persistent cross-layer context without pickling live optimizers/locks.

EXL3Processor

Source: gptqmodel/looper/exllamav3_processor.py. See EXL3.

preprocess() clones effective EXL3 settings and constructs a GPTQ capture task, not a GPTQ weight solver. The capture config uses group_size=-1, desc_act=False, sym=True, and act_group_aware=False, with the effective device/pack dtype. The hook feeds activation batches and the current batch index into that accumulator.

process() finalizes the captured Hessian on the target device, rejects a missing Hessian or nonpositive sample count, converts the source weight to the quantizer's layout, and invokes EXL3 quantization. The CUDA/HIP branch builds H_data with single_use=True and calls quantize_exl3(..., return_weight_q=True). This allows replay reconstruction without retaining an unnecessary reusable Hessian factor in a single-use capture dictionary.

There is also an internal MLX helper branch for CPU/MPS targets when exl3_mlx_quantization_available() succeeds. This does not broaden the public API: the current BaseQModel.quantize() EXL3 entry rejects non-CUDA/HIP quantization before reaching this processor. Do not advertise direct CPU/MPS EXL3 quantize() from the existence of an internal branch. See MLX Apple Silicon for the separate platform surface.

The quantizer returns a reconstructed weight, proxy error, and packed tensor dictionary. Tensors and optional bias are streamed to CPU; reconstructed weights are assigned to the live source-layout module for replay. submodule_finalize() synchronizes, clones recognized payload fields (trellis, suh, svh, su, sv, bias, mcg, mul1) when present, and calls create_exllamav3_module(...) under the parent lock. Dense weight and optional bias registrations on the old module are removed.

finalize() sets the model quantized, METHOD.EXL3, FORMAT.EXL3, and qlinear_kernel = ExllamaV3Linear. The public chain rejects LoRA/EoRA, GPTAQ, and FOEM. The layer stage's generic finalized-module disk-offload guard includes GPTQ/QQQ/AWQ/ParoQuant, not EXL3; checkpoint capture can serialize supported EXL3 packed artifacts independently.

docs/exl3-mi350x-quant-lifecycle.md records a historical MI350X/gfx950 quantizer study and explicitly is not the current implementation contract. Its kernel timings, worker counts, and occupancy observations are not required settings for this class.

EoraProcessor

Source: gptqmodel/looper/eora_processor.py. See EoRA.

EoRA consumes base quantization error and activation statistics to produce low-rank correction factors. It does not quantize the base weights independently. A preceding quantizer or DequantizeProcessor must provide both w_wq_diff and the base reconstructed wq.

preprocess() clones the Lora config, applies per-module adapter.rank dynamic overrides, records the target device, and initializes per-device segment accumulators. Full dynamic exclusions set module.adapter_cfg=None. Hooks compute activation contributions with batch indices; segment accumulation merges contributions into the eigen-scaling matrix without requiring every contribution to make an immediate CPU round trip.

process() asserts the adapter config is Lora, finalizes the eigen-scaling matrix, moves the weight difference to float32 on the target device, and computes A and B at the requested rank. It installs a temporary corrected dense replay weight based on wq + (B @ A), while preserving the uncorrected base wq on CPU for the base packer. TP padding is trimmed appropriately. The corrected reconstruction is checked as FP16/BF16; the factors are moved to CPU in a new Lora object under module state adapter.

The submodule finalizer moves that object into the results map under the full module name. Whole-run finalization clears accumulators, attaches results to model.lora_results for saving, and runs shared cleanup. The base quantizer's later reverse finalizer packs only the base quantized weights, not the temporary dense weight with the correction fused into it.

If EoRA is the first processor, missing calibration is rejected; as a later processor it can report that a previous cache should be reused. set_calibration_dataset() can replace its dataset and update batch count. Its continuation override explicitly serializes each completed Lora as rank/path/factor fields and reconstructs Lora objects on restore rather than pickling them.

NativeProcessor

Source: gptqmodel/looper/native_processor.py.

Native capture preserves pristine per-module inputs for GPTAQ or positive-alpha FOEM. It combines a layer's subsets into one capture plan (fwd_all_modules_in_single_pass=True) and forwards each cached calibration batch through it, with no replay-after-process. The flag does not collapse the dataset or all model layers into a single forward. Each preprocess() allocates a list, and the hook detaches inp[0] and stages it according to the GPTAQ/FOEM device:

Device setting Capture destination
"auto" DEVICE_1
"cpu" CPU
Other string torch.device(value)
torch.device That device
Unrecognized value DEVICE_1 fallback

Do not interpret DEVICE_1 as a promise that a second physical GPU exists; it is the utility's resolved device constant. process() transfers the captured list to module.state["native_inp"] and removes its processor-cache entry. submodule_finalize() removes that shared-state key after consumers finish; finalize() deletes the native cache mapping. There is no packed-module output.

is_skipped() currently returns false for eligible candidates. Calibration is required, and set_calibration_dataset() raises NotImplementedError because construction fixes capture setup.

DequantizeProcessor

Source: gptqmodel/looper/dequantize_processor.py.

The constructor has a deliberately different input contract:

def __init__(self, quantized_modules: Dict[str, TorchLinear]):

It initializes the base with no tokenizer/config/calibration, batch size 1, and no-forward/no-replay execution policy. verify_calibration_dataset() returns false; its dataset setter discards supplied data and resets batch count to zero.

process() pops the existing packed module by module.full_name, calls its optimize(), computes m.dequantize_weight().T on the dense source's device, and adds any required TP pad columns. It computes the difference between the dense source and the reconstruction in float32 and stores w_wq_diff and wq. The quantized-module input mapping is consumed as work progresses.

This is a preparatory stage for _eora_generate()'s DequantizeProcessor -> EoraProcessor chain, not a public model-wide dequantization/export command. Missing mapping entries naturally fail instead of fabricating a reconstruction. Its finalizer removes temporary w and wq; downstream EoRA consumes the difference. Its label is exactly de-quantize.

WeightOnlyProcessor

Source: gptqmodel/looper/weight_only_processor.py. This class is a LoopProcessor subclass but uses the dedicated WeightOnlyLooper protocol: the orchestrator calls quantize_module() rather than the base no-op process().

def __init__(
    self,
    tokenizer,
    qcfg: RTNConfig | GGUFConfig | FP8Config | BitsAndBytesConfig,
):

def quantize_module(
    self,
    module: NamedModule,
    *,
    device: Optional[torch.device] = None,
) -> Optional[RTNConfig | GGUFConfig | FP8Config | BitsAndBytesConfig]:

clone_weight_only_config_for_module(...) applies per-module config/skip policy. The orchestrator excludes skipped modules before device work; quantize_module() can also return None for exclusions. An explicit scheduled device is installed on the clone, which is handed into finalization.

Method Conversion stage Pack stage Log label
RTN RTN.quantize() returns reconstructed weight and scales/zeros/group indices; metadata streams to CPU Create selected module, synchronize and consume metadata, call common pack_module(...) weight_only_rtn
GGUF No activation solve; preserve original dense source qmodule.pack_original(linear=original_layer, scales=None, zeros=None, g_idx=None, smooth=active_qcfg.smooth) weight_only_gguf
FP8 Same direct-pack lifecycle Method-specific pack_original(...) weight_only_fp8
BitsAndBytes Same direct-pack lifecycle Method-specific pack_original(...) weight_only_bitsandbytes

The finalizer creates the module with method-specific format/init kwargs and register_buffers=False, then invokes the appropriate packing path. Direct-pack methods compute a mean absolute reconstruction error after packing and backfill the pending log loss. Their nsamples is zero because no calibration was collected. FP8/GGUF dequantized matrices are transposed for comparison; BitsAndBytes exposes native [out, in] weights and is not transposed in that check.

Consumed scale metadata, TP padding/source references, and the old dense weight registration are released. finalize() marks the model quantized and runs base cleanup; it does not rewrite all methods to GPTQ. Input/output embedding fast conversion currently supports RTN only, and embedding-only operation has separate save/offload restrictions handled by WeightOnlyLooper.

See BitsAndBytes, GGUF, and FP8 and NVFP4 for format/runtime requirements. A no-calibration lifecycle does not imply every kernel can run or pack on every device.

ModulePreProcessor

Source: gptqmodel/looper/module_preprocessor.py.

This additional subclass is part of the execution chain when preprocessors are configured. It strips the primary quantizer's calculate_w_wq_diff argument and initializes with no calibration and no-forward/no-replay policy. Its label is module-preprocessor; calibration verification returns false and process() does nothing.

preprocess() walks configured preprocessors in order:

Config State emitted
AutoModuleDecoderConfig Decoder code/source/target dtype, passthrough forward/save policies, quant_source_decode_policy="defer_until_quantization", optional native floatx runtime plan
TensorParallelPadderConfig Column padding plan computed from LCM of (2, 4, 8) and positive group size
SmootherConfig to_dict() entry in preprocessor_pipeline

It stores the ordered preprocessor_pipeline, optional auto_module_decoder, and nonzero tp_pad_info; absent plans remove stale decoder/padding-related keys. For a transformers.Conv1D, the row/column interpretation follows its transposed weight layout. The class annotates a plan rather than independently performing every decode/pad/smooth transformation in its no-op process().

On the calibration path it is a processor before the primary quantizer. On the weight-only path the orchestrator constructs a local planner and calls it before handing decoded source modules into weight-only conversion. These are distinct integration points.

Continuation, finalizer errors, and extension boundaries

continuation_state_dict() captures completed boundary state: the input cache, log, prepared counts, results, and available statistics. It deliberately excludes live tasks, locks, partial Hessians, and unfinished optimization accumulators. load_continuation_state_dict() restores completed state; normal replay creates future tasks. AWQ additionally saves avg_losses, nsamples, nsamples_total, and quant_batch_size; EoRA and ParoQuant extend the schema for their persistent data as well.

Having this method does not automatically make a new processor resumable. QuantizationCheckpointAdapter.bind() allows exact chains for GPTQ/AWQ/QQQ/ParoQuant (with optional EoRA), weight-only, and EXL3 alone. VPTQ and chains containing ModulePreProcessor, Native, or Dequantize are not on the allowlist. Model/source/config/device restrictions apply in addition; see Quantization Lifecycle.

Extensions observe orchestration, not arbitrary partially completed processor calls. LoopExtensions.start() permits one compatible resume cursor; LoopBoundary.quiesce() waits for all preceding finalizers before returning results or raising. A boundary is borrowed only during its callback and belongs to the orchestration thread. Serializing state from a worker or retaining a boundary for later use is invalid.

wait_for_submodule_finalizers=False permits overlap only where the looper deems it safe. W4A modes, multiple quantization accelerators, and ParoQuant force synchronous per-layer draining. Overall completion still waits for device tasks, stream transfers, and watcher threads. A failure after process() but before packing/offload is a failed quantization, not a completed module just because its loss row was printed.

Developer guidance and pitfalls

Issue Correct integration rule
Wrong module identity Use relative names for the current task namespace and stable full names for replacement, dynamic matching, persisted results, and cross-layer identity
Unobserved MoE expert Implement truthful coverage detection and explicit fallback behavior; a task's existence does not prove it captured activations
Rematerialization changes wq Preserve independent reconstruction storage when another processor needs it later
EoRA accidentally packed into base weights Preserve uncorrected wq; let reverse finalization save the adapter and pack the base separately
Corrupt staged scale metadata Synchronize stream transfers before reading; clone borrowed host payloads where the consumer needs independent ownership
Missing subset context Accept the full process(...) contract; layer-deferred solvers must track pending modules/subsets and run once
Premature packing Keep dense source/reconstruction until downstream capture/process/replay consumers finish
Thread-unsafe replacement Follow parent-module lock conventions in addition to task/result locking
Assuming no-forward means no orchestration Planning/dequantization stages still participate in layer identity, state, skip logic, finalization, and cache hand-off
Assuming all subclasses override process() Weight-only intentionally exposes quantize_module() to its dedicated looper
Treating logs as checkpoints Log objects are diagnostics; safe resume requires quiescent boundary state and published checkpoint artifacts
Treating logs as identical quality metrics GPTQ loss, AWQ search loss, EXL3 proxy error, Paro validation loss, and direct-pack MAE have different meanings
Unsupported hardware inferred from class name Validate method/format/backend/device before construction; portable packing and accelerated inference are separate capabilities
Blanket resume claims Add an explicit supported state adapter and packed schema; the base continuation method is necessary but insufficient

When adding a processor, choose its ExecutionConfig first, define captured data and coverage semantics, isolate per-module overrides, make replay reconstruction and packing payload ownership explicit, and release temporary tensors as soon as their last consumer completes. If checkpointing is intended, specify persistent boundary state without arbitrary objects and extend the explicit adapter/packed-module allowlists. Changes to quantization math, packers, or kernels require the project's numerical/storage validation; processor logs alone do not validate correctness.

Source files

Path Responsibility
gptqmodel/models/base.py Constructs processor chains, selects weight-only path, composes _eora_generate()
gptqmodel/looper/loop_processor.py ExecutionConfig, shared state/results, logging, continuation, lifecycle hooks
gptqmodel/looper/named_module.py, input_cache.py Stable per-module state and batch-aligned inputs
gptqmodel/looper/module_looper.py, stage_layer.py, stage_subset.py, forward_executor.py Processor scheduling, capture, replay, finalization, coverage
gptqmodel/looper/gptq_processor.py GPTQ/GPTAQ/FOEM task dispatch, Hessian capture, replay and packing payloads
gptqmodel/looper/awq_processor.py AWQ activation capture, layer-scale search, clipping, packing
gptqmodel/looper/qqq_processor.py QQQ capture, solver and scale-aware packing
gptqmodel/looper/vptq_processor.py VPTQ Hessian capture, result transfer, codebook packing
gptqmodel/looper/paroquant_processor.py Deferred layer/group optimization, replay context, rotation payload
gptqmodel/looper/eora_processor.py Activation contributions and low-rank adapters
gptqmodel/looper/exllamav3_processor.py EXL3 capture, solve, trellis payload and replacement
gptqmodel/looper/native_processor.py, dequantize_processor.py Pristine-input and existing-base reconstruction stages
gptqmodel/looper/weight_only_processor.py, weight_only_looper.py RTN solve/direct pack and dedicated calibration-free orchestration
gptqmodel/looper/module_preprocessor.py Ordered decoder/TP/smoother plan annotations
gptqmodel/looper/gptq_checkpoint.py, checkpoint_modules.py, continuation.py, extension.py Supported chains, packed continuation schemas, safe boundary persistence
gptqmodel/quantization/gptq.py, gptaq.py, foem.py, qqq.py, vptq.py, rtn.py Numerical task implementations called by processors
gptqmodel/quantization/config.py Method/lifecycle/config policies
gptqmodel/utils/model.py create_quant_module and pack_module
gptqmodel/nn_modules/qlinear/qqq.py, vptq_torch.py, paroquant.py Method-specific packed-module consumers
gptqmodel/nn_modules/exllamav3.py EXL3 packed module creation/storage
docs/looper-checkpoint-redesign.md, docs/multi_gpu_quantization.md, docs/exl3-mi350x-quant-lifecycle.md Checkpoint design, device-pool usage, historical EXL3 lifecycle measurements

See also

Documentation home · Quickstart · Troubleshooting · Repository · PyPI

Source baseline: GPT-QModel 7.7.0 / c197691. Maintained by ModelCloud.ai.

Home

Getting started

Quantization methods

Complementary configs

Using a model

Platforms

Internals

Clone this wiki locally