Skip to content

QuantizeConfig Reference

Jump to bottom
Qubitium edited this page Oct 29, 2026 · 5 revisions

QuantizeConfig Reference

Documentation home · Quickstart · Lifecycle

On this page

Overview

QuantizeConfig is the compatibility factory for method-specific dataclasses, not a single configuration object holding every algorithm's options. New code can construct GPTQConfig, AWQConfig, RTNConfig, GGUFConfig, FP8Config, BitsAndBytesConfig, QQQConfig, VPTQConfig, EXL3Config, or ParoConfig directly.

This reference describes the source API at revision 81ea93ffc475906dc54d8462a9543001206989b9. Defaults below distinguish the declared dataclass default from the value produced by normalization. Config acceptance does not guarantee kernel feasibility: device, bit width, group size, packing dtype, shape alignment, and optional dependencies are checked again when a layer/kernel is selected.

Most public classes are exported from gptqmodel.quantization. Import MoE routing classes and enums such as VramStrategy/GcMode directly from gptqmodel.quantization.config where needed. Runtime Lora is defined in gptqmodel.adapter.adapter.

Workflow at a glance

Configuration normalization and dispatch. Different entry points converge on method-specific configuration objects.

Reading the diagram: Entry lanes are alternatives. JSON and constructor alias precedence differ. Config acceptance does not guarantee kernel compatibility.

Text version of the workflow
  • QuantizeConfig factory: Constructor keywords → Normalize aliases → Dispatch and filter → Run class validation.
  • Direct concrete class: Choose a config class → Normalize aliases → Run class validation.
  • Checkpoint config import: Read saved fields → Import aliases and meta → Dispatch and filter → Run class validation.

Open full-size SVG

Concepts: algorithm, storage, execution

Concept API Responsibility
Algorithm method: METHOD Quantization family, not the inference backend. RTN is a lifecycle/config choice; there is no METHOD.RTN.
Serialized layout format / compatibility property checkpoint_format Defines the tensor contract. GGUF/FP8/BitsAndBytes use a subtype rather than only a family token.
Execution BACKEND passed to model/kernel APIs Chosen independently, subject to support for the algorithm/layout. Marlin runtime does not require a Marlin-format checkpoint.
Calibration-only controls Damping, Hessian, MSE budgets, routing, telemetry, optimizer fields Generally recorded under meta, or intentionally not serialized.
Per-module exception dynamic Ordered first-match overrides or negative exclusions.

Factory dispatch and compatibility

QuantizeConfigMeta.__call__(cls, *args, **kwargs) first normalizes deprecated constructor keywords quant_method -> method and checkpoint_format -> format; canonical names take precedence. is_marlin_format fails immediately. Direct QuantizeConfig(...) then resolves a concrete class, prepares method-specific kwargs, and filters out fields that are not initializer fields of that class. Use explicit concrete classes when you want Python to reject misspelled or inappropriate keywords rather than silently filtering them. Class selection uses keyword payloads; do not rely on positional arguments to select an algorithm.

Resolution order is significant: GGUF hints (format="gguf", registered GGUF bits, a public GGUF subtype, or the legacy weight-only method) take priority; legacy weight-only FP8/BitsAndBytes/RTN follow; then FP8, BitsAndBytes, EXL3, ParoQuant, VPTQ, QQQ, AWQ, and finally GPTQ. AWQ layout hints GEMM/GEMV/GEMV_FAST/LLM_AWQ select AWQ. MARLIN retains the explicit method family. Concrete-class preparation can repair incompatible method/format hints; an unknown value is not a promise of automatic recovery. In particular, method="gguf" alone is not a GGUF factory-dispatch condition in this revision; use GGUFConfig(...) or an explicit GGUF format/bits hint.

isinstance(any_base_config, QuantizeConfig) and issubclass(ConcreteConfig, QuantizeConfig) intentionally recognize BaseQuantizeConfig descendants through the metaclass. QuantizeConfig adds no dataclass fields to the base.

BaseQuantizeConfig: every field

All concrete classes inherit these fields unless overridden below. Default factories create fresh sub-configs. Internal computed fields are included for completeness but are not constructor arguments.

Field Declared type Declared default Meaning and normalization
bits Union[int, str, GGUFBits] 4 Weight precision; normalization and method-specific restrictions are described below. Kernel validation can be narrower than configuration parsing.
dynamic Optional[Dict[str, Dict[str, Union[int, str, bool, GGUFBits]]]] None Ordered PCRE per-module rules. Negative rules are moved before positive rules; first match wins. See Dynamic per-layer overrides.
group_size int 128 Number of input channels sharing quantization parameters; -1 means a whole row. Base validation accepts -1 or any positive value, despite the narrower values printed in its error message. Kernel constraints still apply.
desc_act Optional[bool] None GPTQ activation-ordering toggle. None resolves through default_desc_act(); BaseQuantizeConfig and ParoConfig use True, while GPTQConfig uses False. See the concrete defaults table. Other non-boolean values are bool-coerced.
sym bool True True selects symmetric weight quantization; False requests asymmetric quantization where the method/kernel supports it.
true_sequential bool True Calibrate downstream modules with the preceding quantized computation rather than quantizing every module against the same unmodified layer input.
lm_head bool False Include the output language-model head in ordinary model quantization. Embedding-only APIs have their own targeting rules.
method METHOD METHOD.GPTQ Algorithm family, normalized to METHOD. Concrete classes validate their allowed family; the factory may repair incompatible family hints.
format FORMAT FORMAT.GPTQ Authoritative serialized layout. Usually a FORMAT value; GGUF, FP8, and BitsAndBytes instead expose a subtype string and resolve their family separately.
activation Optional[Union[str, Dict[str, Any]]] None Optional W4A activation-stream policy; normalized to a canonical dictionary. Requires the strict INT4 GPTQ contract documented below. Does not change saved weight packing.
meta Optional[Dict] None Dictionary with string keys for provenance and non-runtime settings. None becomes {}. Values must ultimately be JSON serializable.
device Optional[Union[str, torch.device]] None Requested load/quantization device; normalized by the loading path, not by this base config hook.
telemetry Union[TelemetryConfig, Dict[str, Any]] TelemetryConfig() TelemetryConfig instance or dictionary; controls diagnostics, not quantization/resume algorithm identity.
mse Union[MSEConfig, Dict[str, Any], int, float] MSEConfig() MSEConfig, dictionary, or legacy numeric exponent. Normalized to MSEConfig except in the separate EXL3 post-init path; see serialization caveats.
pack_dtype Optional[Union[str, torch.dtype]] torch.int32 Integer storage word dtype for packed weights/zeros: int8, int16, int32, or int64 as torch.dtype or lowercase-normalized name. None becomes int32. Not every kernel supports every word size.
pack_impl str 'auto' Packing implementation hint: auto, original, cpu/block, or gpu in the packing utilities. The base hook does not validate this string. Packing may fall back when the requested implementation is unavailable.
adapter Optional[Union[Dict[str, Any], Lora]] None Lora instance or a serialized adapter descriptor normalized by normalize_adapter(); None disables EoRA/LoRA recovery.
offload_to_disk bool True Offload completed quantization modules to disk to reduce CPU memory. A truthy rotation disables it automatically.
offload_to_disk_path str None Offload directory. When enabled with an empty path, a managed temporary directory with prefix gptqmodel_ is created.
_offload_temp_dir Optional[_SharedTemporaryDirectory] None Internal shared temporary-directory owner, excluded from initialization, repr, and equality. Copies share cleanup ownership; not a public tuning parameter.
rotation Optional[str] None Optional hadamard or random orthogonal weight rotation. Requires materialized tensors and disables disk offload. The quantization entry point restricts this to Llama/Qwen2 models and GPTQ_TORCH, GPTQ_TRITON, GPTQ_W4AFP8 or GPTQ_W4A_NVFP4. Not supported for QQQ; distinct from ParoQuant's internal rotations.
fallback Optional[Fallback] Fallback() Low-calibration-coverage fallback, normalized from Fallback/dict/scalar/None. None disables it; the default is a fresh Fallback().
compute_device_filter Optional[callable] None Optional callback fn(devices: List) -> List filtering devices for compute-heavy forwarding/quantization stages. It is not serialized.
calibration_data_device Optional[Union[str, torch.device]] None Storage placement for captured calibration data. A specific string/torch.device is canonicalized; balanced (case-insensitive) distributes captures across accelerators.
auto_forward_data_parallel bool True Allow multi-device forwarding clones; can increase startup cost and memory, particularly for MoE.
dense_vram_strategy VramStrategy VramStrategy.EXCLUSIVE Dense-pool strategy: exclusive or balanced. Dense calculation groups, including Q/K/V, remain co-located under balanced placement.
dense_vram_strategy_devices Optional[List[Union[str, torch.device]]] None Optional nonempty explicit dense-device list. Device-like values are canonicalized and duplicate device strings removed, preserving first occurrence. Ordinals are relative to visible devices.
moe_vram_strategy VramStrategy VramStrategy.EXCLUSIVE Expert-pool strategy: exclusive or balanced. Gate/up/down belonging to one expert stay together.
moe_vram_strategy_devices Optional[List[Union[str, torch.device]]] None Optional nonempty explicit expert-device list; same canonicalization and deduplication as the dense pool.
gc_mode GcMode GcMode.INTERVAL interval performs regular staged garbage collection; on_stage_end collects at stage boundaries. Strings are lowercased into GcMode.
wait_for_submodule_finalizers bool False Wait for layer packing/offload finalization before advancing to the next layer; trades overlap for potentially lower memory pressure.
moe Optional[MoEConfig] None MoEConfig, dictionary with a tagged routing payload, or None. See MoE routing.

Base normalization sequence and limits

The base __post_init__ normalizes method, format family, pack dtype, and bits; verifies the class's allowed method/export formats; normalizes fallback; checks bit choices; auto-routes GPTQ with any 5/6/7-bit global or positive dynamic layer to GPTQ_P; moves negative dynamic rules first; normalizes layer bits/group sizes; resolves desc_act; validates activation-stream constraints and meta; normalizes adapters, telemetry and MSE; disables offload for rotation; creates the offload directory; then normalizes placement strategies, GC, MoE and calibration storage devices.

The declared base bit choices are [2, 3, 4, 5, 6, 7, 8]. The generic bit parser recognizes 1..8, but base/concrete metadata may reject 1. Group-size validation is not a whitelist of the sizes shown in its error text: it rejects only values other than -1 that are <=0. A kernel may impose an actual whitelist or divisibility requirement. General sym, pack_impl, and rotation constraints are not all enforced in the base hook.

Concrete defaults and lifecycle

Class Method Initial/default format Bits Group size Resolved desc_act Calibration dataset required by config?
BaseQuantizeConfig GPTQ GPTQ 4 128 True Yes
GPTQConfig / bare factory GPTQ GPTQ 4 128 False Yes
RTNConfig GPTQ GPTQ 4 128 False No
AWQConfig AWQ GEMM 4 128 False Yes
QQQConfig QQQ QQQ 4 128 False Yes
VPTQConfig VPTQ VPTQ derived, default 2 -1 False Yes
ParoConfig PARO PAROQUANT 4 128 True (base default) Yes
EXL3Config EXL3 EXL3 3.0 forced -1 forced False Yes
FP8Config FP8 float8_e4m3fn 8 forced -1 forced False No
BitsAndBytesConfig BITSANDBYTES fp4 for 4-bit; int8 for 8-bit 4 forced -1 forced False No
GGUFConfig GGUF q_0 for default 4-bit 4 fixed -1 fixed False No

These are config lifecycle flags (requires_calibration_dataset() is the negation of uses_weight_only_lifecycle()), not a claim that every model/backend can run without other initialization data.

TelemetryConfig

Field Type Default Normalization
device bool False Detailed device diagnostics. Must be an actual bool, not an integer or truthy string.

Base configuration accepts a typed TelemetryConfig or a dictionary, and rejects other types. Diagnostics do not define the stored weight layout or calibration-resume identity.

Activation stream sub-configs

These policies change activation operands, not INT4 weight packing. Policy classes are defined in gptqmodel.quantization.config.

ActivationConfig

Field Type Declared default Meaning and normalization
mode str "w4afp8" Exactly w4afp8 or w4a_nvfp4. FP8 carries E4M3 codes plus one FP32 scale per token; NVFP4 carries E2M1 codes, one E4M3 scale per 16 elements, and one FP32 global scale per tensor.
recipe Optional[str] None Forbidden for FP8. For NVFP4, None resolves to least_squares.
attention Optional[ActivationAttentionConfig] None Attention-group override; only on an NVFP4 stream.
mlp Optional[ActivationMlpConfig] None Selected MLP decoder layers promoted to FP8; only on an NVFP4 stream.
global_scales Optional[Dict[str, float]] None NVFP4-only nonempty calibrated producer→scale map. Keys must start with model.layers.; values must be positive finite int/float, not bool, and remain positive finite after FP32 rounding.

Canonical recipes are nvidia, nvidia_headroom, four_six, least_squares, least_squares_headroom, least_squares_grid. The recipe helper expands legacy lsq/lsq_headroom/lsq_grid. Global calibrated scales disallow headroom recipes, including an attention override using headroom. Appropriate W4A backend/hardware support is still required; see FP8 and NVFP4.

ActivationConfig.from_value(value) accepts None, an existing object, a mode string, or a dictionary. Dictionary parsing requires mode and rejects unknown keys. Nested attention/MLP values must be dictionaries with required keys. Base configuration serializes the policy back into a canonical dictionary, omitting absent values.

ActivationAttentionConfig

Field Type Default Normalization
mode str "w4afp8" w4afp8 or w4a_nvfp4.
recipe Optional[str] None FP8 forbids it. NVFP4 requires a resolved recipe; the outer dictionary parser inherits the stream recipe if this key is omitted. Direct construction must supply it.

ActivationMlpConfig

Field Type Default Normalization
mode str "w4afp8" Only w4afp8 is accepted.
layers Tuple[int, ...] () Supply a nonempty list/tuple of unique nonnegative int indices, excluding bool. Duplicates raise, accepted indices sort into a tuple. The bare default constructor therefore raises.

Base activation constraints

An activation stream requires METHOD.GPTQ, GPTQ/GPTQ_V2 format, bits=4, group_size=128, sym=True, desc_act=False, pack_dtype=torch.int32, lm_head=False. Dynamic dictionaries cannot override bits, sym, group_size or desc_act. EXL3's separate post-init does not apply the base activation check; this omission is not evidence of stream support there.

from gptqmodel.quantization import GPTQConfig

config = GPTQConfig(activation={
    "mode": "w4a_nvfp4",
    "recipe": "least_squares",
    "attention": {"mode": "w4afp8"},
    "mlp": {"mode": "w4afp8", "layers": [0, 1]},
})

PreProcessorConfig and preprocessing descriptors

PreProcessorConfig extends the base and is inherited by GPTQ, AWQ, RTN, QQQ, VPTQ, ParoQuant, FP8, BitsAndBytes and GGUF. EXL3 inherits the base directly.

Field Declared type Declared default Meaning and normalization
preprocessors Optional[List[Union[BasePreProcessorConfig, Dict[str, Any], str]]] list() Ordered preprocessing list. Entries normalize to BasePreProcessorConfig objects. None becomes an empty list; duplicate codes are rejected.
smoother Optional[Union[SmootherConfig, SmoothMethod, Dict[str, Any], str]] None SmootherConfig, SmoothMethod, dict, string, or None. Overrides the legacy smooth alias; an enabled smoother is placed last in preprocessors.
smooth Optional[Union[SmoothMethod, Dict[str, Any], str]] None Optional SmoothMethod/string/dict. In PreProcessorConfig this is a legacy alias for smoother; in Fallback/WeightOnlyConfig it configures that flow only.

smoother wins over smooth. If neither is supplied, an existing SmootherConfig in the list becomes the smoother. All smoother entries are removed and the selected smoother is appended last; non-smoother order is preserved. Duplicate remaining code values raise. Preprocessors serialize as meta.preprocessors when the class's metadata hook calls the parent hook; see the serialization caveats below.

Entries can be typed descriptors, SmoothMethod objects, recognized code strings, smoothing-name strings, or dictionaries. preprocessors itself must be a list or None.

BasePreProcessorConfig

Field Declared type Declared default Meaning and normalization
code ClassVar[str] '' Class-level serialized preprocessing identifier; not a dataclass constructor parameter.

SmootherConfig

Field Declared type Declared default Meaning and normalization
code ClassVar[str] PreProcessorCode.SMOOTHER.value Class-level serialized preprocessing identifier; not a dataclass constructor parameter.
smooth Optional[SmoothMethod] None Optional SmoothMethod/string/dict. In PreProcessorConfig this is a legacy alias for smoother; in Fallback/WeightOnlyConfig it configures that flow only.

AutoModuleDecoderConfig

Field Declared type Declared default Meaning and normalization
code ClassVar[str] PreProcessorCode.AUTO_MODULE_DECODER.value Class-level serialized preprocessing identifier; not a dataclass constructor parameter.
source_dtype str 'auto' Automatic dtype detection only: auto after strip/lower.
target_dtype Union[str, torch.dtype] torch.bfloat16 Decoder target: torch.float16 or torch.bfloat16; accepts float16/half/fp16 and bfloat16/bf16 strings.
passthrough_forward_policy str 'native' native or decode after strip/lower; native tries an execution view and decode is the portable fallback.
passthrough_save_policy str 'decode' Only decode is accepted: artifacts contain ordinary dense/quantized weights, never runtime wrapper objects.

TensorParallelPadderConfig

Field Declared type Declared default Meaning and normalization
code ClassVar[str] PreProcessorCode.TENSOR_PARALLEL_PADDER.value Class-level serialized preprocessing identifier; not a dataclass constructor parameter.

TensorParallelPadderConfig has no instance tuning fields: padding is derived from module weight shapes. BasePreProcessorConfig.to_dict() emits {"code": self.code}. SmootherConfig adds a serialized smooth descriptor. Decoder serialization emits canonical dtype strings.

The dictionary preprocessor normalizer currently forwards only source_dtype and target_dtype to AutoModuleDecoderConfig; use a typed descriptor if you need a non-default forward policy. The descriptor itself validates passthrough_forward_policy and passthrough_save_policy.

GPTQConfig

Inherits PreProcessorConfig; adds the following fields. Supported exports are GPTQ, GPTQ_V2, GPTQ_P, MARLIN and BITBLAS at the config level. The model quantization entry point rejects new MARLIN-format quantization: use GPTQ and select the Marlin runtime instead.

Field Declared type Declared default Meaning and normalization
damp_percent Optional[float] None Hessian diagonal damping fraction: None becomes 0.05 for GPTQ or 0.005 for QQQ; must satisfy 0 < value < 1.
damp_auto_increment Optional[float] None Damping retry increment: None becomes 0.01 for GPTQ or 0.001 for QQQ. Validation permits zero and rejects negatives.
act_group_aware Optional[bool] None Activation-group-aware ordering; None becomes True only for METHOD.GPTQ. Mutually exclusive with desc_act=True. See ordering rules.
static_groups bool False Use fixed group assignments during GPTQ quantization rather than recomputing groups under activation ordering.
gptaq Optional[GPTAQConfig] None Optional GPTAQConfig/dict using a full-precision reference path for calibration-error correction.
foem Optional[FOEMConfig] None Optional FOEMConfig/dict for FOEM error correction; see alpha/beta semantics and serialization caveat.
gsq Optional[GSQConfig] None Optional GSQConfig/dict for opt-in Gumbel-Softmax refinement of scalar GPTQ export; see GSQConfig constraints below.
gsq_training Optional[GSQTrainingConfig] None Optional GSQTrainingConfig/dict selecting the staged dense-Llama block-training recipe; mutually exclusive with enabled gsq.
mock_quantization bool False Skip heavy quantization computations for fast loading/structure validation. Not a quality or accuracy mode.
hessian Optional[HessianConfig] HessianConfig() HessianConfig/dict/None; None creates default chunking/staging configuration.

Activation ordering

Ordinary GPTQ defaults to desc_act=False, act_group_aware=True. Explicit desc_act=True with no explicit group-aware setting automatically disables group awareness with a warning. Explicitly enabling both raises. Non-GPTQ descendants default group awareness to False. This is different from constructing BaseQuantizeConfig() directly.

GPTAQConfig

Field Declared type Declared default Meaning and normalization
alpha float 0.25 Error-correction coefficient. GPTAQ defaults to 0.25; FOEM defaults to 0. Numeric values are required but no finite/range bound is imposed here.
device Union[str, torch.device] 'auto' Full-precision reference path placement; nonempty string (default auto) or torch.device.

FOEMConfig

Field Declared type Declared default Meaning and normalization
alpha float 0 Error-correction coefficient. GPTAQ defaults to 0.25; FOEM defaults to 0. Numeric values are required but no finite/range bound is imposed here.
beta float 0.2 FOEM correction coefficient, numeric. alpha=beta=0 reduces to GPTQ; beta=0 with alpha>0 is GPTAQ-style correction.
device Union[str, torch.device] 'auto' Full-precision reference path placement; nonempty string (default auto) or torch.device.

HessianConfig

Field Declared type Declared default Meaning and normalization
chunk_size Optional[int] None Optional maximum staged rows per Hessian chunk; positive int when set.
chunk_bytes Optional[int] None Optional staging budget in bytes; positive int when set. Unlike MSE budgets, postfix strings are not accepted.
staging_dtype Union[str, torch.dtype] torch.float32 Hessian staging dtype: float32, float16, or bfloat16, as a torch dtype or lowercased string.
dedup_shared_inputs bool True Collect a Hessian once for explicit model-tree :in= shared-input groups and copy to peers. Must be bool.

MSEConfig

Field Declared type Declared default Meaning and normalization
norm float 0.0 Exponent in abs(dequantized - original) ** norm; accepts int/float except bool and converts to float. Search is active only when norm > 0. This hook does not require a positive finite exponent.
grid_budget Optional[Union[int, str]] None GPTQ/QQQ candidate-chunk workspace target. Nonnegative byte int, numeric string, or K/M/G postfix string with optional B; None selects the device default.
clip_grid_budget Optional[Union[int, str]] None AWQ clip-grid per-scratch-buffer target, using the same units/defaults as grid_budget. Not a process-wide memory cap.

GPTAQ and FOEM device validation only requires a nonempty string or torch.device; it does not resolve auto at config construction. FOEM's documented special cases are alpha=beta=0 (GPTQ), alpha>0/beta=0 (GPTAQ), alpha=0/beta>0 (FOEM), and both positive (combined).

MSE budgets are binary units: "256M" is 256 MiB, "1GB" is 1 GiB, and digit-only strings are bytes. Negative integers, booleans, fractional-unit strings and unsupported units raise. None selects 16 MiB on CPU, 256 MiB on other devices. Zero requests single-candidate chunking (and one-token AWQ product tiles). Budgets size individual chunks/buffers; they cannot cap total memory or make a minimum candidate smaller than its required workspace. bool(MSEConfig) tests norm > 0.

GSQConfig

Optional Gumbel-Softmax refinement of an existing scalar grid, accepted on GPTQConfig, AWQConfig, RTNConfig, QQQConfig, FP8Config, and ParoConfig as a GSQConfig, a dictionary, or None. Per-method format/bit constraints are validated on the owning config; see GSQ.

Field Declared type Declared default Meaning and normalization
enabled bool False Master switch; must be bool.
steps int 100 Refinement steps per layer; integer >= 1.
candidates int 33 Hard-code candidates per row chunk; integer >= 2.
seed int 7 Refinement RNG seed; integer >= 0.
learning_rate float 0.1 Candidate optimizer rate; finite positive.
temperature_start float 1.0 Initial Gumbel-Softmax temperature; finite positive.
temperature_end float 0.1 Final temperature; finite positive.
max_candidate_bytes int 64 * 1024**2 Candidate-bank byte budget per output-row chunk; integer >= 1. Not a total-memory cap.
learn_scales bool False Tune stored scales where the method permits; rejected by QQQ, FP8, and ParoQuant.
modules Optional[Tuple[str, ...]] None Nonempty sequence of PCRE regexes restricting refinement to matching module full names; each must compile.

GSQTrainingConfig

Opt-in staged dense-Llama block training, accepted on GPTQConfig (initializer gptq/rtn) and AWQConfig (initializer awq). Dictionary input is normalized; on AWQConfig a missing initializer defaults to awq. Full staged-path requirements are on GSQ.

Field Declared type Declared default Meaning and normalization
enabled bool False Master switch; must be bool.
initializer str 'gptq' gptq, rtn, or awq; must match the owning config's method.
optimizer str 'lion' lion or adamw.
seed int 7 Training RNG seed; integer >= 0.
epochs int 20 Reconstruction epochs per stage; integer >= 1.
batch_size int 64 Documents per training batch; integer >= 1.
microbatch_size int 2 Microbatch documents; integer >= 1 and <= batch_size.
qk_steps int 2000 Update steps for the Q/K stage; integer >= 1.
damp_percent float 0.01 Staged Hessian damping fraction; finite nonnegative. Renamed qk_damp_percent by training_kwargs().
assignment_lr float 1e-4 Assignment learning rate; finite positive.
scale_lr float 5e-5 Scale learning rate; finite positive.
weight_decay float 1.0 Optimizer weight decay; finite nonnegative.
betas tuple[float, float] (0.9, 0.95) Two optimizer moments, each in [0, 1).
temperature tuple[float, float] (2.0, 0.05) (start, end) sampling-temperature endpoints; both positive.
multiplier tuple[float, float] (100.0, 500.0) (start, end) logit-scale multiplier endpoints; both positive.
warmup_steps int 0 LR warmup steps; integer >= 0.
min_lr float 0.1 Final LR fraction; finite, <= 1.
decay str 'cosine' LR decay: cosine, linear, or constant.
restore_best bool | None None None auto-selects the best held-out validation epoch when validation batches exist (else the last epoch); True requires validation batches every stage; False always exports the last epoch.

RTNConfig and legacy WeightOnlyConfig

RTNConfig fields

Field Declared type Declared default Meaning and normalization
method METHOD METHOD.GPTQ Algorithm family, normalized to METHOD. Concrete classes validate their allowed family; the factory may repair incompatible family hints.
format FORMAT FORMAT.GPTQ Authoritative serialized layout. Usually a FORMAT value; GGUF, FP8, and BitsAndBytes instead expose a subtype string and resolve their family separately.
gsq Optional[GSQConfig] None Optional GSQConfig/dict; scalar GPTQ or AWQ export formats only, AWQ-family formats require bits 4.

RTNConfig uses a weight-only lifecycle and GPTQ-family packing by default; it has no Hessian/damping fields. Config-level export choices are GPTQ, GPTQ_V2, GPTQ_P, GEMM, GEMV, GEMV_FAST and LLM_AWQ. export_quant_method() determines the output family from the layout, so AWQ-shaped RTN exports advertise the AWQ family.

Legacy WeightOnlyConfig fields

Field Declared type Declared default Meaning and normalization
method WeightOnlyMethod WeightOnlyMethod.RTN WeightOnlyMethod enum/string. Normalized lowercase. See factory support caveat for nvfp4.
smooth Optional[SmoothMethod] None Optional SmoothMethod/string/dict. In PreProcessorConfig this is a legacy alias for smoother; in Fallback/WeightOnlyConfig it configures that flow only.

WeightOnlyConfig is a legacy dispatch descriptor, not an inherited field on the concrete configs. QuantizeConfig(weight_only=...) routes to RTN, GGUF, FP8 or BitsAndBytes and transfers smoothing. WeightOnlyMethod contains NVFP4, but the factory explicitly rejects weight_only.method="nvfp4"; use an actual supported NVFP4/GGUF or activation policy rather than assuming every enum member is a direct lifecycle.

AWQConfig

Field Declared type Declared default Meaning and normalization
method METHOD METHOD.AWQ Algorithm family, normalized to METHOD. Concrete classes validate their allowed family; the factory may repair incompatible family hints.
format FORMAT FORMAT.GEMM Authoritative serialized layout. Usually a FORMAT value; GGUF, FP8, and BitsAndBytes instead expose a subtype string and resolve their family separately.
scale_search_chunked_activations bool True Stream/chunk AWQ reference and reconstruction activations to reduce peak scale-search memory; saved in meta.
gsq Optional[GSQConfig] None Optional GSQConfig/dict; 4-bit GEMM/GEMV/GEMV_FAST/LLM_AWQ only, non-GEMM also requires group_size in -1, 32, 64, 128.
gsq_training Optional[GSQTrainingConfig] None Optional GSQTrainingConfig/dict; staged training requires initializer awq and 4-bit GEMM export. Mutually exclusive with enabled gsq.

AWQ defaults to desc_act=False. Valid config exports are GEMM, GEMV, GEMV_FAST, MARLIN, BITBLAS and LLM_AWQ; unsupported recognized formats are repaired to GEMM. Export writes zero_point = not sym and version = format. GEMV_FAST and LLM_AWQ packing are forced to int16 by the model quantization entry point. New MARLIN-format quantization is rejected there even though the config accepts the legacy layout.

QQQConfig

Field Declared type Declared default Meaning and normalization
method METHOD METHOD.QQQ Algorithm family, normalized to METHOD. Concrete classes validate their allowed family; the factory may repair incompatible family hints.
format FORMAT FORMAT.QQQ Authoritative serialized layout. Usually a FORMAT value; GGUF, FP8, and BitsAndBytes instead expose a subtype string and resolve their family separately.

Inherits all GPTQConfig fields; method/format are QQQ. Damping defaults differ: 0.005 and 0.001. desc_act=True is rejected globally and in any dynamic dictionary. QQQ's packed runtime has contiguous group scales and no GPTQ g_idx buffer. to_dict() additionally writes wbits for vLLM's QQQ loader. Config bit parsing is inherited, but the QQQ runtime declares 4-bit support, not all base widths. The inherited gsq field is accepted only with FORMAT.QQQ export and learn_scales=False; staged gsq_training is rejected outright for QQQ packing.

VPTQConfig

Field Declared type Declared default Meaning and normalization
method METHOD METHOD.VPTQ Algorithm family, normalized to METHOD. Concrete classes validate their allowed family; the factory may repair incompatible family hints.
format FORMAT FORMAT.VPTQ Authoritative serialized layout. Usually a FORMAT value; GGUF, FP8, and BitsAndBytes instead expose a subtype string and resolve their family separately.
bits int 2 Nominal label derived as ceil((log2(main centroids) + log2(residual centroids, if enabled)) / main vector length), clamped to 1..8. User input is ignored if different; defaults derive 2.
vector_lens Union[List[int], Tuple[int, int]] [-1, 8] [outlier-vector length, main-vector length], normalized to integer list. Main length > 0; outlier length -1 (disabled) or > 0.
num_centroids Union[List[int], Tuple[int, int]] [-1, 4096] [outlier, main] codebook sizes, normalized to integer list. -1 disables a codebook; other entries must be positive powers of two. Main needs >= 2.
num_res_centroids Union[List[int], Tuple[int, int]] [-1, 16] [outlier, main] residual codebook sizes. Outlier must be -1. Main is -1 or a power of two >= 2; combined main/residual index width <= 32.
npercent float 0.0 Nonnegative percentage of input channels assigned to outlier codebook; >0 requires enabled outlier vector length and >=2 outlier centroids.
group_size int -1 Input channels per codebook; -1 lets group_num decide. Cannot set both group_size and group_num. This is not ordinary scalar-quantization group scaling.
group_num int -1 Number of main codebooks; -1 or positive. Cannot be set together with a non--1 group_size.
kmeans_mode str 'hessian' Weighted k-means selector: empty string for uniform weighting or hessian.
kmeans_iters int 100 Positive maximum k-means iterations.
kmeans_tol float 1e-05 K-means convergence tolerance, strictly between 0 and 1.
kmeans_seed int 0 K-means random seed.
enable_norm bool False Enable per-channel min/max normalization before vector quantization.
norm_dim int 0 Normalization dimension, 0 or 1.
enable_perm bool False Permute input channels by descending Hessian diagonal before VPTQ quantization.
block_size int 128 Positive VPTQ error-propagation block size.
percdamp float 0.01 VPTQ Hessian damping fraction in [0, 1). Separate from inherited GPTQ damp_percent.

VPTQ inherits GPTQConfig but derives nominal bits from vector/codebook settings rather than treating bits as a scalar packing width. For default main codebooks, (log2(4096)+log2(16))/8 = 2 nominal bits per weight, excluding centroid/metadata overhead. Input channels form codebook groups; vectors run along the output-feature dimension.

Outlier residual codebooks are unsupported. npercent has no upper-bound check here. Nonempty dynamic override dictionaries are rejected (configure codebooks globally); use documented negative rules with empty dictionaries to exclude modules, not arbitrary per-layer codebook changes. Packed main+residual index widths must total <=32. kmeans_tol is strictly (0,1), while percdamp allows zero and excludes one.

ParoConfig

Field Declared type Declared default Meaning and normalization
method METHOD METHOD.PARO Algorithm family, normalized to METHOD. Concrete classes validate their allowed family; the factory may repair incompatible family hints.
format FORMAT FORMAT.PAROQUANT Authoritative serialized layout. Usually a FORMAT value; GGUF, FP8, and BitsAndBytes instead expose a subtype string and resolve their family separately.
krot int 8 Number of ParoQuant Givens-rotation stages; int-coerced and required positive. Passed to quantized-layer construction and saved at top level.
opt_rotation_epochs int 10 Rotation optimization epochs; int-coerced, nonnegative.
opt_finetune_epochs int 10 Weight/quantizer fine-tuning epochs; int-coerced, nonnegative.
opt_train_samples int 2048 Optimization training-sample budget; int-coerced, positive.
opt_validation_samples int 64 Validation-sample budget for selecting the best state; int-coerced, positive.
opt_batch_size int 64 Optimization minibatch size; int-coerced, positive.
opt_rotation_lr float 0.05 Rotation learning rate; float-coerced, positive.
opt_weight_lr float 1e-05 Weight learning rate; float-coerced, positive.
opt_quantizer_lr float 1e-06 Quantizer learning rate; float-coerced, positive.
opt_pair_ratio float 0.5 Fraction of channels paired in a rotation stage; float-coerced in (0, 0.5].
opt_seed int 0 Optimization RNG seed, int-coerced.
opt_optimizer str 'adamw' Lowercased optimizer selector: adamw, adam, or sgd.
opt_weight_decay float 0.01 Nonnegative float optimizer weight decay.
opt_betas Tuple[float, float] (0.9, 0.95) Two float Adam/AdamW moments, each in [0, 1); accepts a 2-item list/tuple and normalizes to tuple.
opt_eps float 1e-10 Positive float Adam/AdamW epsilon.
opt_amsgrad bool False bool-coerced Adam/AdamW AMSGrad switch.
opt_sgd_momentum float 0.0 Nonnegative float SGD momentum; must be positive with Nesterov.
opt_sgd_dampening float 0.0 Nonnegative float SGD dampening; must be zero with Nesterov.
opt_sgd_nesterov bool False bool-coerced SGD Nesterov toggle; requires momentum > 0 and dampening == 0.
opt_fused_rotation bool True bool-coerced fused rotation optimization toggle.
opt_gradient_checkpointing Optional[bool] None None resolves to True only for opt_scope=layer, otherwise False. Recognizes string true/false, yes/no, on/off, 1/0, y/n, t/f; other strings raise.
opt_stage_cudagraph bool True bool-coerced stage CUDA-graph optimization toggle.
opt_amp Optional[bool] None Optional bool-coerced automatic mixed precision policy; None leaves selection to the optimizer.
opt_best_state_dtype Union[str, torch.dtype] 'fp32' Best-state snapshot dtype, canonical fp16/bf16/fp32. Also accepts float16/bfloat16/float32 strings and matching torch.dtype; None becomes fp32.
opt_train_on_noisy_inputs bool False bool-coerced selection of quantization-noisy optimization inputs.
opt_scope str 'module' Lowercased optimization scope: module, compute_block, or layer.
opt_stage_impl str 'fast' Lowercased stage implementation: fast or reference.
opt_pair_impl str 'fast' Lowercased channel-pair implementation: fast or reference.
opt_quantizer_impl str 'reference' Lowercased quantizer implementation: fast or reference.
opt_channel_scale_clamp_min float PAROQUANT_OPT_SCALE_CLAMP_MIN_DEFAULT Positive float channel-scale lower bound, strictly less than the upper bound. Constant default resolves to 0.01.
opt_channel_scale_clamp_max float PAROQUANT_OPT_SCALE_CLAMP_MAX_DEFAULT Positive float channel-scale upper bound, strictly greater than the lower bound. Constant default resolves to 100.0.
gsq Optional[GSQConfig] None Optional GSQConfig/dict; requires bits 4, opt_scope module, and opt_train_on_noisy_inputs False.

ParoQuant's algorithm token is METHOD.PARO = "paroquant", not "paro". format is repaired to PAROQUANT. Integer/float fields are explicitly coerced before their bounds are checked; boolean-like options use Python bool conversion except for the explicit gradient-checkpointing string parser. Channel-scale clamp defaults are 1e-2 and 1e2. Optimizer controls are stored under meta; krot is top-level and passed to layer constructors.

FP8Config

Field Declared type Declared default Meaning and normalization
bits int 8 Must normalize to 8; other widths raise.
method METHOD METHOD.FP8 Algorithm family, normalized to METHOD. Concrete classes validate their allowed family; the factory may repair incompatible family hints.
format Optional[str] 'float8_e4m3fn' Authoritative serialized layout. Usually a FORMAT value; GGUF, FP8, and BitsAndBytes instead expose a subtype string and resolve their family separately.
group_size int -1 Forced to -1; use weight_scale_method and weight_block_size to choose FP8 scaling granularity.
desc_act Optional[bool] False Fixed False by FP8 normalization.
sym bool True Forced True; this field does not select an asymmetric FP8 path.
weight_scale_method str 'row' FP8 scaling granularity: tensor, row, or block. block requires weight_block_size; a block size is forbidden with other methods.
weight_block_size Optional[Union[List[int], Tuple[int, int]]] None Optional 2-item positive integer [rows, cols] list/tuple, stored as list. Through the factory, specifying this without a scaling method infers block.
weight_scale_semantics str 'inverse' Only inverse is accepted: quantization multiplies dense weights by scale_inv; reconstruction divides stored FP8 values by scale_inv.
gsq Optional[GSQConfig] None Optional GSQConfig/dict; requires fixed inverse scales (learn_scales rejected) and finite E4M3/E5M2 weight storage (float8_e8m0fnu rejected).
gsq_calibration bool False Collect calibration statistics for FP8 GSQ through the looper; requires enabled gsq and weight_scale_method row or block. Flips FP8 off the weight-only lifecycle.

FP8 forces group_size=-1, desc_act=False, sym=True and requires bits=8. Config construction checks that the installed Torch provides the selected float8 dtype. Hardware-native arithmetic support is a separate backend requirement. The factory infers block scaling when a block size is given without a scaling method; direct FP8Config(weight_block_size=[...]) retains its declared method row and therefore requires explicitly setting weight_scale_method="block".

BitsAndBytesConfig

Field Declared type Declared default Meaning and normalization
bits int 4 Must normalize to 4 or 8, with a matching format subtype.
method METHOD METHOD.BITSANDBYTES Algorithm family, normalized to METHOD. Concrete classes validate their allowed family; the factory may repair incompatible family hints.
format Optional[str] None Authoritative serialized layout. Usually a FORMAT value; GGUF, FP8, and BitsAndBytes instead expose a subtype string and resolve their family separately.
group_size int -1 Forced to -1; block_size controls BitsAndBytes block quantization.
desc_act Optional[bool] False Fixed False by BitsAndBytes normalization.
sym bool True Forced True; the fp4, nf4 or int8 subtype defines the encoding.
block_size int 64 BitsAndBytes block-quantization size; allowed values are listed below.
compress_statistics bool True bool-coerced BitsAndBytes nested/double statistics quantization toggle.

BitsAndBytes forces group_size=-1, desc_act=False, sym=True. Four-bit formats are fp4 (default) and nf4; eight-bit format is int8. Block sizes are exactly 32, 64, 128, 256, 512, 1024, 2048 or 4096. bnb_quant_type, bnb_block_size and bnb_compress_statistics are compatibility properties/JSON aliases; they are not additional dataclass fields. Execution/packing needs the bitsandbytes implementation except for portable consumers such as the MLX packed-checkpoint path.

EXL3Config

Field Declared type Declared default Meaning and normalization
bits float 3.0 Weight precision; normalization and method-specific restrictions are described below. Kernel validation can be narrower than configuration parsing.
method METHOD METHOD.EXL3 Algorithm family, normalized to METHOD. Concrete classes validate their allowed family; the factory may repair incompatible family hints.
format FORMAT FORMAT.EXL3 Authoritative serialized layout. Usually a FORMAT value; GGUF, FP8, and BitsAndBytes instead expose a subtype string and resolve their family separately.
group_size int -1 Forced to -1 by EXL3 normalization; not a GPTQ-style group-size tuning knob.
desc_act Optional[bool] False Fixed False by EXL3 normalization.
sym bool True Forced True by EXL3 normalization.
head_bits Optional[float] None Optional EXL3 fractional output-head bits-per-weight target; finite in [1, 8]. None inherits the main target in logging/quantization paths.
out_scales Optional[str] 'auto' EXL3 output-scale policy. Strings true/false map to always/never; none maps to auto; always, never, auto accepted. Python None remains None.
codebook str 'mcg' EXL3 codebook selector normalized with strip/lower: mcg, mul1, or 3inst.
tensor_storage Optional[Dict[str, Any]] None Optional EXL3 tensor storage descriptor dictionary, built by the writer for saved models.
calibration Optional[Dict[str, int]] None Optional EXL3 calibration metadata dictionary; keys become strings and values ints.

EXL3 implements a separate post-init instead of calling the base hook. It validates fractional targets, family/format, fallback, dynamic bits, output scales, codebook, tensor/calibration metadata, adapter, offload, placement, GC and MoE. It forces whole-row/symmetric/no-desc-act settings. It does not perform all base normalizations, notably telemetry/MSE/activation/calibration_data_device; retain typed defaults and do not assume all base-field dictionaries behave identically for EXL3. runtime_bits is max(1, floor(bits)); the serialized target remains float.

GGUFConfig and structured bits

GGUFConfig fields

Field Declared type Declared default Meaning and normalization
bits Union[int, str, GGUFBits] 4 Weight precision; normalization and method-specific restrictions are described below. Kernel validation can be narrower than configuration parsing.
format Optional[str] None Authoritative serialized layout. Usually a FORMAT value; GGUF, FP8, and BitsAndBytes instead expose a subtype string and resolve their family separately.
method METHOD METHOD.GGUF Fixed GGUF, init=False: not a direct GGUFConfig constructor argument.
group_size int -1 Fixed -1, init=False. Block geometry belongs to the chosen GGUF subtype.
desc_act Optional[bool] False Fixed False, init=False: not a GGUF constructor argument.
sym bool True Fixed True, init=False. This marker does not override the subtype's scale/minimum or codebook semantics.
_gguf_bits GGUFBits computed; init=False Internal computed GGUFBits runtime specification; not accepted in init and excluded from repr/equality.

GGUF's method/group_size/desc_act/sym fields are init=False; direct constructors must not pass them. The factory filters them out. Public bits is an integer and format a subtype such as q_0, q_k_m, iq_xxs, mxfp4 or nvfp4; runtime_bits holds the combined GGUFBits object. A full alias in bits and an explicit incompatible subtype raise rather than silently changing one. There is no seven-bit GGUF config.

GGUFBits fields

Field Declared type Declared default Meaning and normalization
bits int required Positive integer logical width. from_string additionally enforces a registered alias; raw component construction validates components, not membership in the alias table.
version str required GGUFBits encoding family: q, iq, tq, mxfp, or nvfp.
variant str required GGUFBits subtype component: empty string, 0, 1, k, xxs, xs, s, m, or nl.
quality Optional[str] None Optional GGUF quality tag: None, xs, s, m, l, or g128.

QuantBits is an alias of GGUFBits, not a second class. GGUFBits.from_string() / .from_alias() parse registered names with strip/lower and hyphen-to-underscore normalization; .to_string() / .serialize() produce the full alias and .to_public_format() removes width except for width-in-name formats. BaseComplexBits supplies integer/index/float conversion, width/name properties, width-based arithmetic and ordering; equality between complex specifications compares the full alias, while equality to an integer compares width. Do not treat two same-width GGUF encodings as identical storage.

Every registered GGUF bits alias

Full alias Width Public format Approximate BPW used by config logging
q1_0 1 q_0 1.125
q1_0_g128 1 q_0_g128 1.125
q2_0 2 q_0 2.25
q4_0 4 q_0 4.5
q4_1 4 q_1 5.0
q5_0 5 q_0 5.5
q5_1 5 q_1 6.0
q8_0 8 q_0 8.5
q8_1 8 q_1 9.0
q2_k 2 q_k 2.625
q2_k_s 2 q_k_s 2.625
q2_k_m 2 q_k_m 2.625
q3_k 3 q_k 3.4375
q3_k_s 3 q_k_s 3.4375
q3_k_m 3 q_k_m 3.4375
q3_k_l 3 q_k_l 3.4375
q4_k 4 q_k 4.5
q4_k_s 4 q_k_s 4.5
q4_k_m 4 q_k_m 4.5
q5_k 5 q_k 5.5
q5_k_s 5 q_k_s 5.0
q5_k_m 5 q_k_m 5.5
q6_k 6 q_k 6.0
q8_k 8 q_k 9.125
iq1_s 1 iq_s 1.5625
iq1_m 1 iq_m 1.75
iq2_xxs 2 iq_xxs 2.0625
iq2_xs 2 iq_xs 2.3125
iq2_s 2 iq_s 2.5625
iq3_xxs 3 iq_xxs 3.0625
iq3_s 3 iq_s 3.4375
iq4_nl 4 iq_nl 4.5
iq4_xs 4 iq_xs 4.25
tq1_0 1 tq_0 1.6875
tq2_0 2 tq_0 2.0625
mxfp4 4 mxfp4 4.25
nvfp4 4 nvfp4 4.5

Default aliases by integer width are 1→q1_0, 2→q2_0, 3→q3_k, 4→q4_0, 5→q5_k_m, 6→q6_k, 8→q8_0. Alias recognition is a config capability, not a guarantee that every GGUF backend implements that subtype; consult GGUF.

Generic bits and FP8 format strings

The generic parser accepts integer widths, numeric strings and integral floats (nonintegral values require EXL3's override). EXL3 accepts float-convertible strings and finite targets in [1.0,8.0]. GPTQ 5/6/7 automatically changes the entire checkpoint family to gptq_p, including when only a positive dynamic rule uses one of those widths. Explicit gptq_p supports 2..8, but kernels require int32 words and planar alignment.

Accepted FP8 aliases Canonical format
e4m3, float8_e4m3, float8_e4m3fn float8_e4m3fn
e5m2, float8_e5m2 float8_e5m2
e4m3fnuz, float8_e4m3fnuz float8_e4m3fnuz
e5m2fnuz, float8_e5m2fnuz float8_e5m2fnuz
e8m0, e8m0fnu, float8_e8m0, float8_e8m0fnu float8_e8m0fnu

None, empty string, or the fp8 family token defaults to float8_e4m3fn. BitsAndBytes similarly accepts a family token/empty string and selects fp4 or int8 from bits. FP8/BitsAndBytes family resolution uses the method as well as subtype strings; do not coerce these public subtype strings through FORMAT(...).

Fallback and smoothing

Field Declared type Declared default Meaning and normalization
strategy FallbackStrategy FallbackStrategy.RTN Fallback center/range rule; normalized to FallbackStrategy when nested in BaseQuantizeConfig.
threshold int | float | str '0.5%' Trigger below this captured-token count or calibration percentage string. Fallback() defaults to 0.5%; dict normalization without a threshold uses 1.0%.
smooth Optional[SmoothMethod] None Optional SmoothMethod/string/dict. In PreProcessorConfig this is a legacy alias for smoother; in Fallback/WeightOnlyConfig it configures that flow only.

fallback=None disables fallback. Scalar strings/numbers are thresholds and select RTN; dictionaries normalize strategy and smoothing. Be explicit about threshold: Fallback() uses "0.5%", but fallback={} uses "1.0%". This is a strict low-coverage trigger: fewer captured module-forward tokens than the absolute count, or the percentage relative to calibration. It is useful when expert routing underexposes modules.

FallbackStrategy Center / range rule
RTN / rtn Quantizer min/max round-to-nearest
MIDPOINT / midpoint Center (min+max)/2, range max-min
MEAN / mean Mean center, range 2*max(abs(w-mean))
MEDIAN / median Median center, range 2*max(abs(w-median))
STDCLIP / stdclip Mean center with standard-deviation clipping

Fallback dictionary compatibility accepts smooth_method/clip_method and legacy knobs smooth_percentile, smooth_mad_k, smooth_mse_steps, smooth_mse_maxshrink, smooth_outlier_pct, smooth_rms_k, smooth_log_mu, smooth_axis. Prefer a typed smoother or a smooth dictionary. Smoothing remains opt-in for fallback and whole-model RTN.

Every Smooth method

All methods inherit name and group_size_threshold=128 from SmoothMethod. The concrete constructors set name; do not pass it yourself. Their constructors store parameters without general range validation. Dict/string parsing uses type or name; none disables smoothing, and missing type/name produces None. Serialized descriptors use type and group_size_threshold.

SmoothMethod

Field Declared type Declared default Meaning and normalization
name str required Smoother name identifying the algorithm in serialized descriptors.
group_size_threshold int 128 Apply smoothing only when the quantization group size reaches this threshold; inherited by every smoother.

SmoothPercentile

Field Declared type Declared default Meaning and normalization
percentile float 99.0 Percentile used for magnitude clipping (or companded log magnitudes in SmoothLog). 
def __init__(self, percentile: float = 99.0, group_size_threshold: int = 128):
    ...

SmoothPercentileAsymmetric

Field Declared type Declared default Meaning and normalization
low float 0.5 Lower raw-weight percentile for asymmetric clipping.
high float 99.5 Upper raw-weight percentile for asymmetric clipping. 
def __init__(self, low: float = 0.5, high: float = 99.5, group_size_threshold: int = 128):
    ...

SmoothMAD

Field Declared type Declared default Meaning and normalization
k float 2.75 MAD clipping multiplier in SmoothMAD; normalized RMS/z-score bound in SmoothSoftNorm. 
def __init__(self, k: float = 2.75, group_size_threshold: int = 128):
    ...

SmoothMSE

Field Declared type Declared default Meaning and normalization
steps int 32 Shrink-grid candidate count; the descriptor constructor does not validate it.
maxshrink float 0.8 Smallest range multiplier in SmoothMSE candidate search. 
def __init__(self, steps: int = 32, maxshrink: float = 0.8, group_size_threshold: int = 128):
    ...

SmoothOutlier

Field Declared type Declared default Meaning and normalization
pct float 1.0 Percentage of highest-magnitude values treated as outliers. 
def __init__(self, pct: float = 1.0, group_size_threshold: int = 128):
    ...

SmoothSoftNorm

Field Declared type Declared default Meaning and normalization
k float 3.0 MAD clipping multiplier in SmoothMAD; normalized RMS/z-score bound in SmoothSoftNorm. 
def __init__(self, k: float = 3.0, group_size_threshold: int = 128):
    ...

SmoothLog

Field Declared type Declared default Meaning and normalization
percentile float 99.0 Percentile used for magnitude clipping (or companded log magnitudes in SmoothLog).
mu float 8.0 Log companding strength for log1p(mu*abs(weight)) clipping and inversion. 
def __init__(self, percentile: float = 99.0, mu: float = 8.0, group_size_threshold: int = 128):
    ...

SmoothRowCol

Field Declared type Declared default Meaning and normalization
axis str 'row' row or col RMS normalization direction for SmoothRowCol. 
def __init__(self, axis: str = "row", group_size_threshold: int = 128):
    ...

SmoothMAD() has constructor default k=2.75, but dictionary/string parsing with no k uses 3.0. percentile_asymmetric is accepted as an alias for percentile_asym. Percentile clips magnitudes; asymmetric percentile clips raw tails; MAD clips around the median; MSE searches a shrink range; Outlier clips highest magnitudes; SoftNorm clips normalized deviations; Log clips companded magnitudes and inverts the threshold; RowCol normalizes by row/column RMS and carries the scale through quantization. These settings are not interchangeable with the MSEConfig quantizer objective/workspace controls.

NormConfig and embedding targeting

NormConfig fields

Field Declared type Declared default Meaning and normalization
teacher Any None Required unquantized reference model path/HF id or loaded module for normalization distillation; None raises.
types Tuple[NormType, ...] (NormType.RMSNORM,) Normalization families to tune. Accepts one string/NormType or an iterable, deduplicates in order to a nonempty tuple; None selects RMSNorm.
steps int 400 Positive normalization-tuning step count.
lr float 0.001 Finite positive normalization-tuning learning rate.
reg float 0.001 Finite nonnegative L2 pull toward each checkpoint parameter, relative to its mean absolute effective value.
seed int 0 Normalization-tuning RNG seed.
skip Tuple[str, ...] () Tagged module suffixes excluded from normalization tuning; normalized to tuple.

NormConfig is a requantize(norm=...) argument, not a BaseQuantizeConfig field. It distills next-token distributions from an unquantized teacher while training only eligible model-tree :norm modules plus pre_lm_head_norm_module; packed quantized tensors are not modified. RMSNorm includes mean-free gain normalization and Gemma-style 1 + weight; LayerNorm includes mean-centered normalization with optional bias. Pass a teacher explicitly; NormConfig() alone raises.

QuantizeEmbedConfig fields

Field Declared type Declared default Meaning and normalization
embed_quant_mode QuantizeEmbed QuantizeEmbed.OUTPUT Which embedding boundary to quantize: input, output, or both. Pass a QuantizeEmbed enum via the embedding APIs, not a BaseQuantizeConfig field.
embed_only bool True True limits the embedding-targeted operation to embeddings; False permits the rest of the model quantization flow too.

QuantizeEmbedConfig likewise belongs to embedding quantization/requantization APIs (embed_quant_config), not the factory field table. QuantizeEmbed.INPUT, .OUTPUT, .BOTH select boundaries. The model normalizer accepts a QuantizeEmbed enum or QuantizeEmbedConfig, and rejects supplying embed_quant_config together with legacy embed_quant_mode. Embedding-only flows can disable disk offload. See Quantization Lifecycle.

MoE routing

MoEConfig

Field Declared type Declared default Meaning and normalization
routing BaseMoERouting required Required BaseMoERouting instance; a tagged dictionary is normalized only when supplied through BaseQuantizeConfig.moe.

ExpertsRoutingOverride

Field Declared type Declared default Meaning and normalization
num_experts_per_tok Union[int, str] MOE_ALL_EXPERTS Positive int/numeric string or all (trimmed, lowercase). all resolves to expert count; a larger integer is clamped to available experts.

ExpertsRoutingBypass

Field Declared type Declared default Meaning and normalization
batch_size Optional[int] None Optional number of expert modules per bypass batch to limit VRAM. None processes all together; no positivity check in this dataclass.

BaseMoERouting is an empty marker dataclass. MoEConfig.to_dict() writes {"routing": {"class": "ExpertsRoutingOverride" or "ExpertsRoutingBypass", ...}}; dictionary parsing requires that class tag. Override changes effective top-k; bypass forwards the full calibration data to each expert and can batch modules to reduce memory. These controls affect calibration, not the final model's intended inference routing policy.

AdapterConfig / Lora

There is no AdapterConfig class or export in this source revision. The adapter field accepts the runtime Lora object or a data-only dictionary with name="lora". Do not import an invented AdapterConfig; use:

from gptqmodel.adapter.adapter import Lora

# Source signature:
# def __init__(self, rank: int, path: str = None, lora_A: torch.Tensor = None, lora_B: torch.Tensor = None):
adapter = Lora(rank=16, path="./eora")
Parameter Default Meaning
rank: int required Adapter rank; loading can resolve an unspecified rank from saved PEFT config.
path: str None Adapter directory/repository, not a .safetensors filename. Paths are stripped; path-like values become filesystem strings. Remote loading has separate pin/integrity validation.
lora_A: torch.Tensor None Optional preloaded first projection.
lora_B: torch.Tensor None Optional preloaded second projection.

normalize_adapter() accepts Adapter instances or a dictionary, requires a registered name (currently only lora), and rejects non-tensor supplied projection values. Lora.to_dict() emits name/path/rank, not matrix payloads. The adapter update is (x @ lora_A) @ lora_B, added to output; matrices follow input device/dtype. EoRA is saved as PEFT-style adapter_config.json and adapter_model.safetensors, with dynamic adapter rank patterns handled separately by the writer. See EoRA.

Enum reference

METHOD

Member Serialized value
METHOD.GPTQ gptq
METHOD.GGUF gguf
METHOD.FP8 fp8
METHOD.BITSANDBYTES bitsandbytes
METHOD.QQQ qqq
METHOD.AWQ awq
METHOD.EXL3 exl3
METHOD.PARO paroquant
METHOD.VPTQ vptq

FORMAT

Member Serialized value
FORMAT.GPTQ gptq
FORMAT.GPTQ_V2 gptq_v2
FORMAT.GPTQ_P gptq_p
FORMAT.GGUF gguf
FORMAT.FP8 fp8
FORMAT.BITSANDBYTES bitsandbytes
FORMAT.MARLIN marlin
FORMAT.BITBLAS bitblas
FORMAT.QQQ qqq
FORMAT.EXL3 exl3
FORMAT.VPTQ vptq
FORMAT.GEMM gemm
FORMAT.GEMV gemv
FORMAT.GEMV_FAST gemv_fast
FORMAT.LLM_AWQ llm-awq
FORMAT.PAROQUANT paroquant

WeightOnlyMethod

Member Serialized value
WeightOnlyMethod.RTN rtn
WeightOnlyMethod.GGUF gguf
WeightOnlyMethod.FP8 fp8
WeightOnlyMethod.BITSANDBYTES bitsandbytes
WeightOnlyMethod.NVFP4 nvfp4

VramStrategy

Member Serialized value
VramStrategy.EXCLUSIVE exclusive
VramStrategy.BALANCED balanced

NormType

Member Serialized value
NormType.RMSNORM rmsnorm
NormType.LAYERNORM layernorm

GcMode

Member Serialized value
GcMode.INTERVAL interval
GcMode.ON_STAGE_END on_stage_end

QuantizeEmbed

Member Serialized value
QuantizeEmbed.INPUT input
QuantizeEmbed.OUTPUT output
QuantizeEmbed.BOTH both

PreProcessorCode

Member Serialized value
PreProcessorCode.SMOOTHER smoother
PreProcessorCode.AUTO_MODULE_DECODER auto_module_decoder
PreProcessorCode.TENSOR_PARALLEL_PADDER tensor_parallel_padder

FallbackStrategy is listed with its behavior above. Norm type strings are validated as enum values rather than generally lowercased; supply rmsnorm/layernorm exactly. PreProcessorCode values identify passes, not quantization algorithms.

Dynamic per-layer overrides

from gptqmodel.quantization import GPTQConfig, MSEConfig

qcfg = GPTQConfig(
    bits=4,
    group_size=128,
    mse=MSEConfig(norm=2.4, grid_budget="256M"),
    dynamic={
        r"-:^model\.layers\.0\.self_attn\.q_proj$": {},
        r"+:^model\.layers\.1\.mlp\.down_proj$": {"bits": 8},
        r"^model\.layers\.[0-9]+\.mlp\.": {"mse": {"norm": 2.0}},
    },
    offload_to_disk=False,
)

Rules use full module paths, not just leaf names. PCRE .match() is start-anchored; use $ to require a full match and escape literal dots. +: marks a positive rule; no prefix is also positive. -: returns False (skip that module). Use {} as the payload of a negative rule: ordinary config normalization iterates payload dictionaries, so a raw boolean payload is not generally safe even though some internal helpers recognize False. All keys starting - are moved first during post-init; ordering among each group remains stable. The first matching rule wins, and dictionaries from different matching rules are not merged.

dynamic_get(layer_name, key=None, default=None, sub_key=None) returns a copy of the override dictionary (or selected value), False for an exclusion, or the supplied default for no match/key. The module-level dynamic_matching_pattern() identifies the deciding rule. Exact anchored literal patterns use a fast lookup; mixed regex rules preserve the same ordering. Mutable tracked dictionaries/lists invalidate content-keyed caches after in-place edits. Cache helpers are implementation details, not serialization fields.

Validation/consumption differs by method:

MLX conversion has an additional consumer-specific restriction: every nonempty dynamic dictionary must explicitly include bits; a group-only rule raises KeyError in the conversion wrapper. This is not a general restriction on inherited GPTQ overrides. See MLX Apple Silicon.

Method Recognized dynamic behavior / restrictions
GPTQ bits/group_size normalized; processor cloning additionally applies sym, desc_act, damping, static_groups, group awareness, MSE, Hessian, GPTAQ/FOEM and related controls where supported. MSE dictionaries merge field-wise with the base, numbers replace norm and preserve budgets. Arbitrary keys are not a universal override API.
RTN Clone can change bits, sym, group_size, desc_act and smoother/smooth.
GGUF bits and/or format normalize together; supply a compatible subtype when changing alias/width. Runtime bits are recomputed.
FP8 bits must stay 8; group_size must be -1/None. format/fmt, scale method, block size and inverse semantics normalize per module.
BitsAndBytes bits in 4/8; group_size -1/None; no desc_act; sym must be True. Per-layer format/block_size/compress_statistics and legacy aliases are recognized. Changing bits may also require a matching format override.
EXL3 Fractional bits/head_bits normalize; group_size must remain -1/None.
QQQ Inherited GPTQ checks plus rejection of desc_act=True in any dictionary.
VPTQ No nonempty per-layer override dictionaries.

The staged gsq_training path interprets dynamic differently from ordinary processing: it accepts only a per-block width assignment — every entry must name a whole decoder block (model.layers.<n>, literal name rather than a general pattern) and set only bits (W2/W3/W4) — and publishes the same assignment back into the exported config. Any other module or field raises. See GSQ — non-uniform bit allocation.

Usage examples

import torch
from gptqmodel.quantization import (
    QuantizeConfig, GPTQConfig, HessianConfig, MSEConfig,
    GGUFConfig, FP8Config, BitsAndBytesConfig, EXL3Config,
    SmootherConfig, SmoothMAD, TelemetryConfig,
)

qcfg = GPTQConfig(
    bits=4, group_size=128,
    hessian=HessianConfig(chunk_size=4096, staging_dtype=torch.float16),
    mse=MSEConfig(norm=2.4, grid_budget="256M", clip_grid_budget="128M"),
    telemetry=TelemetryConfig(device=True),
)
assert isinstance(qcfg, QuantizeConfig)

gguf = GGUFConfig(bits=4, format="q_k_m")
fp8 = FP8Config(weight_scale_method="block", weight_block_size=[128, 128])
bnb = BitsAndBytesConfig(bits=4, format="nf4", block_size=64)
exl3 = EXL3Config(bits=3.5, head_bits=6.0)
rtn = QuantizeConfig(weight_only="rtn", bits=4,
                     smoother=SmootherConfig(smooth=SmoothMAD(k=2.75)))

These construct configs; install the appropriate runtime/backend and choose a supported model/device before quantizing. Full model loading/calibration/save examples are in Quantization Lifecycle.

Serialization and checkpoint loading

The canonical filename is quantize_config.json, not quant_config.json. Compatibility search order is quantize_config.json → quant_config.json → config.json; the last reads the nested quantization_config object. from_pretrained is the local-directory config reader; model loading resolves remote artifacts separately.

Source signatures:

def save_pretrained(self, save_dir: str, **kwargs):
    ...

@classmethod
def from_quant_config(cls, quantize_cfg, format: str = None):
    ...

@classmethod
def from_pretrained(cls, save_dir: str, **kwargs):
    ...

def to_dict(self):
    ...
import copy
import json
from pathlib import Path
from gptqmodel.quantization import QuantizeConfig

Path("./config-only").mkdir(exist_ok=True)
qcfg.save_pretrained("./config-only")
restored = QuantizeConfig.from_pretrained("./config-only")
payload = copy.deepcopy(qcfg).to_dict()
print(json.dumps(payload, indent=2))

save_pretrained() on the config writes JSON only, not model weights, and does not create the directory itself. The model writer writes both the standalone config and config.json.quantization_config, plus provenance and weights.

Item Serialized behavior
Method/format Both method + quant_method and format + checkpoint_format are emitted. Canonical code fields are method and format.
Common top-level bits, dynamic, group_size, desc_act, lm_head, activation, pack_dtype name, rotation; None/empty dictionaries are omitted. sym is added by GPTQ/RTN/QQQ, while AWQ/ParoQuant use zero_point.
Metadata Starts from a shallow copy of meta; adds fallback, offload, telemetry, pack_impl, GC/finalization/forward-parallel and placement strategy settings; MoE when enabled. Hooks add method-specific data.
GPTQ meta contains scalar mse.norm, mock_quantization, act_group_aware, hessian and selected GPTAQ/FOEM branch. The model writer additionally records damping, static_groups, true_sequential, quantizer version and URI. gsq and gsq_training are emitted at top level when set.
AWQ Adds zero_point, version, format and meta.scale_search_chunked_activations; gsq and gsq_training are emitted at top level when set.
RTN Adds sym and meta.weight_only with smoothing descriptor.
ParoQuant Adds zero_point and krot at top level; all opt_* fields go under meta.
VPTQ Adds codebook, vector, grouping, k-means, norm, permutation, block and damping settings at top level.
FP8 Adds subtype and scale method/block size/semantics, gsq_calibration, and gsq when set.
BitsAndBytes Adds subtype, block_size and compress_statistics.
EXL3 Adds fractional bits/head_bits, out_scales, codebook, tensor_storage and calibration when present.
GGUF Omits group_size, desc_act, pack_dtype and GPTQ-specific offload/fallback/packing/placement metadata; retains subtype and bits.
Dynamic adapter entries Removed from dynamic during to_dict(); serialized separately as rank patterns by adapter saving.

from_quant_config() lowercases JSON keys, recognizes field synonyms, ignores unknown parameters with a log message, restores selected meta controls when not explicitly top-level, selects a concrete class and normalizes it. Manual format= conflicts with any serialized format/checkpoint_format rather than silently overriding it. backend="llm-awq" forces AWQ/LLM_AWQ/int16 detection. BITBLAS and BitsAndBytes force desc_act=False in this reader. Missing sym defaults to True with a warning except for selected fixed-format families.

Keep canonical and compatibility keys consistent. In constructor kwargs, method takes precedence over quant_method, and format over checkpoint_format. During JSON import, method and its quant_method synonym are processed in mapping order, so the later one can overwrite the earlier one. checkpoint_format is held as a fallback when no explicit format field was processed. Do not rely on these differences to repair contradictory checkpoint metadata.

Round-trip caveats

This is a checkpoint/inference descriptor, not a lossless dump of every constructor field:

  • MSE workspace budgets are not emitted by the GPTQ metadata hook: meta.mse remains the legacy scalar exponent. Reapply budgets in a new calibration job.
  • to_dict() serializes a deep copy of dynamic, so adapter keys and serialized bits are stripped from the payload without mutating the live config.
  • GPTQConfig's metadata override does not call the preprocessing parent hook, so its preprocessors are not automatically emitted. The same applies to GPTQ-derived QQQ/VPTQ; ParoConfig also overrides without calling that parent hook.
  • The standalone GPTQ metadata branch writes gptaq=None when gptaq is absent; FOEM-only provenance is supplied by the model writer. When both are present, the branch writes FOEM. Do not use config-only JSON as a complete calibration audit trail.
  • static_groups is recorded by the model writer under meta but is not included in the reader's meta-restoration map. true_sequential and damping are restored from meta. User meta keys can be overwritten by generated runtime/provenance keys.
  • device, compute_device_filter, calibration_data_device and adapter are not ordinary top-level fields in the base to_dict payload. An adapter manifest is managed by its dedicated save path.
  • EXL3 uses a separate normalization path: its MSE dictionary is not converted into MSEConfig, and its telemetry dictionary is not converted into TelemetryConfig. The writer reads MSE with guarded getattr, so an unnormalized dictionary loses the exponent rather than raising there; generic to_dict() still requires telemetry to be a dataclass for asdict(). Prefer typed values.

Implementation internals

QuantizeConfigMeta owns factory compatibility and dispatch. _resolve_quantize_config_class, _prepare_target_quantize_config_kwargs, and _filter_quantize_config_payload_for_target_cls prepare concrete objects. resolve_quant_format separates public subtypes from family identifiers. clone_weight_only_config_for_module deep-copies RTN/GGUF/FP8/BitsAndBytes configs and applies that module's rule; GPTQ processor cloning handles calibration-specific overrides. calculate_bits_per_weight() logs an estimate rather than returning measured model size; GGUF/EXL3 override its estimate.

Pitfalls and edge cases

  • A bare factory is not equivalent to bare BaseQuantizeConfig: the former produces GPTQConfig with different activation-ordering defaults.
  • Constructor compatibility aliases are narrower than JSON compatibility aliases. Prefer canonical kwargs on concrete dataclasses.
  • New quantization cannot target FORMAT.MARLIN despite its presence in supported-format lists for legacy loading.
  • GPTQ_P requires planar-aware kernels; changing a JSON format label is not tensor conversion.
  • Enums/subtypes indicate parser capabilities, not an exhaustive promise of GPU/ROCm/MLX hardware support. Read the backend and method pages before selecting non-default packing.
  • Typed objects avoid surprising dictionary defaults (Fallback threshold, SmoothMAD k) and factory filtering of irrelevant fields.

Source files

  • gptqmodel/quantization/config.py — all dataclasses, enums, normalization, dynamic rules and JSON serialization.
  • gptqmodel/quantization/gsq_training_config.py — GSQTrainingConfig fields and validation (see GSQ).
  • gptqmodel/quantization/__init__.py — public configuration exports.
  • gptqmodel/adapter/adapter.py — Adapter/Lora, descriptor normalization and runtime application.
  • gptqmodel/models/base.py — quantize/requantize, embedding argument normalization and MARLIN export rejection.
  • gptqmodel/models/loader.py — device/rotation/family loading.
  • gptqmodel/models/writer.py — checkpoint/provenance and adapter serialization.
  • gptqmodel/looper/gptq_processor.py — module-specific calibration configuration.
  • gptqmodel/looper/weight_only_looper.py, gptqmodel/looper/stage_layer.py — embedding and weight-only lifecycle.
  • gptqmodel/utils/packing.py, gptqmodel/utils/importer.py — pack implementation and kernel feasibility.

See also

Documentation home · Quickstart · Troubleshooting · Repository · PyPI

Source baseline: GPT-QModel 7.7.0 / c197691. Maintained by ModelCloud.ai.

Home

Getting started

Quantization methods

Complementary configs

Using a model

Platforms

Internals

Clone this wiki locally