Skip to content

Packing Formats and Compatibility

Jump to bottom
Qubitium edited this page Oct 29, 2026 · 1 revision

Packing Formats and Compatibility

Documentation home · Quickstart · Lifecycle

On this page

Overview

A quantized checkpoint is a tensor layout plus a configuration contract, not simply a collection of integer weights. Its method, format, bit width, group size, zero-point convention, packing word dtype, optional permutations/codebooks and model definition must agree. Changing a JSON label does not convert tensors.

Here, checkpoint means a saved model artifact for loading or distribution. model.save() does not create the interrupted-quantization resume state managed by CheckpointConfig; see Quantization Lifecycle for that separate contract.

This page describes revision c1976918cfe3703fbcf6d1ce2440a82b26394375. The backend declarations and loading code in this revision are authoritative; an installed Transformers/Optimum/AutoGPTQ version may implement a smaller set. See QuantizeConfig Reference for every constructor field and normalization rule.

Workflow at a glance

Portable artifacts and runtime layouts. A format label, tensor layout and executable backend must agree.

Reading the diagram: Loading and saving are independent concerns. Runtime repacking does not relabel a checkpoint or universally convert method families.

Text version of the workflow
  • Load for inference: Read metadata → Validate a backend → Load packed tensors → Initialize and run.
  • Save for distribution: Checkpoint-facing model → Write tensors and config → Compatible reader.

Open full-size SVG

Concepts: three independent contracts

Contract Example Meaning
Quantization algorithm METHOD.GPTQ, METHOD.AWQ, METHOD.PARO How codes, scales, and auxiliary state are obtained.
Checkpoint layout FORMAT.GPTQ, FORMAT.GPTQ_P, FORMAT.GEMM How those codes and auxiliary tensors are serialized.
Inference backend BACKEND.GPTQ_MARLIN, BACKEND.MLX How a compatible checkpoint is executed, sometimes after a runtime repack.

Marlin inference can consume ordinary GPTQ weights; it does not require producing format="marlin". A backend's private, post-initialization buffer ABI is not automatically a portable checkpoint format. Conversely, equal tensor names do not imply equal layouts: GPTQ and AWQ GEMM both use qweight/qzeros/scales but pack different axes and code orders.

The on-disk container is usually SafeTensors, not the format enum itself. GGUF is especially important: GPT-QModel's GGUF-family SafeTensors checkpoint contains GGUF block bytes inside qweight; a standalone .gguf file is a different container with GGUF headers, tensor names and metadata.

Every FORMAT value

The following names and values are the complete enum in gptqmodel/quantization/config.py.

Member / serialized family Producing method / lifecycle Storage contract Consumers and portability
GPTQ / gptq GPTQ; RTN can export it Continuous GPTQ code stream; legacy v1 qzeros convention Broadest GPTQ interchange target. Many GPTQ kernels; selected external GPTQ loaders when contract matches.
GPTQ_V2 / gptq_v2 GPTQ; RTN Same continuous weight packing; corrected true-zero qzeros GPT-QModel >=0.9.0; do not assume AutoGPTQ accepts it.
GPTQ_P / gptq_p GPTQ; RTN; automatic for GPTQ 5/6/7-bit Word-aligned split bit planes; true-zero semantics Planar-aware Torch/MLX consumers in this revision; not legacy GPTQ interchange.
GGUF / gguf GGUF weight-only config GGUF subtype-specific block bytes in uint8 qweight, or native GGUF input GGUF Torch/C++/CUDA/Triton/MLX paths with different subtype coverage; native GGUF architecture profiles are separately gated.
FP8 / fp8 FP8 weight-only config float8 weight + FP32 weight_scale_inv FP8 Torch/MLX; subtype and inverse-scale semantics must match.
BITSANDBYTES / bitsandbytes BitsAndBytes weight-only config Packed 4-bit codebook + quant state, or int8 row-scaled weight BitsAndBytes runtime and MLX reader; not automatically the standard HF BitsAndBytes saved schema.
MARLIN / marlin Historical GPTQ/AWQ layout token Family-sensitive Marlin loading contract; runtime tile repack Matching GPTQ/AWQ Marlin backend, or MLX. New quantization to this token is rejected.
BITBLAS / bitblas Config accepts GPTQ/AWQ export; runtime conversion utilities BitBLAS-transformed/compressed weights, output-major scales, compressed optional zeros Matching BitBLAS backend or MLX path. Preserve original method and actual buffer layout.
QQQ / qqq QQQ QQQ B tile weights, channel/group scales QQQ native, QQQ Torch, MLX; not ordinary GPTQ.
EXL3 / exl3 EXL3 Trellis/codebook data and Hadamard/scale side tensors Special EXL3 loader/backend path; MLX EXL3 support is outside the normal qlinear registry.
VPTQ / vptq VPTQ Centroids, optional residual/outlier codebooks, packed indices and optional normalization/permutation VPTQ Torch and CUDA; not scalar GPTQ codes.
GEMM / gemm AWQ; RTN can target AWQ packing Output-axis-packed AWQ qweight/qzeros, group-major scales Native/portable AWQ GEMM consumers and runtime repackers; common AutoAWQ-style layout.
GEMV / gemv AWQ; RTN Input-axis-packed, output-major AWQ weights, padded scale/zero groups AWQ GEMV and MLX; not interchangeable with GEMM by transpose alone.
GEMV_FAST / gemv_fast AWQ; RTN Interleaved int16 packed weights; FP16 scale/zero grids AWQ GEMV_FAST and MLX; int16 packing required.
LLM_AWQ / llm-awq AWQ; RTN LLM-AWQ GEMV_FAST-family packing, scaled zeros naming convention Dedicated LLM-AWQ and MLX paths; backend tag is detected during config reading.
PAROQUANT / paroquant METHOD.PARO AWQ GEMM-like codes plus learned rotations/pairs/channel scales ParoQuant native/Triton/MLX; dropping auxiliary state changes the model.

For GGUF, FP8 and BitsAndBytes, the config's public format field is a subtype, e.g. q_k_m, float8_e4m3fn, or nf4. resolve_quant_format(format, method) returns the family enum used for backend selection. FORMAT("nf4") is not a valid substitute for that resolver.

Tensor layouts in detail

Notation: K is in_features, N is out_features, G is the number of quantization groups, b is bits, and W is the integer packing-word width. Shapes below describe aligned/default contracts; kernel padding or a specialized format can impose stricter constraints.

GPTQ and GPTQ_V2

Tensor Typical int32 aligned shape Role
qweight [K*b/32, N] Codes packed down input-channel rows; continuous b-bit stream.
qzeros [G, N*b/32] Group zero points packed along output channels.
scales [G, N] Floating-point dequantization scales.
g_idx [K], int32 Input-channel-to-group mapping; important for activation ordering.
bias [N], optional Dense bias.

Classic continuous widths are 2,3,4,8. Three-bit codes cross word boundaries; decoding must stitch split fields. It is not equivalent to using 32 // 3 complete codes per word. Config accepts other integer storage word dtypes but kernel support and special INT3 handling are narrower; int32 is the portable default here.

Logical reconstruction is scale[group, output] * (code - zero[group, output]). gptq (v1) stores the historical zero-point-minus-one representation; readers apply its correction before using kernels that require v2. GPTQ_V2 stores actual zero points. REQUIRES_FORMAT_V2 determines whether load-time conversion is necessary for a selected kernel.

MIN_VERSION_WITH_V2 = "0.9.0". Saving GPTQ_V2 emits a warning naming this minimum GPT-QModel version. The v1↔v2 conversion is a zero-point convention change, not calibration/requantization, but asymmetric extrema can underflow/overflow under the historical v1 encoding. Do not promise lossless symmetric/asymmetric interchange indiscriminately.

GPTQ_P: planar GPTQ

gptq_p retains qweight/qzeros/scales/g_idx names but splits codes into planes whose widths divide 32. Every block of 32 logical codes uses b adjacent int32 words, low planes first. Inside each plane, fields are packed least-significant-first. Weight packing follows rows and zero packing follows columns.

b Planes Reconstruction
2 2 single plane
3 2 + 1 lo OR (hi << 2)
4 4 single plane
5 4 + 1 lo OR (hi << 4)
6 4 + 2 lo OR (hi << 4)
7 4 + 2 + 1 lo OR (mid << 4) OR (hi << 6)
8 8 single plane

Planar zeros always use v2 semantics. They never undergo a v1 +1 correction. Packed dimensions must be divisible by 32 and words must be int32; misaligned embeddings/vocabularies are a known limitation. At 2/4/8 bits the plane words are identical to the continuous representation, but this is not permission to relabel v1 zeros. At 3 bits the packing changes. Five/six/seven-bit configs automatically route GPTQ/GPTQ_V2 to planar, even when only a dynamic layer needs those widths.

Use a declared GPTQ_P consumer, rather than assuming a CUDA backend that accepts GPTQ_V2 also understands planes. The source design note describes GPU-native planar kernels as a later phase; current portable execution must be selected according to the actual registry.

AWQ GEMM

Tensor Default 4-bit/int32 contract Role
qweight [K, N/8] Codes packed along outputs, including the AWQ code-order permutation.
qzeros [G, N/8] True AWQ zero points with matching output-axis packing.
scales [G, N] Group-major FP16 scales (runtime may normalize dtype).
bias [N], optional Dense bias.

AWQ uses zero_point = not sym in config serialization; version is the historical AWQ layout alias. The portable AwqTorchLinear advertises broader bit widths than the native four-bit GEMM extension. That does not make every wider-bit AWQ checkpoint readable by external AutoAWQ kernels.

AWQ GEMV

GEMV stores qweight as [N, K/(W/b)]; codes run along inputs, not outputs. qzeros is [N, Z] in the packing dtype and scales is [N, Z*(W/b)] in FP16, where Z = calculate_zeros_width(K, group_size) includes padding. Its packer pads group slots before packing. Standard runtime declares four-bit, int32 storage. This is a distinct tensor contract, not merely a selection of the GEMM kernel for small batches.

AWQ GEMV_FAST and LLM_AWQ

GEMV_FAST's pack_intweight(unpacked_qweight, interleave, kstride) permutes 32-code input blocks, reorders eight-code groups as 0,2,4,6,1,3,5,7, interleaves four output rows, and packs four nibbles into int16 words. With interleave=4, qweight is [N/4, K] for four-bit weights. The scale and zero grids are FP16 and have padded group extent derived from calculate_zeros_width, not packed GPTQ integer zeros.

LLM_AWQ uses this packing family but distinguishes the scaled-zero tensor naming/handling from ordinary GEMV_FAST. backend="llm-awq" in an external AWQ config forces method AWQ, format LLM_AWQ and pack_dtype int16 during import. This backend field is the AWQ packing-producer tag, not GPT-QModel's BACKEND enum. Do not discard it before detecting the layout. See AWQ.

MARLIN

The enum retains historical loading compatibility. The current GPTQ Marlin class allocates checkpoint-facing qweight, g_idx, scales and qzeros parameters, then post_init() repacks/reorders them for Marlin tiles, permutations and workspace. The AWQ counterpart starts from AWQ-facing data and uses its own conversion. Treat serialized family and post-init kernel ABI separately; dumping post-init buffers under an unchanged GPTQ/AWQ label is not portable checkpoint creation.

The loader forces FORMAT.MARLIN to matching GPTQ_MARLIN/AWQ_MARLIN when AUTO is used; other explicit CUDA backends are rejected. MLX is a separate allowed transcode path. New quantize() calls reject MARLIN format entirely and instruct callers to use GPTQ layout with Marlin inference.

NVIDIA requirements differ by method in this revision: GPTQ Marlin supports compute capability >=7.5, with FP16 only on 7.5; AWQ Marlin requires >=8.0. Both CUDA paths reject ROCm. Other shape/group/alignment limits still apply, including restrictions for activation ordering and padded tiles.

BITBLAS

BitBLAS does not preserve standard GPTQ qweight dimensions internally. Its weight transform determines retrieve_weight_shape() and a compressed storage dtype (usually int8). Scales are output-major [N,G], in compute dtype. If needed, qzeros is [G, N*b/storage_bits] in that compressed dtype, otherwise an empty buffer. A zeros attribute aliases qzeros; it is not an extra independent tensor.

prepare_model_for_bitblas_load() distinguishes an already-BitBLAS serialized checkpoint from standard GPTQ/AWQ tensors requiring a repack. convert_to_bitblas(..., repack=True) calls the appropriate family converter; repack=False creates the native layout for direct loading. repack_from_gptq and repack_from_awq handle their different code axes and zero conventions. Do not take a GPTQ qzeros int32 tensor and reinterpret it as BitBLAS compressed zeros.

CUDA BitBLAS depends on a compatible BitBLAS/TVM installation and a supported NVIDIA device; the registry is CUDA-only for these kernels. MLX's separate reader does not import the CUDA runtime. The config helper's unambiguous BITBLAS export-family map is GPTQ; callers manipulating AWQ/BitBLAS artifacts must preserve and verify the method field and actual tensor contract instead of inferring family solely from the token.

For a serialized FORMAT.BITBLAS checkpoint, the local loader resolves AUTO to the matching GPTQ_BITBLAS or AWQ_BITBLAS backend and rejects mismatched explicit local backends. MLX is a separate allowed conversion route, not a generic exception to layout checks. See format-specific loader branches.

QQQ

Tensor Shape / dtype Role
B [K/16, N*16/8], int32 Permuted/tiled packed four-bit weights.
s_channel [1,N], float32 Per-output-channel scale.
s_group [K/group_size,N], float16, or empty for a single whole-row group Optional groupwise scale relative to channel scale.
bias [N], optional float16 Dense bias.

Workspace/reduction buffers are nonpersistent. There is no GPTQ g_idx: desc_act=True is rejected. QQQConfig.to_dict() adds wbits alongside bits for vLLM's QQQ loader. Native QQQ declares CUDA/ROCm devices; QQQ Torch is portable, and MLX has a dedicated reader. QQQ's four-bit weights and execution details are described in QQQ; B is not a GPTQ qweight tensor.

EXL3

The EXL3 module wrapper persists a subset of trellis, suh, svh, su, sv, bias, mcg, mul1, depending on the quantized layer. The configuration records tensor_storage[module_name].stored_tensors, including each qualified tensor name, shape and torch_dtype, plus quant_format, bits_per_weight and optional codebook multiplier values. The writer rebuilds this metadata from actual modules before saving.

Targets may be fractional (1.0..8.0 BPW); they are not a simple integer qweight word width. Trellis buffers and transformation/scale side data must be preserved together. EXL3 bypasses the ordinary BaseQuantLinear support-map path; its special Torch/CUDA/MLX adapters and vendored implementation determine feasibility. Never attempt a GPTQ qzeros converter on EXL3. See EXL3.

VPTQ

Persistent item Contract
centroids.weight Main codebook embedding: [group_num, num_centroids*vector_len].
res_centroids.weight Optional residual codebook of the same vector orientation.
outlier_centroids.weight Optional outlier codebook.
indices int32 [group_num, num_indices, packed_words], packing main/residual codebook indices.
outlier_indices Optional int32 outlier index stream.
weight_scale / weight_bias Optional normalization state, sized by norm_dim.
perm Optional persistent input-channel permutation; inverse_perm is reconstructed/nonpersistent.
bias Optional dense layer bias.

Bits is a derived nominal index budget per weight, not the full checkpoint size or a GPTQ scalar code width. Default 4096 main +16 residual centroids and vector length 8 produce nominal 2 BPW. Codebook storage adds overhead. Main/residual indices total at most 32 bits. Config does not support nonempty per-layer override dictionaries. Portable Torch and CUDA VPTQ consumers share this contract; hardware/dependency checks are separate. See VPTQ.

FP8

weight is [N,K] in the selected Torch float8 dtype. weight_scale_inv is FP32: scalar for tensor scaling, [N] for row scaling, or [N/block_rows,K/block_cols] for block scaling. Optional bias remains dense. Block dimensions must divide the matrix shape in the quantizer.

The name is important: inverse scale is the multiplier used when quantizing (q = cast_fp8(weight * scale_inv)); reconstruction divides q by scale_inv. It is not an external checkpoint's arbitrary scale tensor with a renamed key. Only weight_scale_semantics="inverse" is accepted. Family is FP8 but format is a subtype such as float8_e4m3fn.

Config recognizes E4M3FN, E5M2, FNUZ variants and E8M0FNU when the installed Torch exposes the dtype. The Torch quantizer explicitly rejects producing dense E8M0 weights; that subtype is for dequantization of existing checkpoints. Storing an FP8 tensor on a GPU does not imply native FP8 matmul/cast support; the Torch implementation has fallback handling, and NPU is not declared for this backend. W4A FP8/NVFP4 activation streams are not FP8 weight checkpoints: their weights remain GPTQ INT4.

BitsAndBytes

Precision Persistent layout
4-bit FP4/NF4 Packed uint8 weight plus mapped QuantState tensors: weight_absmax, weight_quant_map, weight_quant_state, and nested statistics fields when enabled. The packed state identifies fp4/nf4 and shape/block information.
8-bit int8 weight [N,K] plus float32 weight_scb [N]; optional dense bias.

_packed_state_key_to_buffer_name() maps bitsandbytes' dotted QuantState names into module buffer names; _quant_state_payload() reverses the representation for runtime reconstruction. This mapping is why an arbitrary HF bitsandbytes artifact is not automatically byte-for-byte interchangeable. Four-bit block size and double-statistics compression must agree with the serialized state. The runtime validator requires bitsandbytes >=0.49.0, whereas the bitsandbytes package extra requests >=0.50.2 in this revision. Native support is declared CPU/CUDA, with a separate MLX packed reader. See BitsAndBytes.

GGUF-family SafeTensors and native GGUF

GPT-QModel's GGUF linear stores subtype-specific block bytes as uint8 qweight [N, bytes_per_row], padding K to the encoding's block size. Each block embeds its own scales, signs/high bits/codebook information according to the GGUF subtype; this is not a separate GPTQ qzeros/scales representation. Bits and public format reconstruct the combined GGUFBits alias. Optional bias is dense FP16.

Native .gguf input has GGUF architecture/tensor metadata instead. The loader detects a direct file or a directory/repository containing exactly one GGUF and no dense weight files, then dispatches an architecture/subtype-appropriate path. The utility normalize_model_id_or_path_for_hf_gguf() forwards an internal gguf_file kwarg to Transformers; GPT-QModel's public APIs reject caller-supplied gguf_file and ask for the artifact as model_id_or_path. Multiple GGUF variants in one directory are not auto-selected.

The internal runtime can install Prism Q1_0_g128 compatibility for Transformers builds lacking it. The direct-to-Torch internal GGUF loader is a separately gated optimization, not universal architecture coverage. Loading GGUF through a dense HF conversion path may materialize floating-point weights; that is not the same as native packed GGUF inference. See GGUF.

PAROQUANT

ParoQuant builds on AWQ GEMM-like scalar packing but additionally saves theta, pairs, and channel_scales for its learned rotations and per-channel scaling. krot defines rotation stages. Its native/Triton/MLX consumers apply this auxiliary state; treating qweight/qzeros/scales alone as a normal AWQ artifact drops part of the computation. The format is fixed to paroquant, and the algorithm enum member is METHOD.PARO. See ParoQuant.

Backend compatibility

Support is discovered from each kernel's SUPPORTS_METHODS, SUPPORTS_FORMATS and SUPPORTS_BACKENDS, then validated for bits, group size, dtype, shape, device, dependencies, sharding and dynamic overrides. AUTO uses declared priority, not a fresh benchmark. EXL3 uses a separate path. The following source-declaration inventory is included to make family/layout differences explicit; an entry is not a promise that validation succeeds for every member of the family.

In the table, backend/method/format names are enum member names. ALL means DEVICE.ALL, not a promise about optional native extensions. Files are relative to gptqmodel/nn_modules/qlinear/. These are layout declarations; priorities and non-layout feasibility constraints are intentionally not expanded here. TorchQuantEmbeddings uses the Torch GPTQ formats but is explicitly excluded from general backend selection.

Backend(s) Method Accepted format(s) Bits Declared device(s) Source file
GPTQ_TORCH GPTQ GPTQ, GPTQ_V2, GPTQ_P 2–8 ALL torch.py
GPTQ_TORCH_ATEN GPTQ GPTQ, GPTQ_V2 4 CPU torch_aten_kernel.py
GPTQ_TORCH_CPU_AVX512 GPTQ GPTQ, GPTQ_V2 4 CPU torch_cpu_avx512.py
GPTQ_TORCH_FUSED GPTQ GPTQ, GPTQ_V2 4 CPU, XPU torch_fused.py
GPTQ_TORCH_INT8 GPTQ GPTQ, GPTQ_V2 8 CPU torch_int8.py
GPTQ_TRITON GPTQ GPTQ, GPTQ_V2 2,4,8 CUDA tritonv2.py
GPTQ_EXLLAMA_V2 GPTQ GPTQ, GPTQ_V2 4 CUDA exllamav2.py
GPTQ_MARLIN GPTQ GPTQ, GPTQ_V2, MARLIN 4,8 CUDA marlin.py
GPTQ_MACHETE GPTQ GPTQ 4,8 CUDA machete.py
GPTQ_SWORDFISH GPTQ GPTQ 4,8 CUDA swordfish.py
GPTQ_BITBLAS GPTQ BITBLAS, GPTQ, GPTQ_V2 2,4,8 CUDA bitblas.py (BITBLAS_SUPPORTED_BITS)
GPTQ_W4AFP8 GPTQ GPTQ, GPTQ_V2 4 CUDA w4a_floatx.py
GPTQ_W4A_NVFP4 GPTQ GPTQ, GPTQ_V2 4 CUDA w4a_nvfp4.py
AWQ_TORCH AWQ GEMM 2–8 ALL torch_awq.py
AWQ_TORCH_ATEN AWQ GEMM 4 CPU torch_aten_kernel_awq.py
AWQ_TORCH_CPU_AVX512 AWQ GEMM 4 CPU torch_cpu_avx512_awq.py
AWQ_TORCH_FUSED AWQ GEMM 4 CPU, XPU torch_fused_awq.py
AWQ_TORCH_INT8 AWQ GEMM 4 CPU torch_int8_awq.py
AWQ_GEMM AWQ GEMM 4 CUDA gemm_awq.py
AWQ_GEMM_TRITON AWQ GEMM 4 CUDA, ROCM gemm_awq_triton.py
AWQ_GEMV AWQ GEMV 4 CUDA, ROCM gemv_awq.py
AWQ_GEMV_FAST AWQ GEMV_FAST, LLM_AWQ (separate LLMAwqLinear) 4 CUDA, ROCM gemv_fast_awq.py
AWQ_EXLLAMA_V2 AWQ GEMM 4 CUDA exllamav2_awq.py
AWQ_MARLIN AWQ GEMM, MARLIN 4,8 CUDA marlin_awq.py
AWQ_MACHETE AWQ GEMM, MARLIN 4,8 CUDA machete_awq.py
AWQ_SWORDFISH AWQ GEMM 4 CUDA swordfish.py
AWQ_BITBLAS AWQ GEMM, BITBLAS 4 CUDA bitblas_awq.py
QQQ QQQ QQQ 4 CUDA, ROCM qqq.py
QQQ_TORCH QQQ QQQ 4 ALL qqq.py
VPTQ_TORCH VPTQ VPTQ 1–8 ALL vptq_torch.py
VPTQ_CUDA VPTQ VPTQ 1–8 (inherited) CUDA vptq_cuda.py
PAROQUANT_CUDA PARO PAROQUANT 4 ALL paroquant.py
PAROQUANT_TRITON PARO PAROQUANT 4 CUDA paroquant_triton.py
FP8_TORCH FP8 FP8 8 CPU, CUDA, ROCM, XPU, MPS fp8.py
BITSANDBYTES BITSANDBYTES BITSANDBYTES 4,8 CPU, CUDA bitsandbytes.py
GGUF_TORCH GGUF GGUF 1,2,3,4,5,6,8 ALL gguf.py
GGUF_CPP_CPU GGUF GGUF 1,2,3,4,5,6,8 CPU gguf_cpp.py
GGUF_CPP_CUDA GGUF GGUF 1,2,3,4,5,6,8 CUDA gguf_cpp.py
GGUF_TRITON GGUF GGUF 1,4,5,6 CUDA gguf_triton.py
MLX GPTQ GPTQ, GPTQ_V2, GPTQ_P 2–8 MPS mlx.py / MlxQuantLinear
MLX GPTQ MARLIN 4,8 MPS mlx.py / GPTQMarlinMlxQuantLinear
MLX GPTQ BITBLAS 2,4,8 MPS mlx.py / GPTQBitBLASMlxQuantLinear
MLX AWQ GEMM 2–8 MPS mlx.py / AwqMlxQuantLinear
MLX AWQ MARLIN 4,8 MPS mlx.py / AWQMarlinMlxQuantLinear
MLX AWQ BITBLAS 2–8 MPS mlx.py / AWQBitBLASMlxQuantLinear
MLX AWQ GEMV, GEMV_FAST, LLM_AWQ (distinct classes) 4 MPS mlx.py
MLX PARO PAROQUANT 4 MPS mlx.py / ParoMlxQuantLinear
MLX QQQ QQQ 4 MPS mlx.py / QQQMlxQuantLinear
MLX GGUF GGUF 1,2,4,5,6,8 MPS mlx.py / GGUFMlxQuantLinear
MLX FP8 FP8 8 MPS mlx.py / FP8MlxQuantLinear
MLX BITSANDBYTES BITSANDBYTES 4,8 MPS mlx.py / BitsAndBytesMlxQuantLinear

CUDA in a declaration is not a synonym for ROCm: only kernels explicitly accepting ROCM (or a portable ALL implementation) should be considered. Marlin, Machete and CUDA-only native extensions have separate checks. MPS/MLX requires Apple Silicon and the optional MLX stack; an MPS family declaration alone does not install it. CPU-specialized kernels may require host ISA features. Consult Kernel Backends Reference for full shape/hardware constraints, and probe the exact contract instead of copying another machine's AUTO choice.

Quantization config files and field aliases

Constant Literal Meaning
QUANT_CONFIG_FILENAME quantize_config.json Canonical standalone config written by save_pretrained/save_quantized.
QUANT_CONFIG_FILENAME_COMPAT [quantize_config.json, quant_config.json, config.json] Local-reader priority. config.json uses its nested quantization_config object.
FORMAT_FIELD_CODE format Canonical layout/subtype.
FORMAT_FIELD_CHECKPOINT checkpoint_format Deprecated compatibility alias, still emitted.
METHOD_FIELD_CODE method Canonical algorithm/export-family field.
QUANT_METHOD_FIELD quant_method Deprecated compatibility alias, still emitted.
PACK_DTYPE_FIELD pack_dtype Word dtype name, e.g. int32, not activation dtype.
META_FIELD meta Quantizer provenance and auxiliary calibration/runtime controls.
FORMAT_FIELD_COMPAT_MARLIN is_marlin_format Hard-deprecated; presence raises.
AWQ_PACKING_BACKEND_FIELD backend AWQ producer/packing tag, notably llm-awq; not BACKEND.

Example excerpt (not the entire generated JSON):

{
  "bits": 4,
  "group_size": 128,
  "desc_act": false,
  "sym": true,
  "method": "gptq",
  "quant_method": "gptq",
  "format": "gptq",
  "checkpoint_format": "gptq",
  "pack_dtype": "int32"
}

Other common output includes dynamic, lm_head, activation, rotation and meta when applicable. None/empty mappings are omitted. w_bit/wbits map to bits; q_group_size maps to group_size; AWQ version maps to format and zero_point is negated into sym. BitsAndBytes aliases map bnb_quant_type/bnb_block_size/bnb_compress_statistics into canonical format/block_size/compress_statistics.

Constructor alias handling is narrower than JSON import. QuantizeConfigMeta explicitly handles quant_method/checkpoint_format and rejects is_marlin_format; use canonical kwargs on concrete dataclasses. The JSON reader detects historical quant_method values marlin/bitblas as layout hints, repairs selected method/format combinations, and promotes supported meta controls. Missing sym defaults True with a warning where appropriate. Explicit format= conflicts with a serialized format or checkpoint_format.

Keep duplicated fields consistent. Constructor kwargs give method precedence over quant_method and format precedence over checkpoint_format. JSON import is different: from_quant_config() processes method and its quant_method synonym in mapping iteration order, so the later key can overwrite the earlier one. An explicit format field takes precedence over the legacy checkpoint_format fallback. Other projects may read only an alias. A mismatched file pair can also cause different consumers to select different configs.

Hugging Face, Optimum and AutoGPTQ interoperability

There are three different questions: whether a library can parse metadata, whether it recognizes tensor packing, and whether it has a usable kernel on the target hardware. Passing the first does not establish the others.

Artifact Interoperability guidance
Conventional gptq v1, int32, standard 2/3/4/8-bit contract Intended compatibility target for GPTQ/AutoGPTQ-style readers. Check their actual bit/group/desc_act/kernel restrictions; custom dynamic widths, adapters or model-specific modules can exceed them.
GPTQ_V2 GPT-QModel minimum version is 0.9.0. Do not assume an older AutoGPTQ/Transformers integration recognizes or corrects its zeros. Use the explicit symmetric v2→v1 compatibility helper when applicable.
GPTQ_P Requires explicit planar-aware loading. Neither legacy continuous-code readers nor a generic quant_method=gptq label establish support.
AWQ GEMM/GEMV/GEMV_FAST/LLM_AWQ Match the external AWQ version and producer tag, not only quant_method=awq. Word dtype, scale orientation and scaled-zero names matter.
MARLIN/BITBLAS Private/transformed layouts require the corresponding loader ABI. A library supporting Marlin runtime on GPTQ does not necessarily accept every artifact labeled marlin.
FP8 Match exact dtype, weight/scaling names, granularity and inverse semantics. A Transformers FP8 quantizer with different scale conventions is not interchangeable by metadata aliasing.
BitsAndBytes HF's BitsAndBytesConfig and GPT-QModel's BitsAndBytesConfig are distinct APIs/schema contracts. Verify QuantState tensor names/mapping before exchanging saved files.
GGUF HF dense GGUF import and GPT-QModel packed-native GGUF are separate paths. Architecture/subtype recognition and custom Prism support are version-sensitive.
QQQ/EXL3/VPTQ/ParoQuant Need their specific loaders and auxiliary tensors; no general AutoGPTQ compatibility follows from being “low-bit.”

The repository provides stable hf_-prefixed integration hooks, not a blanket guarantee about every external release. gptqmodel.utils.hf handles Transformers version/API/model-config compatibility (including pre-5/5.x imports, tied weights, dtype keywords and GGUF bridges); it is not a universal quantized format converter. gptqmodel.nn_modules.converter.MODULE_CONVERTER_MAP is currently empty and its comments delegate selected architecture defusion to Defuser>=0.0.10. Do not mistake that module for checkpoint bit repacking.

Usage: probe, pack, initialize, repack

Use stable functions in gptqmodel.utils.packing for integrators. Unprefixed pack_module/select_quant_linear are internal. Representative source signatures:

def hf_check_packing_feasibility(
    bits: int,
    group_size: int = -1,
    desc_act: bool = False,
    sym: bool = True,
    *,
    format: Union[str, FORMAT] = FORMAT.GPTQ,
    quant_method: Union[str, METHOD] = METHOD.GPTQ,
    backend: Optional[Union[str, BACKEND]] = BACKEND.AUTO,
    device=None,
    dtype: Optional[torch.dtype] = None,
    pack_dtype: Optional[torch.dtype] = None,
    dynamic: Optional[Dict[str, Dict[str, Any]]] = None,
    in_features: Optional[int] = None,
    out_features: Optional[int] = None,
    allow_marlin: bool = True,
    is_sharded: bool = False,
    repack: bool = False,
) -> bool:
    ...

def hf_pack_layer(
    module: BaseQuantLinear,
    linear: nn.Module,
    scales: torch.Tensor,
    zeros: Optional[torch.Tensor] = None,
    g_idx: Optional[torch.Tensor] = None,
    *,
    pack_impl: Optional[str] = None,
    device=None,
    block_in: int = 8192,
    workers: int = 1,
    scales_extra: Optional[torch.Tensor] = None,
    post_init: bool = False,
    post_init_kwargs: Optional[Dict[str, Any]] = None,
    checkpoint_format: Optional[Union[str, FORMAT]] = None,
) -> BaseQuantLinear:
    ...

def hf_repack_layer(
    module: BaseQuantLinear,
    backend: Optional[Union[str, BACKEND]] = None,
    *,
    device=None,
    dtype: Optional[torch.dtype] = None,
    post_init: bool = True,
    post_init_kwargs: Optional[Dict[str, Any]] = None,
) -> BaseQuantLinear:
    ...
from gptqmodel.utils.packing import (
    hf_check_packing_feasibility,
    hf_check_best_packing_format,
)

can_pack = hf_check_packing_feasibility(
    bits=4, group_size=128, sym=True, desc_act=False,
    format="gptq_v2", quant_method="gptq", device="cuda:0",
    in_features=4096, out_features=4096,
)
best_format = hf_check_best_packing_format(
    bits=4, group_size=128, sym=True, desc_act=False,
    device="cuda:0", formats=["gptq", "gptq_v2"],
)

The probe returns False for an unsupported contract, but invalid enums/arguments raise ValueError. repack=True probes consumers that convert an existing packed layout rather than raw-weight packers. Marlin/ExllamaV2 lack a raw pack() implementation yet can be valid repack targets. Best-format selection ranks source-declared priorities and breaks ties by format name; it does not time the model.

The best-format selector has this complete source signature:

def hf_check_best_packing_format(
    bits: int,
    group_size: int = -1,
    desc_act: bool = False,
    sym: bool = True,
    *,
    quant_method: Union[str, METHOD] = METHOD.GPTQ,
    device=None,
    dtype: Optional[torch.dtype] = None,
    pack_dtype: Optional[torch.dtype] = None,
    dynamic: Optional[Dict[str, Dict[str, Any]]] = None,
    in_features: Optional[int] = None,
    out_features: Optional[int] = None,
    formats: Optional[Sequence[Union[str, FORMAT]]] = None,
    allow_marlin: bool = True,
) -> FORMAT:
    ...

formats restricts the candidate layouts. Unlike hf_check_packing_feasibility, this selector has no backend, is_sharded, or repack argument; it probes AUTO raw-weight packing and raises ValueError if none of the candidate formats is feasible. A checkpoint that works after runtime repacking is therefore not automatically a raw-weight packing target.

For hf_pack_layer, scales/zeros use quantizer-native [N,G] layout. Symmetric zeros may be omitted and are synthesized as 2**(bits-1); asymmetric packing requires explicit zeros. pack_impl=None uses original; explicit choices are auto, original, block/cpu, and gpu, with fallback to original when unsupported. AWQ and QQQ use their single packer rather than this implementation selector. Even GPU packing leaves storage buffers on CPU: move them to the runtime device afterwards. checkpoint_format= controls the GPTQ zero-point flavor left by packing.

from gptqmodel.utils.packing import hf_post_init, hf_repack_layer

# source is a quantized layer retaining checkpoint-layout buffers.
target = hf_repack_layer(source, backend="marlin", device="cuda:0")

# ExllamaV2 needs a separately supplied scratch-space argument:
from gptqmodel.utils.exllamav2 import ScratchSpace
target_exl2 = hf_repack_layer(source, backend="exllama_v2",
                            device="cuda:0", post_init=False)
hf_post_init(target_exl2,
             scratch_space=ScratchSpace(target_exl2.temp_dq_size(), dev="cuda:0"))

Repacking into an already-selected backend returns the original module. Unsupported target backends raise ValueError; incompatible source formats raise NotImplementedError. Native repack_from_gptq/awq hooks take priority; otherwise tensors are copied into a layout-compatible target and post_init performs runtime conversion. Copy repacking aligns v1/v2 zeros. Kernels whose post_init destroys checkpoint layout (Marlin, Machete, Swordfish, BitBLAS) are valid targets, not valid copy-repack sources. Start from a checkpoint-layout kernel such as Torch.

Conversion and migration paths

From → to Source utility/path Limits and safe use
gptq v1 → gptq_v2 utils.model.convert_gptq_v1_to_v2_format/module; hf_convert_gptq_v1_to_v2_format Correct only modules requiring v2; preserve each module's bits/pack dtype. This is not planar conversion.
gptq_v2 → v1 convert_gptq_v2_to_v1_format/module; hf_convert_gptq_v2_to_v1_format Stable HF helper converts only sym=True and checkpoint_format=gptq_v2, returning (model, True); otherwise unchanged/False. Asymmetric export has historical underflow risk.
Continuous ↔ planar Planar-aware pack/unpack primitives in qlinear packing paths Must unpack logical codes/repack and preserve true-zero flavor. No documented whole-checkpoint CLI merely relabels a three-bit artifact. 5/6/7 must remain planar.
GPTQ/AWQ → Marlin/ExllamaV2 and compatible consumers hf_repack_layer plus post_init Runtime kernel conversion, not generic cross-family quantization. Keep original checkpoint-layout source.
GPTQ/AWQ → BitBLAS prepare_model_for_bitblas_load, convert_to_bitblas, native repack hooks Method-sensitive axes/scales/zero conventions; direct BitBLAS loading differs from repack loading.
GPTQ Torch model → dense HF GPTQModel.export(..., format="hf") Calls torch.dequantize_model, replaces TorchLinear modules with CPU FP16 nn.Linear, and removes quantization_config. Other BaseQuantLinear subclasses are explicitly rejected. This is not a universal family dequantizer.
Supported quantized source → MLX artifact GPTQModel.export(..., format="mlx") and convert_gptq_to_mlx_weights Separate artifact conversion, not the same as BACKEND.MLX runtime loading. Requires mlx_lm/MLX optional dependencies and supported converter coverage.
GPTQ → AWQ/QQQ/EXL3/VPTQ/ParoQuant/FP8/BitsAndBytes/GGUF No universal lossless family converter in the listed APIs Usually start from the original dense source and quantize using the target config. Dequantize/requantize compounds error and discards algorithm-specific state.

Source export signature:

def export(model_id_or_path: str, target_path: str, format: str, trust_remote_code: bool = False):
    ...

The current export dispatcher implements the strings mlx and hf. There is no generic export(format="any FORMAT member") interface and no .gguf file exporter in this dispatcher. GGUF block encoding and a native GGUF writer/container must not be conflated.

An important export limitation: the dispatcher defaults unrecognized method families to GPTQ Torch. QQQ/VPTQ artifacts therefore must not be assumed supported by this convenience function simply because load() supports them. Likewise the dense helper must recognize the actual module class. Keep the original checkpoint, inspect emitted config/tensors, and validate numerical outputs after migration.

Writer behavior and container files

ModelWriter(cls) is a class decorator, not a writer class. It installs save_quantized(), redirects the model wrapper's save_pretrained() to it, and installs the checkpoint reconstruction helpers. The installed method has this source signature:

def save_quantized(
        self,
        save_dir: str,
        safetensors_metadata: Optional[Dict[str, str]] = None,
        max_shard_size: Optional[Union[int, str]] = DEFAULT_MAX_SHARD_SIZE,
        meta_quantizer: Optional[str] = None,
        eora_path: Optional[str] = None,
        split_by: Optional[str] = None,
):
    ...

save_quantized() requires a quantized model, creates the output directory, writes quant_log.csv when available, records a gptqmodel:version fingerprint/URI and selected calibration controls, normalizes model config compatibility, and saves both config.json.quantization_config and quantize_config.json. It preserves dynamic rule order when copying the nested config: Transformers key sorting must not reorder first-match rules. Config-only BaseQuantizeConfig.save_pretrained() is a different API and does not create its destination directory.

Artifact/control Behavior
model.safetensors Single-file default basename.
Shards + model.safetensors.index.json Streaming writer maps tensor names to shard files and records metadata.total_size. A stale index is removed for ordinary single-file output.
safetensors_metadata Converted to string keys/values; container metadata format is forced to pt. This pt tag is not the quantization layout.
max_shard_size int bytes or binary suffix string; backend shard support may disable it.
split_by Layer-oriented splitting is disabled in this revision: SUPPORTED_SPLIT_BY = {None}. None, an empty string, and the string "none" select ordinary output; split_by="layer" raises ValueError, despite the error text and retained internal layer-directory writer mentioning it.
Loaded quantized model resave Except EXL3, writer rebuilds/reloads a checkpoint-facing model before collecting tensors, avoiding blindly saving runtime-repacked buffers.
EXL3 Rebuilds tensor_storage from actual quantized modules.
EoRA Dedicated PEFT-style adapter files; not embedded as dense weight updates.
Tokenizer/processor/remote code Saved/copied when available/allowed; processor follows tokenizer so multimodal templates are not overwritten.

utils.safetensor.inspect_safetensors(directory) prints tensor keys/shapes/dtypes for inspection. untie_weights(model) clones tied tensors because SafeTensors cannot represent arbitrary shared weight storage. Neither changes a quantization format. The writer also contains a module-replacement/embedding checkpoint path, which updates only affected tensor prefixes while retaining the rest of the source checkpoint and updating its index/config.

Pitfalls and validation checklist

  1. Read both method and format, resolving special-family subtype strings. Do not infer layout from a filename or bits alone.
  2. Compare config aliases, canonical filename and nested HF metadata before blaming a kernel. A stale preferred config file overrides a corrected fallback file.
  3. Inspect tensor names, shapes and dtypes. GPTQ qzeros, AWQ zeros, FP8 inverse scales and BitsAndBytes QuantState are different contracts.
  4. Preserve g_idx, codebooks, permutations, rotation state and EXL3 side tensors. They are part of the model, not optional training logs.
  5. Probe the exact device, dtype, group size, shape, sharding and dynamic overrides. A family declaration is not sufficient feasibility evidence.
  6. Use checkpoint-facing modules when repacking or resaving; private post-init weights may no longer be valid sources.
  7. Preserve a pristine source artifact. After migration, compare outputs against its dequantized/reference behavior before deploying.
  8. Do not claim that changing bits or algorithm families is lossless conversion. Requantization changes approximation and may require a new calibration job.

Source files

  • gptqmodel/quantization/config.py — FORMAT/METHOD, file constants, aliases, concrete export families and normalization.
  • gptqmodel/quantization/__init__.py — public exports.
  • gptqmodel/models/loader.py — format/backend routing, native GGUF/EXL3 handling.
  • gptqmodel/models/writer.py — config/tensor/adapter save paths and streaming shard integration.
  • gptqmodel/models/auto.py — GPTQModel.export dispatcher.
  • gptqmodel/nn_modules/converter.py — architecture conversion registry (empty in this revision), not bit-layout conversion.
  • gptqmodel/nn_modules/qlinear/ — kernel support declarations, scalar layouts, packers and runtime ABI conversion.
  • gptqmodel/nn_modules/exllamav3.py, gptqmodel/utils/exllamav3.py — EXL3 wrapper and tensor-storage descriptors.
  • gptqmodel/utils/model.py — GPTQ zero-point conversions and packing/model utilities.
  • gptqmodel/utils/packing.py — stable hf_ packing/repacking APIs.
  • gptqmodel/utils/importer.py, gptqmodel/utils/backend.py — kernel discovery/validation and BACKEND aliases.
  • gptqmodel/utils/bitblas.py, gptqmodel/utils/marlin.py, gptqmodel/utils/mlx.py — backend-specific conversion/runtime helpers.
  • gptqmodel/utils/hf.py — Transformers API/model/GGUF compatibility.
  • gptqmodel/utils/safetensor.py — inspection and untied storage helpers.
  • docs/hf_packing_api.md, docs/gptq_planar.md — integration API and planar wire-format design notes.

See also

Documentation home · Quickstart · Troubleshooting · Repository · PyPI

Source baseline: GPT-QModel 7.7.0 / c197691. Maintained by ModelCloud.ai.

Home

Getting started

Quantization methods

Complementary configs

Using a model

Platforms

Internals

Clone this wiki locally