Repository navigation
Dtypes and Formats
🏠 Home · 🧭 Choose a method · 🚀 Quickstart · 🧰 Help
On this page
Understand what a checkpoint actually stores before choosing how to run it. This guide is for readers comparing “four-bit” models, estimating storage, or diagnosing a format/backend mismatch.
Three reading depths: choose and understand → inspect and use → number representations, math, and advanced details.
| Choose this guide when… | Consider another path when… |
|---|---|
| A model name says INT4, FP4, NF4, GGUF, or NF12 and you need to know what that means | You want to pick an optimizer: use Choosing a Method and Dtype |
| You are estimating payload versus total memory | You need exact tensor names/layouts: use Packing Formats and Compatibility |
| You are checking whether two checkpoints are interchangeable | You already know the layout and need a loader: use Loading and Inference |
| Question | Example | Why it matters |
|---|---|---|
| What was the source dtype? | BF16 source weights, or prequantized FP8/FP4 source tensors | Converting a source to BF16 can lose information before a “lossless BF16” codec runs |
| What numeric representation is stored? | Uniform INT4, E2M1, NF4 indices, vector-codebook indices | The same four payload bits can decode to different numbers |
| Which method produced it? | GPTQ, AWQ, MACaQ, VPTQ | Different optimizers may use similar bit widths but different side state and constraints |
| Which checkpoint layout/container carries it? |
gptq_v2 packed words in safetensors; a standalone .gguf file |
Layout defines tensor organization; the file container is a different layer |
| What does the backend compute with? | Packed weights, dynamic INT8 activations, higher-precision accumulation/output | Storage dtype does not determine activation, accumulator, or output dtype |
W4A16 describes nominal weight/activation precision, not a complete format specification. An INT32 qweight tensor can contain many four-bit codes; its Torch dtype is a packing word, not “32-bit weights.” A kernel may temporarily decode a bounded tile without changing the checkpoint's stored representation.
- Source dtype: the weights before the target conversion. Automatic NF12 source storage can reduce calibration-source storage while restoring original eligible BF16 bits before quantization.
-
Stored dtype/encoding: packed codes plus scales, zero points, codebooks, exceptions, transforms, or adapter state. A config's
bitsfield generally describes the selected family—not complete file bits per weight. -
Activation/compute dtype: what flows into and through the kernel. FP8 weights need not mean FP8 activations; QQQ dynamically quantizes activations to INT8; MACaQ can use
w4a4orw4a16. Accumulator/output rounding remains a separate runtime contract.
See FP8/NVFP4, QQQ, MACaQ, and NF12 source storage for concrete examples.
| Family | Main payload and important side state | Meaningful distinction / follow-up |
|---|---|---|
| GPTQ / GPTAQ / RTN scalar integers | Integer codes, scales, zero-point semantics, optional group indices | GPTAQ and RTN are solver/lifecycle choices, not independent number systems; GPTQ |
| AWQ | Scalar integer codes with AWQ packing/scale conventions | Activation-aware solve is distinct from GEMM/GEMV serialization; AWQ |
| QQQ | INT4 weights, grouped/channel scale hierarchy; dynamic INT8 activations at runtime | Not ordinary GPTQ INT4 with a renamed backend; QQQ |
| MACaQ MXFP4 | E2M1 codes, one E8M0 scale per 32 values, plus enabled transform state |
mxfp4_macaq with transforms; mxfp4 when both are disabled. Retain method="macaq"; see MACaQ. |
| EXL3 | Trellis-coded integers with per-module widths, scales and transform/layout state | Model-average target width differs from module width and physical storage; EXL3 |
| VPTQ | Indices into vector centroids, optional residual centroids, scaling/state | Index bits amortized over vector length omit codebook and metadata cost; VPTQ |
| Paro | Four-bit symmetric packed weights plus learned rotation/scale side tensors | Side tensors are required to reproduce execution; Paro |
| FP8 | E4M3/E5M2 floating-point weight bytes with row/block/tensor scales | Exponent/mantissa semantics and scale granularity differ; not a blanket FP8 matmul guarantee; FP8 |
| NVFP4 | E2M1 values with NVFP4's block/global scaling contract | Native FP4 source import differs from GPTQ W4A-NVFP4, which stores GPTQ INT4 weights and quantizes activations; NVFP4 |
| NF12 | Twelve main bits/value plus variable exponent escapes and metadata | Exact eligible BF16 recovery, not guaranteed fixed 12-bit lossy quantization; NF12 |
| Tenstorrent block floats | Shared exponent per 16 values plus signed mantissa codes |
bf8_b, bf4_b, bf2_b are not IEEE FP8/FP4/FP2; model and standalone-helper scopes differ; Tenstorrent
|
| GGUF / GGML families | Q, K, IQ, TQ and supported MXFP4/NVFP4 block layouts; each defines scales/subblocks/codebooks | GGUF is a container/family label, not one dtype; GGUF lists supported aliases and import/export boundaries |
| bitsandbytes NF4 / FP4 / INT8 | Nonuniform NF4 or FP4 code indices with block maxima/nested stats; rowwise INT8 with scales | NF4 is neither NF12 nor MXFP4/NVFP4; BNB state must travel with the codes; BitsAndBytes |
For
| Example | Payload plus local block metadata | What this estimate omits |
|---|---|---|
| Ideal INT4, group size 128, one FP16 scale/group |
|
Zero points, group indices, padding and non-quantized tensors; not a complete GPTQ file-size claim |
| MXFP4 E2M1 + E8M0, blocks of 32 |
|
MACaQ transforms, biases, alignment and other tensors |
| GGML Q4_0, blocks of 32 |
|
File/container metadata, unquantized tensors, alignment |
| NF12 | Main stream is 12 bits/value; escapes add variable storage | Distribution-dependent exceptions/metadata; total can erase the nominal saving |
| Vector codebook with |
Index contribution |
Centroid storage, residual indices/codebooks, scales, alignment |
These are explanatory arithmetic examples, not measured model sizes or speed predictions. Compare actual saved artifacts and runtime measurements under the same workload.
Prerequisite: a local GPT-QModel checkpoint directory. This standard-library-only example reads metadata, not tensor payloads; missing fields are not a reason to guess a dtype. Embedded or older metadata may live in config.json, so use the loader and compatibility reference for authoritative normalization.
import json
from pathlib import Path
checkpoint = Path("/path/to/checkpoint")
metadata_path = checkpoint / "quantize_config.json"
metadata = json.loads(metadata_path.read_text(encoding="utf-8"))
for key in ("method", "quant_method", "format", "checkpoint_format",
"bits", "group_size", "sym", "desc_act", "pack_dtype"):
if key in metadata:
print(f"{key}: {metadata[key]}")Then match method + format + bits/subtype + scale/group geometry + model + backend + device. Preserve every shard, index, config and side tensor. Renaming a file or changing a JSON string does not convert a layout.
These are alternative configuration examples, not a request to load three models at once:
from gptqmodel.quantization import BitsAndBytesConfig, FP8Config, NF12Config
nf4_weights = BitsAndBytesConfig(bits=4, format="nf4")
fp8_weights = FP8Config(format="e4m3", weight_scale_method="row")
lossless_bf16_storage = NF12Config()All three use calibration-free weight-only lifecycles, but the outputs have different representations and loader contracts. Follow BitsAndBytes, FP8, or NF12 for complete load → convert → save → reload examples and prerequisites. For calibrated recipes use Choosing a Method and Dtype.
METHOD, FORMAT, and BACKEND are separate registries. The pinned public FORMAT values include:
-
GPTQ-family layouts:
gptq,gptq_v2,gptq_p; specializedmarlinandbitblaslayouts also exist. -
AWQ-family layouts:
gemm,gemv,gemv_fast,llm-awq. -
Other families:
qqq,exl3,vptq,paro,fp8,bitsandbytes,nf12,tt_blockfloat,gguf,mxfp4_macaq.
The same spelling can appear in more than one registry without making the concepts identical. Some families have subtype fields rather than one FORMAT member per numeric encoding. For example, selecting NF4 uses the bitsandbytes config subtype, while FORMAT.BITSANDBYTES identifies the checkpoint family.
GPT-QModel defaults to safetensors checkpoints, including non-quantized tensors. A standalone .gguf file is a different container: the public GGUF path supports eligible imports and GPT-QModel directory outputs, not a general standalone .gguf writer. See GGUF and the public serialization rule.
For an affine integer group with positive scale
The exact signed convention, zero storage, packing order and group-axis mapping are format-specific. GPTQ v1/v2 zero semantics, for instance, must be handled by the proper reader, not inferred from this generic equation. GPTQ and Packing Formats give the actual contracts.
Normal binary floating-point values have sign, exponent and fraction: schematically
For block floating point, the exponent is shared rather than stored independently for every value. Tenstorrent's vendor layout and rounding differ from IEEE-like FP8 and from MXFP4; use its defined representation and CPU example, not a guessed FP4 decoder.
VPTQ reconstructs a vector from a centroid selected by an index, optionally adding a residual centroid. This changes the unit of quantization from a scalar to a vector, so an index-width label alone cannot predict total memory. The VPTQ math defines shapes, weighting and residual state.
NF12 instead preserves eligible BF16 bits through a compact exponent window plus escaped groups. It does not solve the scalar integer objective above. Compression depends on the exponent distribution; the NF12 specification defines the actual tiled layout. Exact restored weights do not guarantee identical outputs across kernels with different accumulation/output rounding.
- Refinement is not a dtype: GSQ refines/allocates supported representations; EoRA is low-rank compensation with additional state.
- Preprocessing is not a format: rotations/smoothing/norm tuning must be admitted by the selected lifecycle and preserved where execution needs them.
- Backend is not hardware proof: use explicit NVIDIA CUDA and AMD ROCm declarations plus architecture gates. Triton availability alone establishes neither.
- Fusion is not recompression: Projection Fusion changes eligible execution while preserving native packed-layout contracts.
-
Source compression is not output selection:
auto_dtype_castcan affect calibration-source storage without changing the final target format. - A valid config is not a quality result: evaluate the reloaded model and report actual conditions.
Scoped to source 13ee428f2. Registry/config definitions: config.py. Float source dtype/decode helpers: dtype.py. Runtime registry: backend.py.
Next: choose a method, inspect packing compatibility, or load a checkpoint.
Documentation home · Quickstart · Troubleshooting · Repository · PyPI
Version scope: public source 54888582e (package version 7.12.0); older pinned evidence is identified on individual pages. Choose a method · Dtypes and formats. Maintained by ModelCloud.ai.
🧭 1 · Choose and understand
🚀 2 · Use it
- Installation and Quickstart
- Loading and Inference
- Evaluation
- Serving
- Troubleshooting and FAQ
- Memory, Offload, Multi-GPU
- MLX / Apple Silicon
- Tenstorrent / Block Floats
- Hugging Face Integration
- Multimodal and MoE Models
🎛️ Methods and representations
Each guide: choose → examples → math / advanced
- GPTQ / GPTAQ / RTN
- AWQ
- QQQ
- MACaQ / MXFP4
- EXL3
- VPTQ
- Paro
- FP8 and NVFP4
- NF12: lossless BF16 storage
- GGUF families
- BitsAndBytes: NF4 / FP4 / INT8
🧰 Refinement and execution
🧠 3 · Theory and advanced reference