Repository navigation
Choosing a Method and Dtype
🏠 Home · 🧭 Choose a method · 🚀 Quickstart · 🧰 Help
On this page
Choose a compression recipe that your deployment can actually load, then measure whether it preserves the behavior you need. This guide is for model builders selecting a first recipe and operators deciding whether an existing checkpoint fits their hardware.
Three reading depths: choose and understand → use a recipe → theory, math, and advanced controls.
| Choose this guide when… | Consider another path when… |
|---|---|
| You are comparing methods, calibration effort, or weight/activation precision | You already have a checkpoint: start with Loading and Inference |
| You need to separate a method from its stored representation | You need exact layout fields: use Dtypes and Formats and packing compatibility |
| You want a shortlist, not an unsupported speed/accuracy leaderboard | You need an answer for your workload: run Evaluation and deployment-specific measurements |
- Fix the destination. Is this native GPT-QModel inference, Apple Silicon MLX, a Tenstorrent runtime, or an external engine? A format supported by one is not automatically accepted by another.
- Check the model and kernel contract. Match architecture, device family, GPU generation, activation dtype, matrix shapes, group size, packing, and optional dependencies. Use Supported Models, Inference Backends, and Kernel Backends Reference.
- Choose your calibration budget. Representative activation data enables different optimizers; no-data packing is a different lifecycle, not a promise of equal quality.
- Set a quality budget. Compare against the same dense model with the same tokenizer, prompt template, sequence lengths, and held-out tasks. Evaluate after save/reload, not just immediately after solving.
- Measure total deployment cost. Checkpoint bytes, runtime weight memory, KV cache, temporary workspaces, throughput, latency, and quality are different measurements.
Compatibility is not qualification. A public enum, accepted config, or declared device family is evidence of an API/support contract—not a benchmark or proof that every model/device combination was tested. This guide deliberately gives no universal speed or accuracy ranking.
Each linked method page has a practical example first, followed by mathematical intuition and advanced controls.
| Family | Why you might choose it | Calibration and honest limits | Example, math, and controls |
|---|---|---|---|
| GPTQ | Calibrated scalar low-bit weights with GPTQ-family checkpoint choices | Representative activations; group/shape/backend constraints depend on width and layout | GPTQ |
| GPTAQ | GPTQ with native-reference information for accumulated-error correction | Additional capture/state; not a new storage dtype; routed-expert and zero-sample caveats matter | GPTQ / GPTAQ |
| RTN | A calibration-free scalar rounding baseline or packing path | No Hessian solve; useful baseline, not an accuracy-equivalent substitute for calibration | GPTQ / RTN |
| AWQ | Activation-aware scaling/clipping before weight packing | Representative activations; native GEMM/GEMV paths are four-bit, while portable mixed-width paths have separate limits | AWQ |
| QQQ | INT4 weights with dynamic INT8 activation execution | Hessian-based calibration; native NVIDIA CUDA and portable/MLX paths have different gates; no automatic model rotation | QQQ |
| MACaQ / MXFP4 | Calibrated E2M1 weights and optional MXFP4 activations | Requires calibration; default transforms use mxfp4_macaq. Disabling both selects mxfp4; transform-free W4A16 supports stock MLX. |
MACaQ |
| EXL3 | Trellis-coded weights with per-module integer widths and target average allocation | Calibration and solver resources; target average bits is not exact file bits/weight; native and portable runtimes differ | EXL3 |
| VPTQ | Vector-codebook compression with optional residual codebooks | Calibration and codebook optimization; codebook/index/state overhead matters, especially for small matrices | VPTQ |
| Paro | Learned pairwise rotations/scaling with low-bit weights | Calibration/optimization; four-bit symmetric contract and required side tensors | Paro |
| FP8 | Floating-point weight packing, calibration-free by default, with optional calibrated GSQ refinement | E4M3/E5M2 and scaling granularity must match the runtime; gsq_calibration=True requires enabled GSQ and calibration; FP8 weights alone do not mean FP8 activations |
FP8 and GSQ mode |
| NVFP4 / W4A streams | Evaluate explicitly supported low-precision activation execution | At this pin, GPTQ W4A-NVFP4 stores GPTQ INT4 weights, not NVFP4 weights; native source FP4 loading is another path; NVIDIA-specific gates | FP8 and NVFP4 |
| NF12 | Exact recovery of BF16 weight bits, or compressed calibration-source storage | No activation calibration; 12-bit main payload plus variable escapes/metadata; converting a non-BF16 source first can be lossy | NF12 |
Tenstorrent bf8_b / bf4_b / bf2_b |
Shared-exponent block-float experiments or compatible block-float deployment | Model checkpoint conversion/runtime and standalone tensor helper are separate; bf2_b CPU helper is not model-loader support |
Tenstorrent |
| GGUF families | Select named block layouts: classic Q types, K quants, IQ, TQ, MXFP4, and NVFP4 subtypes | Normal packing is calibration-free; type-specific support, conversion, and optional allocation have separate requirements; not a standalone .gguf export promise |
GGUF |
| bitsandbytes NF4 / FP4 / INT8 | Calibration-free codebook four-bit or rowwise eight-bit weights | Requires compatible library/state schema; GPT-QModel's four-bit default is FP4, so select NF4 explicitly | BitsAndBytes |
For exact public method/format identifiers and why “four-bit” is ambiguous, continue to Dtypes and Formats.
These are where to check, not blanket compatibility claims.
| Destination | Start by checking | Do not assume |
|---|---|---|
| NVIDIA CUDA | Chosen backend's SM, OS, activation dtype, shape, and extension requirements | Every NVIDIA GPU has native FP8/FP4 or supports every CUDA kernel |
| AMD ROCm | Explicit DEVICE.ROCM declarations and AMD architecture restrictions |
CUDA availability or Triton implementation implies ROCm support; HIP capability tuples are NVIDIA SM versions |
| Apple Silicon | MLX, compatible model wrapper, supported packed family, Metal/dependency gates | A Torch mps path and a native MLX path are the same runtime |
| CPU | Portable kernel declarations, dependency availability, memory/workspace needs | Packed weights mean no temporary dequantization or all kernels use low-bit CPU arithmetic |
| Tenstorrent cards | Vendor runtime and model route; distinguish XLA, TTNN, and native block-float execution | A standalone ttbfp CPU example proves card execution, or any Torch kernel accepts XLA |
| vLLM / SGLang / other engines | Exact engine version, model support, serialized format, and Serving | A checkpoint GPT-QModel loads is automatically accepted by another engine |
In family-valued GPT-QModel configuration, keep NVIDIA DEVICE.CUDA and AMD DEVICE.ROCM explicit. At Torch execution boundaries, an AMD HIP build also spells devices cuda:N; that spelling does not declare NVIDIA support. See the public device-family contract.
| What you have | A useful next step | What still needs checking |
|---|---|---|
| Representative text and time to optimize | Compare calibrated GPTQ/GPTAQ, AWQ, QQQ, MACaQ, EXL3, VPTQ, or Paro on a bounded pilot | Their optimization costs, replay needs, model restrictions, and deployment contracts differ |
| No representative activation corpus | Start with RTN, FP8, bitsandbytes, normal GGUF packing, or BF16 NF12 storage as appropriate | Calibration-free does not mean lossless, except BF16 NF12's codec contract |
| Long-context, multilingual, code, multimodal, or MoE workloads | Match calibration length/content, modalities, masks, and routed-expert coverage to intended use | A few short generic strings cannot validate these behaviors |
| Only a tiny test fixture | Use it as an API demo, then replace it before making a quality decision | Passing the call sequence is not an accuracy recipe |
Keep evaluation data disjoint from calibration. Inspect filtering and effective token counts; adding strings that are too short to survive preprocessing adds no useful calibration. Under memory pressure, use offload and multi-GPU controls rather than silently changing the data distribution.
Prerequisites: a matching installation, an unquantized supported model, enough device/host storage, and a UTF-8 JSON file containing a list of representative calibration strings. Paths below are placeholders. These are API steps, not a prescribed sample count or quality recipe.
import json
from pathlib import Path
from gptqmodel import CalibrationConfig, GPTQModel
from gptqmodel.models._const import DEVICE
from gptqmodel.quantization import GPTQConfig
calibration = json.loads(Path("calibration.json").read_text(encoding="utf-8"))
if not isinstance(calibration, list) or not calibration:
raise ValueError("Provide a nonempty list of representative strings")
if not all(isinstance(text, str) and text.strip() for text in calibration):
raise ValueError("Every calibration entry must be a nonempty string")
config = GPTQConfig(bits=4, group_size=128, device=DEVICE.CUDA)
# For AMD, use DEVICE.ROCM and a matching HIP build; check backend gates.
model = GPTQModel.load("/path/to/dense-model", quantize_config=config)
model.quantize(CalibrationConfig(dataset=calibration, batch_size=1))
model.save("/path/to/gptq-checkpoint")Then reload the saved checkpoint following Loading and Inference, evaluate it using Evaluation, and test the intended serving engine. A successful save() is not the finish line.
To build a calibration-free GPTQ-layout baseline, replace the config with RTNConfig(bits=4, group_size=128, device=DEVICE.CUDA) imported from gptqmodel.quantization, then use model.quantize() without calibration. This changes the solve/lifecycle, not automatically the output family.
For other candidates, use the linked method page's complete example instead of swapping a method string into the GPTQ recipe. In particular, NF12, bitsandbytes, MACaQ, GGUF, and Tenstorrent routes do not share every GPTQ option.
Record the source revision, full normalized config, model/tokenizer revisions, calibration preprocessing, checkpoint format, backend, hardware/runtime, and evaluation prompts. Compare equivalent deployment conditions; do not rank methods from unrelated historical benchmark tables.
For a linear layer, let
The Gram matrix
A lower layer-reconstruction loss on calibration inputs does not guarantee better held-out generation. Likewise, NF12 is a representation/codec contract rather than an algorithm needing an invented calibration objective.
| Layer of choice | What it does | Read next |
|---|---|---|
| Method / solve | Selects or optimizes quantized weights and method state | Method pages above |
| GSQ refinement / RCO allocation | Refines eligible scalar solutions or allocates supported layouts under a budget; not a replacement file format | GSQ |
| EoRA adapter | Adds low-rank error compensation on supported calibrated paths; not a replacement storage dtype | EoRA |
| Rotation, smoothing, norm tuning | Changes supported preprocessing or recovery; not a universal drop-in guarantee | Controls |
| Checkpoint layout and container | Defines stored codes, scales, state, and how they are written | Dtypes and Formats · Packing |
| Backend / fusion | Executes compatible packed state; fusion is not a new quantization algorithm | Backends · Projection Fusion |
- It will not load: inspect exact method, layout, backend, dependency and architecture gates before requantizing.
- It fits on disk but not at runtime: include KV cache, activations, workspaces, non-quantized tensors and any adapter state in the budget.
- Quality regressed: compare a non-quantized baseline with identical tokenizer/template first, then inspect calibration coverage, fallback logs, held-out metrics and sensitive modules.
- You need more control: use QuantizeConfig Reference and Quantization Lifecycle; verify each option is admitted by the chosen lifecycle.
This guide is scoped to source 13ee428f2, whose package version is 7.11.0. Method/config/layout contracts are in config.py, public config exports in quantization/init.py, and runtime identifiers in backend.py.
Next: understand Dtypes and Formats, run the Quickstart, or open the chosen method's full guide above.
Documentation home · Quickstart · Troubleshooting · Repository · PyPI
Version scope: public source 54888582e (package version 7.12.0); older pinned evidence is identified on individual pages. Choose a method · Dtypes and formats. Maintained by ModelCloud.ai.
🧭 1 · Choose and understand
🚀 2 · Use it
- Installation and Quickstart
- Loading and Inference
- Evaluation
- Serving
- Troubleshooting and FAQ
- Memory, Offload, Multi-GPU
- MLX / Apple Silicon
- Tenstorrent / Block Floats
- Hugging Face Integration
- Multimodal and MoE Models
🎛️ Methods and representations
Each guide: choose → examples → math / advanced
- GPTQ / GPTAQ / RTN
- AWQ
- QQQ
- MACaQ / MXFP4
- EXL3
- VPTQ
- Paro
- FP8 and NVFP4
- NF12: lossless BF16 storage
- GGUF families
- BitsAndBytes: NF4 / FP4 / INT8
🧰 Refinement and execution
🧠 3 · Theory and advanced reference