Skip to content

Choosing a Method and Dtype

Qubitium edited this page Nov 17, 2026 · 4 revisions

🧭 Choosing a Method and Dtype

🏠 Home · 🧭 Choose a method · 🚀 Quickstart · 🧰 Help

On this page

Choose a compression recipe that your deployment can actually load, then measure whether it preserves the behavior you need. This guide is for model builders selecting a first recipe and operators deciding whether an existing checkpoint fits their hardware.

Three reading depths: choose and understand → use a recipe → theory, math, and advanced controls.

Choose this guide when… Consider another path when…
You are comparing methods, calibration effort, or weight/activation precision You already have a checkpoint: start with Loading and Inference
You need to separate a method from its stored representation You need exact layout fields: use Dtypes and Formats and packing compatibility
You want a shortlist, not an unsupported speed/accuracy leaderboard You need an answer for your workload: run Evaluation and deployment-specific measurements

🧭 Choose and understand

Start with the runtime, not the smallest number

  1. Fix the destination. Is this native GPT-QModel inference, Apple Silicon MLX, a Tenstorrent runtime, or an external engine? A format supported by one is not automatically accepted by another.
  2. Check the model and kernel contract. Match architecture, device family, GPU generation, activation dtype, matrix shapes, group size, packing, and optional dependencies. Use Supported Models, Inference Backends, and Kernel Backends Reference.
  3. Choose your calibration budget. Representative activation data enables different optimizers; no-data packing is a different lifecycle, not a promise of equal quality.
  4. Set a quality budget. Compare against the same dense model with the same tokenizer, prompt template, sequence lengths, and held-out tasks. Evaluate after save/reload, not just immediately after solving.
  5. Measure total deployment cost. Checkpoint bytes, runtime weight memory, KV cache, temporary workspaces, throughput, latency, and quality are different measurements.

Compatibility is not qualification. A public enum, accepted config, or declared device family is evidence of an API/support contract—not a benchmark or proof that every model/device combination was tested. This guide deliberately gives no universal speed or accuracy ranking.

Method and dtype shortlist

Each linked method page has a practical example first, followed by mathematical intuition and advanced controls.

Family Why you might choose it Calibration and honest limits Example, math, and controls
GPTQ Calibrated scalar low-bit weights with GPTQ-family checkpoint choices Representative activations; group/shape/backend constraints depend on width and layout GPTQ
GPTAQ GPTQ with native-reference information for accumulated-error correction Additional capture/state; not a new storage dtype; routed-expert and zero-sample caveats matter GPTQ / GPTAQ
RTN A calibration-free scalar rounding baseline or packing path No Hessian solve; useful baseline, not an accuracy-equivalent substitute for calibration GPTQ / RTN
AWQ Activation-aware scaling/clipping before weight packing Representative activations; native GEMM/GEMV paths are four-bit, while portable mixed-width paths have separate limits AWQ
QQQ INT4 weights with dynamic INT8 activation execution Hessian-based calibration; native NVIDIA CUDA and portable/MLX paths have different gates; no automatic model rotation QQQ
MACaQ / MXFP4 Calibrated E2M1 weights and optional MXFP4 activations Requires calibration; default transforms use mxfp4_macaq. Disabling both selects mxfp4; transform-free W4A16 supports stock MLX. MACaQ
EXL3 Trellis-coded weights with per-module integer widths and target average allocation Calibration and solver resources; target average bits is not exact file bits/weight; native and portable runtimes differ EXL3
VPTQ Vector-codebook compression with optional residual codebooks Calibration and codebook optimization; codebook/index/state overhead matters, especially for small matrices VPTQ
Paro Learned pairwise rotations/scaling with low-bit weights Calibration/optimization; four-bit symmetric contract and required side tensors Paro
FP8 Floating-point weight packing, calibration-free by default, with optional calibrated GSQ refinement E4M3/E5M2 and scaling granularity must match the runtime; gsq_calibration=True requires enabled GSQ and calibration; FP8 weights alone do not mean FP8 activations FP8 and GSQ mode
NVFP4 / W4A streams Evaluate explicitly supported low-precision activation execution At this pin, GPTQ W4A-NVFP4 stores GPTQ INT4 weights, not NVFP4 weights; native source FP4 loading is another path; NVIDIA-specific gates FP8 and NVFP4
NF12 Exact recovery of BF16 weight bits, or compressed calibration-source storage No activation calibration; 12-bit main payload plus variable escapes/metadata; converting a non-BF16 source first can be lossy NF12
Tenstorrent bf8_b / bf4_b / bf2_b Shared-exponent block-float experiments or compatible block-float deployment Model checkpoint conversion/runtime and standalone tensor helper are separate; bf2_b CPU helper is not model-loader support Tenstorrent
GGUF families Select named block layouts: classic Q types, K quants, IQ, TQ, MXFP4, and NVFP4 subtypes Normal packing is calibration-free; type-specific support, conversion, and optional allocation have separate requirements; not a standalone .gguf export promise GGUF
bitsandbytes NF4 / FP4 / INT8 Calibration-free codebook four-bit or rowwise eight-bit weights Requires compatible library/state schema; GPT-QModel's four-bit default is FP4, so select NF4 explicitly BitsAndBytes

For exact public method/format identifiers and why “four-bit” is ambiguous, continue to Dtypes and Formats.

Hardware-aware starting points

These are where to check, not blanket compatibility claims.

Destination Start by checking Do not assume
NVIDIA CUDA Chosen backend's SM, OS, activation dtype, shape, and extension requirements Every NVIDIA GPU has native FP8/FP4 or supports every CUDA kernel
AMD ROCm Explicit DEVICE.ROCM declarations and AMD architecture restrictions CUDA availability or Triton implementation implies ROCm support; HIP capability tuples are NVIDIA SM versions
Apple Silicon MLX, compatible model wrapper, supported packed family, Metal/dependency gates A Torch mps path and a native MLX path are the same runtime
CPU Portable kernel declarations, dependency availability, memory/workspace needs Packed weights mean no temporary dequantization or all kernels use low-bit CPU arithmetic
Tenstorrent cards Vendor runtime and model route; distinguish XLA, TTNN, and native block-float execution A standalone ttbfp CPU example proves card execution, or any Torch kernel accepts XLA
vLLM / SGLang / other engines Exact engine version, model support, serialized format, and Serving A checkpoint GPT-QModel loads is automatically accepted by another engine

In family-valued GPT-QModel configuration, keep NVIDIA DEVICE.CUDA and AMD DEVICE.ROCM explicit. At Torch execution boundaries, an AMD HIP build also spells devices cuda:N; that spelling does not declare NVIDIA support. See the public device-family contract.

Calibration-aware choices

What you have A useful next step What still needs checking
Representative text and time to optimize Compare calibrated GPTQ/GPTAQ, AWQ, QQQ, MACaQ, EXL3, VPTQ, or Paro on a bounded pilot Their optimization costs, replay needs, model restrictions, and deployment contracts differ
No representative activation corpus Start with RTN, FP8, bitsandbytes, normal GGUF packing, or BF16 NF12 storage as appropriate Calibration-free does not mean lossless, except BF16 NF12's codec contract
Long-context, multilingual, code, multimodal, or MoE workloads Match calibration length/content, modalities, masks, and routed-expert coverage to intended use A few short generic strings cannot validate these behaviors
Only a tiny test fixture Use it as an API demo, then replace it before making a quality decision Passing the call sequence is not an accuracy recipe

Keep evaluation data disjoint from calibration. Inspect filtering and effective token counts; adding strings that are too short to survive preprocessing adds no useful calibration. Under memory pressure, use offload and multi-GPU controls rather than silently changing the data distribution.

🚀 Use it

First pilot: a calibrated GPTQ checkpoint

Prerequisites: a matching installation, an unquantized supported model, enough device/host storage, and a UTF-8 JSON file containing a list of representative calibration strings. Paths below are placeholders. These are API steps, not a prescribed sample count or quality recipe.

import json
from pathlib import Path

from gptqmodel import CalibrationConfig, GPTQModel
from gptqmodel.models._const import DEVICE
from gptqmodel.quantization import GPTQConfig

calibration = json.loads(Path("calibration.json").read_text(encoding="utf-8"))
if not isinstance(calibration, list) or not calibration:
    raise ValueError("Provide a nonempty list of representative strings")
if not all(isinstance(text, str) and text.strip() for text in calibration):
    raise ValueError("Every calibration entry must be a nonempty string")

config = GPTQConfig(bits=4, group_size=128, device=DEVICE.CUDA)
# For AMD, use DEVICE.ROCM and a matching HIP build; check backend gates.
model = GPTQModel.load("/path/to/dense-model", quantize_config=config)
model.quantize(CalibrationConfig(dataset=calibration, batch_size=1))
model.save("/path/to/gptq-checkpoint")

Then reload the saved checkpoint following Loading and Inference, evaluate it using Evaluation, and test the intended serving engine. A successful save() is not the finish line.

Change one choice at a time

To build a calibration-free GPTQ-layout baseline, replace the config with RTNConfig(bits=4, group_size=128, device=DEVICE.CUDA) imported from gptqmodel.quantization, then use model.quantize() without calibration. This changes the solve/lifecycle, not automatically the output family.

For other candidates, use the linked method page's complete example instead of swapping a method string into the GPTQ recipe. In particular, NF12, bitsandbytes, MACaQ, GGUF, and Tenstorrent routes do not share every GPTQ option.

Record the source revision, full normalized config, model/tokenizer revisions, calibration preprocessing, checkpoint format, backend, hardware/runtime, and evaluation prompts. Compare equivalent deployment conditions; do not rank methods from unrelated historical benchmark tables.

🧠 Go deeper

Why calibration can change rounding decisions

For a linear layer, let $X\in\mathbb{R}^{T\times K}$ hold calibration input rows and $W\in\mathbb{R}^{N\times K}$ be dense weights. A reconstructed weight $\widehat W$ produces output error

$$ L=|X(W-\widehat W)^\top|_F^2 =\operatorname{tr}\big((W-\widehat W)X^\top X(W-\widehat W)^\top\big). $$

The Gram matrix $X^\top X$ tells an optimizer which input directions matter on that corpus. RTN instead rounds to a selected grid without this activation-weighted solve. Calibration methods do not all optimize this exact objective in the same way: GPTQ uses second-order error compensation, AWQ searches scaling/clipping, MACaQ combines transforms and MXFP4 refinement, and Paro optimizes rotations/scales. The individual pages define their normalization, surrogate loss, constraints, and finite-precision caveats.

A lower layer-reconstruction loss on calibration inputs does not guarantee better held-out generation. Likewise, NF12 is a representation/codec contract rather than an algorithm needing an invented calibration objective.

Refinement, adapters, and execution are separate knobs

Layer of choice What it does Read next
Method / solve Selects or optimizes quantized weights and method state Method pages above
GSQ refinement / RCO allocation Refines eligible scalar solutions or allocates supported layouts under a budget; not a replacement file format GSQ
EoRA adapter Adds low-rank error compensation on supported calibrated paths; not a replacement storage dtype EoRA
Rotation, smoothing, norm tuning Changes supported preprocessing or recovery; not a universal drop-in guarantee Controls
Checkpoint layout and container Defines stored codes, scales, state, and how they are written Dtypes and Formats · Packing
Backend / fusion Executes compatible packed state; fusion is not a new quantization algorithm Backends · Projection Fusion

Troubleshooting a choice

  • It will not load: inspect exact method, layout, backend, dependency and architecture gates before requantizing.
  • It fits on disk but not at runtime: include KV cache, activations, workspaces, non-quantized tensors and any adapter state in the budget.
  • Quality regressed: compare a non-quantized baseline with identical tokenizer/template first, then inspect calibration coverage, fallback logs, held-out metrics and sensitive modules.
  • You need more control: use QuantizeConfig Reference and Quantization Lifecycle; verify each option is admitted by the chosen lifecycle.

Public source and next steps

This guide is scoped to source 13ee428f2, whose package version is 7.11.0. Method/config/layout contracts are in config.py, public config exports in quantization/init.py, and runtime identifiers in backend.py.

Next: understand Dtypes and Formats, run the Quickstart, or open the chosen method's full guide above.

Clone this wiki locally