Repository navigation
VPTQ
Documentation home · Quickstart · Lifecycle
On this page
VPTQ (Vector Post-Training Quantization) compresses a linear layer with learned vector codebooks rather than independently rounding every scalar weight. Each small weight vector is represented by an index into a table of centroids; an optional residual codebook refines the reconstruction. GPT-QModel learns these tables from model weights and calibration activations, then applies GPTQ-style Hessian error compensation across input columns.
The implementation follows the recipe described by microsoft/VPTQ and VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models. It contains a Torch quantizer, a calibration looper processor, a portable Torch runtime, and a hybrid native CUDA runtime. No external VPTQ package or pre-generated Hessian files are required by this implementation.
| Identifier | Value | Purpose |
|---|---|---|
METHOD.VPTQ |
"vptq" |
Quantization algorithm |
FORMAT.VPTQ |
"vptq" |
Codebooks, indices, and reconstruction metadata stored in checkpoints |
BACKEND.VPTQ_TORCH |
"vptq_torch" |
Portable reference inference |
BACKEND.VPTQ_CUDA |
"vptq_cuda" |
Fused CUDA small-row inference; CUDA dequantization plus GEMM for other cases |
BACKEND.TORCH |
"torch" |
Method-aware alias resolved to VPTQ_TORCH for VPTQ |
Both VPTQ runtimes consume the same checkpoint layout. The method is calibration-based, even though the inference layer is a weight-only representation and its base class is named WeightOnlyQuantLinear. VPTQConfig.uses_weight_only_lifecycle() explicitly returns False.
Reading the diagram: Alternative recipes using calibration. Residuals are disabled with num_res_centroids=[-1, -1]; outlier residual codebooks are unsupported.
Text version of the workflow
- Residual disabled: Learn main codebooks → Assign and compensate → Pack main indices.
- Residual enabled: default: Learn main codebooks → Run first pass → Fit residual codebooks → Rerun and pack.
A linear weight is treated as [out_features, in_features]. For each input column, VPTQ splits the output-feature axis into vectors of vector_lens[1] elements. With vector length 8, each index reconstructs eight adjacent output weights from one centroid, rather than one scalar weight.
| Term | Meaning in GPT-QModel |
|---|---|
| Vector | A slice down one input column along out_features; the last slice is zero-padded when necessary |
| Centroid | A learned floating vector of the same length as the weight vector |
| Codebook | The table of centroids shared by vectors in one input-channel group |
| Main index | The integer selecting a main-codebook centroid |
| Residual centroid/index | An optional second table/index representing the remaining vector error after the main centroid |
| Group | Contiguous input columns that share a codebook, not a group of scalar quantization scales |
| Outlier group | A leading set of input columns with its own vector length and codebook |
Conceptually, a vector is reconstructed as main_codebook[main_index], or as main_codebook[main_index] + residual_codebook[residual_index] when residual quantization is enabled. Normalization and input permutation, if configured, are reversed during reconstruction.
Residual quantization is optional but enabled by default: num_res_centroids=[-1, 16] adds a 16-entry residual codebook for each main input-channel group. Use num_res_centroids=[-1, -1] to disable it. This removes the residual codebook/index stream, not calibration or Hessian compensation; a one-entry residual codebook is rejected rather than treated as disabled.
Vector length is along the output axis. group_size is along the input axis. Confusing the two produces an incompatible layout.
VPTQConfig.bits is derived, not an independent scalar quantization width. Let C be the main centroid count, R the residual count when enabled, and L the main vector length. The index-only rate is:
index_bits_per_vector = log2(C) + (log2(R) if residual is enabled else 0)
index_only_bits_per_weight = index_bits_per_vector / L
bits = clamp(ceil(index_only_bits_per_weight), 1, 8)
The default C=4096, R=16, L=8 uses 12 main-index bits plus 4 residual-index bits per eight-weight vector: 2 index bits per weight. A 4096-centroid codebook without residuals over length-8 vectors has a 1.5-bit index rate, but its integer bits label is still 2.
Neither number includes codebook storage, word padding, outlier indices, normalization tensors, or bias. Small layers and configurations with many codebooks can have substantial centroid overhead. bits=2 alone does not prove a two-bit total checkpoint size.
To choose the compression recipe, set vector_lens, num_centroids, and num_res_centroids. Passing a contradictory bits value logs that it is ignored and replaces it with the derived label. Labels are restricted to (1, 2, 3, 4, 5, 6, 7, 8); combined index width is independently limited to 32 bits.
The two entries in each paired configuration mean [outlier, main]. -1 disables an optional component.
| Configuration | Result |
|---|---|
group_size=-1, group_num=-1 |
One main codebook for all non-outlier input channels |
Positive group_size, group_num=-1
|
Enough codebooks to cover the non-outlier channels with that input-channel group size; last group may be padded |
group_size=-1, positive group_num
|
Requested number of main codebooks; group width is resolved from the input width and outlier policy |
| Both positive | Rejected |
Outliers are enabled only when npercent > 0, the outlier vector length is positive, and the outlier centroid count is greater than one. The initial outlier size is ceil(npercent / 100 * in_features), clamped to the input width. npercent=1.0 means one percent, not a fraction of one.
With an active outlier codebook and explicit group_num, remainder columns are added to the outlier set so every main group has equal width. Without an outlier codebook, the helper keeps outlier_size=0 and rounds the group width up; the runtime drops trailing padding during reconstruction. It does not silently create an outlier codebook merely to absorb a remainder. Keep every requested group nonempty: group_num <= remaining_columns alone does not guarantee a nonempty final group after rounding. See Group-count edge case.
The quantizer assigns the leading columns in its working order to the outlier group. enable_perm=True orders input columns by descending Hessian diagonal first; without it, the leading columns retain their original order. npercent alone is not an independent magnitude-based outlier search. Residual codebooks for the outlier group are not supported.
Calibration captures the per-module input Gram matrix H = mean(xᵀx). The accumulator uses FP64 and counts flattened input tokens, not dataset rows. This is built online: pass ordinary calibration data to model.quantize() instead of a hessian_path.
The optional transforms are distinct:
| Option | Effect |
|---|---|
kmeans_mode="hessian" |
Uses Hessian-diagonal importance as sample weights when learning centroids |
kmeans_mode="" |
Uniform k-means weighting; does not disable the later Hessian compensation pass |
enable_perm=True |
Sorts input columns by descending Hessian diagonal and permutes the Hessian consistently; the inverse permutation restores the original layout |
enable_norm=True, norm_dim=0 |
Per-input-column min/range normalization across output rows |
enable_norm=True, norm_dim=1 |
Per-output-row min/range normalization across input columns |
VPTQ's enable_perm is its own stored reconstruction feature. It is not an orthogonal model rotation and is not a reason to enable the unrelated shared rotation field.
This uses a supported model from the repository's VPTQ example. Access to the model may require accepting its upstream license and authenticating with Hugging Face.
import torch
from datasets import load_dataset
from gptqmodel import GPTQModel
from gptqmodel.quantization import VPTQConfig
from gptqmodel.utils.backend import BACKEND
model_id = "meta-llama/Llama-3.2-1B-Instruct"
output_path = "Llama-3.2-1B-Instruct-VPTQ"
calibration = load_dataset(
"allenai/c4",
data_files="en/c4-train.00001-of-01024.json.gz",
split="train",
).select(range(256))["text"]
qcfg = VPTQConfig(
vector_lens=[-1, 8],
num_centroids=[-1, 4096],
num_res_centroids=[-1, 16],
group_size=-1,
kmeans_mode="hessian",
)
model = GPTQModel.load(model_id, qcfg, dtype=torch.float16)
model.quantize(calibration, batch_size=1, backend=BACKEND.VPTQ_TORCH)
model.save(output_path)
# Native CUDA inference consumes the same saved codebooks/indices.
cuda_model = GPTQModel.load(
output_path, device="cuda:0", backend=BACKEND.VPTQ_CUDA
)For portable or CPU inference:
from gptqmodel import GPTQModel
from gptqmodel.utils.backend import BACKEND
model = GPTQModel.load(
"Llama-3.2-1B-Instruct-VPTQ",
device="cpu",
backend=BACKEND.VPTQ_TORCH,
)The quantization lifecycle uses VPTQProcessor and packs VPTQLinear with BACKEND.VPTQ_TORCH. Choosing a CUDA inference backend later does not require relearning the codebooks. For generation and tokenizer usage, see Loading and Inference.
VPTQConfig derives from GPTQConfig; these are its method-specific fields and defaults. List defaults are independently created for each config instance.
| Field | Default | Meaning/validation |
|---|---|---|
method |
METHOD.VPTQ |
Only VPTQ accepted by this class |
format |
FORMAT.VPTQ |
Only VPTQ export layout |
bits |
2, derived |
Replaced by the main/residual index-rate calculation; labels 1 through 8 |
vector_lens |
[-1, 8] |
[outlier_vector_len, main_vector_len]; main positive, outlier -1 or positive |
num_centroids |
[-1, 4096] |
[outlier, main] counts; main must be a power of two at least 2 |
num_res_centroids |
[-1, 16] |
Main residual -1 disables it; enabled count must be a power of two at least 2. Outlier residual must remain -1
|
npercent |
0.0 |
Nonnegative percentage of input channels for the outlier codebook; positive values require a usable outlier vector/codebook |
group_size |
-1 |
Input channels per main codebook; -1 lets group_num/single-group layout decide |
group_num |
-1 |
Number of main codebooks; otherwise positive; cannot be set together with group_size
|
kmeans_mode |
"hessian" |
Only "hessian" and "" accepted |
kmeans_iters |
100 |
Maximum Lloyd iterations; at least 1 |
kmeans_tol |
1e-5 |
Relative inertia-change tolerance; strictly between 0 and 1 |
kmeans_seed |
0 |
Seed for device-local centroid initialization |
enable_norm |
False |
Min/range normalization before learning/quantizing vectors |
norm_dim |
0 |
Only 0 or 1 accepted |
enable_perm |
False |
Hessian-diagonal input ordering with saved permutation |
block_size |
128 |
Input-column block size for error propagation; at least 1 |
percdamp |
0.01 |
Inverse-Hessian damping fraction; 0 <= percdamp < 1
|
All three paired fields must be two-item lists/tuples. Centroid counts are -1 or positive powers of two, subject to the stricter main/outlier requirements above. A one-entry main residual table is rejected because it would have no index bits. Main plus residual index width must be at most 32.
vptq_group_setting() additionally requires positive input width, group_size=-1 or positive, group_num=-1 or positive, and positive resolved group width/count. An explicit group count cannot exceed the initial non-outlier channel count. An outlier percentage leaving no main channels is therefore unusable even though the config validates npercent primarily as nonnegative. For explicit group counts without outliers, also check the nonempty-group invariant below; it is not fully enforced by the helper.
| Shared/inherited field | Default or relevant rule | VPTQ interpretation |
|---|---|---|
desc_act |
False after normalization |
Use enable_perm, not GPTQ activation ordering, for VPTQ's saved permutation |
sym |
True |
Not a selector between scalar symmetric/asymmetric VPTQ codebooks |
pack_dtype |
torch.int32 |
Runtime-supported packed index dtype |
dynamic |
None |
Layer exclusion can be honored; nonempty per-layer override dictionaries are rejected |
fallback |
Fallback(strategy=FallbackStrategy.RTN, threshold="0.5%", smooth=None) |
VPTQ interprets insufficient coverage as Hessian-free vector quantization, not scalar RTN |
true_sequential |
True |
Shared sequential replay lifecycle |
lm_head |
False |
Shared output-head selection |
offload_to_disk |
True |
Shared quantization memory management |
rotation |
None |
Not the VPTQ permutation feature; keep unset for this recipe |
mse, damp_percent, damp_auto_increment, static_groups, GPTAQ/FOEM options |
Inherited GPTQ-family surface | Not the VPTQ codebook-learning controls. VPTQ uses its own k-means fields, block_size, and percdamp
|
The runtime validator deliberately does not apply the scalar GPTQ group-size, symmetry, or activation-order contract to VPTQ. It checks the VPTQ format contract, shared device/dtype/adapter constraints, and nominal bit labels instead. See QuantizeConfig Reference for the shared config surface.
For K input channels and resolved (O, S, G) = (outlier_size, group_size, group_num), every main group needs at least one real column:
(G - 1) * S < K - O <= G * S
In this revision, K=5, group_num=4, group_size=-1 without outliers resolves to (O=0, S=2, G=4). Although 4 <= 5 passes the helper's group-count check, the fourth group starts beyond the matrix. The quantizer leaves that group's codebook absent and packing raises a missing-codebook error. Prefer an explicit positive group_size with group_num=-1, or choose a group count satisfying the invariant for every selected layer. Automatic padding covers partial groups, not empty codebooks.
Disable main residual quantization explicitly:
from gptqmodel.quantization import VPTQConfig
qcfg = VPTQConfig(
vector_lens=[-1, 8],
num_centroids=[-1, 4096],
num_res_centroids=[-1, -1],
)
# Index-only rate: 12/8 = 1.5 bits/weight; nominal qcfg.bits is 2.The repository guide also gives this larger-codebook, three-bit index-rate recipe:
qcfg = VPTQConfig(
vector_lens=[-1, 8],
num_centroids=[-1, 65536],
num_res_centroids=[-1, 256],
enable_perm=True,
enable_norm=True,
kmeans_mode="hessian",
)
# (16 + 8) / 8 = 3 index bits/weight, before codebooks and metadata.Larger codebooks increase centroid storage and distance-search work; this is not a free accuracy upgrade. For a concrete outlier/grouped layout:
qcfg = VPTQConfig(
vector_lens=[4, 8],
num_centroids=[256, 4096],
num_res_centroids=[-1, 16],
npercent=1.0,
group_size=128,
group_num=-1,
enable_perm=True,
)The actual group/outlier sizes are resolved separately for each layer's input width. The bits label continues to describe the main index-rate recipe, not an outlier-weighted checkpoint-size estimate.
The source exposes these VPTQ method signatures:
def __init__(self, module: nn.Module, qcfg: Optional[VPTQConfig] = None):
def add_batch(self, inp: torch.Tensor, out: Optional[torch.Tensor] = None) -> None:
def materialize_hessian(self, target_device: Optional[torch.device] = None) -> torch.Tensor:
def quantize(self) -> Tuple[VPTQQuantResult, float, float]:
def quantize_weight(
self,
weight: torch.Tensor,
hessian: torch.Tensor,
target_dtype: Optional[torch.dtype] = None,
) -> Tuple[VPTQQuantResult, float]:These are signatures for methods in gptqmodel/quantization/vptq.py, not standalone runnable definitions. quantize() returns (result, duration_seconds, hessian_weighted_loss). quantize_weight() operates on [out_features, in_features] weights and a matching input Hessian and returns (result, loss).
VPTQ accepts nn.Linear and Transformers Conv1D. The latter stores its weight transposed, so quantization and replay explicitly convert layouts. Actual nn.Conv1d and nn.Conv2d modules are rejected: reconstructing a dense F.linear weight would not preserve their convolution semantics.
Quantization needs FP64 Hessian accumulation and Cholesky support on its execution devices. The portable inference-device declarations do not guarantee that every such device can execute calibration and quantization.
| Step | Implementation behavior |
|---|---|
| Capture | Flatten inputs into tokens, accumulate FP64 xᵀx under a lock in per-device buffers, then merge/divide by sample count |
| Coverage/dead channels | Use identity Hessian if no usable calibration Hessian or configured low-coverage fallback applies; otherwise identify exactly zero diagonal channels and zero those weight columns while repairing the diagonal |
| Scale Hessian | Normalize by its mean diagonal |
| Normalize/permutate | Apply optional weight min/range normalization, form Hessian-diagonal k-means weights, then optionally reorder input columns and Hessian |
| Main codebooks | Split each group's columns into output-axis vectors and run weighted Lloyd k-means |
| First pass | Quantize successive input columns using nearest centroids and update unprocessed columns with the upper inverse-Hessian Cholesky factor |
| Residual codebooks | When enabled, learn main-group residual centroids from first-pass error and rerun the pass with main-plus-residual lookup |
| Reconstruction | Restore original input order, undo normalization, and round consistently with stored codebook/model dtype |
| Reporting | Return reconstructed replay weights, codebooks/indices/metadata, normalized Hessian-weighted loss, and stats |
_weighted_kmeans() seeds a device-local generator, initializes from a random permutation of training vectors, assigns nearest centroids in chunks, and updates centroids with weighted means. Empty clusters retain their previous centroids. The stopping condition compares inertia change to kmeans_tol * max(abs(previous_inertia), epsilon).
When a layer has fewer training vectors than requested centroids, initialization repeats vectors so the stored codebook still has exactly the configured number of entries. This preserves the layout contract; it does not reduce the codebook's storage cost. The algorithm is seeded, but that is not a guarantee of bitwise-identical floating arithmetic across devices.
Distance assignment sizes its chunks against a fixed 256 MiB workspace budget instead of constructing the complete vector-by-codebook distance matrix. Hessian-weighted loss evaluation sizes row chunks against a fixed 64 MiB budget. These are internal chunk targets, not total-memory limits: dense Hessians still scale quadratically with input width, and weights, codebooks, indices, and other temporaries consume additional memory.
_inverse_hessian_cholesky() adds percdamp * mean(diag(H)) to the diagonal before computing the upper Cholesky factor of the inverse. If the damped matrix is not positive definite, the error explicitly recommends increasing percdamp or supplying more calibration data. There is no adaptive damping retry loop here.
The reported loss is the ratio of summed Hessian-weighted squared reconstruction error to the corresponding weight energy, evaluated in FP64 with the absolute denominator clamped away from zero. It is not unnormalized scalar weight MSE, and it is not a model perplexity score.
The intermediate result is a dataclass, not the saved module itself:
| Field | Contents |
|---|---|
centroids, indices
|
Dictionaries for main/outlier centroid tensors and unpacked vector indices |
res_centroids, res_indices
|
Dictionaries for optional main residual tables and indices |
weight_scale, weight_bias
|
Optional normalization reversal tensors; None when disabled |
perm |
Optional input permutation; None when disabled |
outlier_size, group_size, group_num
|
Resolved layer-specific layout |
index_bits, res_index_bits
|
Main/residual widths per vector index |
stats |
Includes loss, sample count, duration, and fallback reporting as populated by the quantizer |
quantized_weight |
Temporary floating replay weight, not a persistent inference weight cache |
Dictionary key 0 is reserved for the outlier codebook when active; 1..group_num identify main groups.
| Stage | Behavior |
|---|---|
| Construction | Receives calibration data/tokenizer/preparation controls; requires calibration forwards |
| Dataset verification | Rejects missing calibration data; the processor does not support replacing its captured calibration dataset later |
| Preprocess | Honors a layer exclusion, deep-copies config, creates VPTQ, resolves fallback and expected token count |
| Forward hook | Passes captured input/output into add_batch()
|
| Process | Calls quantize(), releases task statistics, moves codebooks/indices/scales/permutation to CPU, logs duration/loss/sample counts, and installs reconstructed weights for replay |
| Conv1D replay | Transposes the quantized linear-layout weight back to Transformers Conv1D storage order |
| Submodule finalization | Creates VPTQLinear using qcfg.quant_linear_init_kwargs(), then calls pack(linear=..., vptq_result=result)
|
| Finalization | Marks model.quantized=True and sets method/format to VPTQ, then invokes shared finalization |
The original dense weight and intermediate result are released as the packed module takes ownership of codebooks and indices. See Looper Processors and Memory Offload and Multi-GPU.
VPTQ substitutes an identity Hessian when the captured matrix is unusable or the configured coverage threshold triggers. That keeps vector-codebook quantization available without inter-column Hessian compensation and avoids zeroing an entirely unvisited layer as if all its input channels were individually dead. The quantizer sets its fallback_used attribute and stores the value in result.stats["fallback"].
This is not scalar RTN, regardless of the shared fallback object's default strategy name. The VPTQ algorithm uses the threshold/coverage policy to choose Hessian-free vector quantization; it does not switch to QQQ/GPTQ scalar fallback math.
If fallback is disabled with fallback=None, the surrounding subset executor can remove never-invoked modules from quantization and record exclusions. Do not assume every MoE expert received the full dataset's token count. Review capture counts and exclusions when auditing coverage.
The inherited default fallback is enabled. Under that default, an unvisited module can still be packed using identity-Hessian vector quantization; pruning uncovered modules is specifically the disabled-fallback policy, not an unconditional behavior.
Main indices use a contiguous little-endian bitstream stored in signed torch.int32 words. Residual indices share the same stream:
merged = (residual_index << index_bits) | main_index
words_per_row = ceil(group_size * (index_bits + res_bits) / 32)
Indices may cross word boundaries. Signed int32 word values are storage containers for unsigned bit patterns; a negative packed word is not a negative centroid index. The layout matches the reference VPTQ bitstream, but bitstream compatibility alone is not full model-checkpoint compatibility.
The helpers are defined in gptqmodel/utils/vptq.py:
def vptq_index_bits(num_centroids: int) -> int:
def vptq_packed_words(num_elements: int, total_bits: int) -> int:
def vptq_group_setting(
in_features: int,
*,
npercent: float,
group_size: int,
group_num: int,
enable_outlier: bool = True,
) -> Tuple[int, int, int]:
def pack_vptq_indices(
indices: torch.Tensor,
index_bits: int,
*,
res_indices: Optional[torch.Tensor] = None,
res_bits: int = 0,
) -> torch.Tensor:
def unpack_vptq_indices(
packed: torch.Tensor,
index_bits: int,
num_elements: int,
*,
res_bits: int = 0,
) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:| Helper | Contract |
|---|---|
vptq_index_bits |
Computes ceil(log2(num_centroids)); count must be positive |
vptq_packed_words |
Computes the int32 word count; element count nonnegative and total bits positive |
vptq_group_setting |
Returns (outlier_size, resolved_group_size, resolved_group_num)
|
vptq_outlier_enabled |
Checks positive percentage/vector length and more than one outlier centroid |
vptq_group_for_column |
Returns 0 for outlier columns and 1-based main group ids thereafter |
pack_vptq_indices |
Validates index ranges/shapes, optional residual width, and total width no greater than 32 |
unpack_vptq_indices |
Reconstructs requested main indices and optional residual indices from packed words |
For example, the default 12-bit main and 4-bit residual indices round-trip as follows:
import torch
from gptqmodel.utils.vptq import pack_vptq_indices, unpack_vptq_indices
main = torch.tensor([[0, 17, 4095]], dtype=torch.int64)
residual = torch.tensor([[1, 2, 15]], dtype=torch.int64)
packed = pack_vptq_indices(main, 12, res_indices=residual, res_bits=4)
decoded_main, decoded_residual = unpack_vptq_indices(
packed, 12, num_elements=3, res_bits=4
)
assert torch.equal(main, decoded_main)
assert torch.equal(residual, decoded_residual)Let G be resolved main group count, S resolved group size, N output width, L main vector length, C main centroid count, and R enabled residual centroid count. I = ceil(N / L) and P = ceil(S * (index_bits + res_bits) / 32).
| State | Shape | Storage |
|---|---|---|
centroids.weight |
(G, C * L) |
Floating, frozen embedding parameter; viewed as (G, C, L) during reconstruction |
res_centroids.weight |
(G, R * L) when enabled |
Floating, frozen embedding parameter |
indices |
(G, I, P) |
Packed torch.int32 main/residual indices |
outlier_centroids.weight |
(1, outlier_centroids * outlier_vector_len) when enabled |
Floating, frozen embedding parameter |
outlier_indices |
(1, ceil(N / outlier_vector_len), outlier_size) |
Unpacked torch.int32 indices; not a 16-bit buffer |
weight_scale, weight_bias
|
(in_features,) for norm_dim=0, (out_features,) for norm_dim=1
|
Floating normalization metadata when enabled |
perm |
(in_features,) |
Persistent torch.int32 permutation when enabled |
inverse_perm |
(in_features,) |
Non-persistent torch.int32, rebuilt from perm
|
bias |
(out_features,) when present |
Floating linear bias |
Padding of output vectors and the last input group is removed during dequantization. Outlier indices use int32 to avoid wrapping codebook entries above the signed-int16 range. Codebooks and normalization metadata are stored in the module's codebook dtype, not universally FP32.
VPTQLinear.pack() requires a quantization result; it does not learn centroids merely from the dense linear argument:
def pack(self, linear: nn.Module = None, *, vptq_result=None, **kwargs):
def load_quant_result(self, result, bias: Optional[torch.Tensor] = None) -> None:
def dequantize(self) -> torch.Tensor:
def forward(self, x: torch.Tensor) -> torch.Tensor:Packing rejects missing group codebooks, required residual codebooks/indices, and a missing required permutation. post_init() calls _validate_loaded_state(), which checks the packed indices shape and rebuilds the inverse permutation with argsort. It does not comprehensively validate every loaded tensor shape, index range, or permutation bijectivity; externally assembled checkpoints must satisfy the full layout contract rather than relying on this check to detect every malformed tensor.
For developers constructing VPTQLinear directly, its low-level defaults are not the VPTQConfig recipe: the constructor defaults to vector_lens=(8, 8), num_centroids=(4096, 4096), num_res_centroids=(-1, -1), and kmeans_mode="". The model lifecycle passes qcfg.quant_linear_init_kwargs() explicitly; do the same instead of assuming the default config and default runtime constructor select identical codebooks. The constructor uses FP16 codebooks unless dtype is one of FP16/BF16/FP32.
| Property |
VPTQLinear / VPTQ_TORCH
|
VPTQCudaLinear / VPTQ_CUDA
|
|---|---|---|
| Method/format | VPTQ / VPTQ | Same |
| Nominal labels | 1 through 8 | Same |
| Packed dtype | torch.int32 |
Same |
| Floating dtypes | FP16, BF16, FP32 | Same declared contract |
| Devices | All Torch device types, subject to available operations | CUDA |
| Platforms | All declared | Inherits same declaration; requires usable CUDA extension/toolchain |
| Feature divisibility | 1; automatic padding supported | Same layout/padding support |
| Shards | Supported | Supported |
| Training | Not supported | Not supported |
| Adapters | Lora |
Lora |
Torch inference unpacks indices, gathers main/residual/outlier centroids, removes padding, restores input order and normalization, then calls F.linear using the input dtype/device. Bias is converted to the activation dtype if needed; an adapter is applied afterward. Empty inputs return an appropriately shaped empty linear output.
optimize() compiles the dequantization graph, not its dense output. It chooses inductor by default except on MPS (aot_eager), and skips compilation on NPU. Every forward still reconstructs weights from the live packed indices/codebooks; no persistent dense weight cache is retained. This portable reference path can be costly for large matrices and very small batches because it reconstructs a dense weight for the current operation.
VPTQCudaLinear.validate_once() rejects ROCm with an actionable NotImplementedError pointing at VPTQ_TORCH: the extension is built from CUDA sources and CUDA-only intrinsics that hipcc cannot compile, and because ROCm exposes the CUDA device namespace, torch.cuda.is_available() alone cannot be the gate. AUTO on AMD GPUs therefore selects the portable VPTQ_TORCH.
VPTQCudaLinear subclasses the Torch layer and reuses its state/packing. Its optimize() marks the layer optimized without compiling the portable decoder, because native inference does not use that graph.
The dispatch counts flattened activation rows, so [batch, sequence, hidden] contributes batch * sequence rows:
| Condition | Route |
|---|---|
| Zero rows | Empty-output fast path |
| Non-CUDA activations | Runtime error recommending VPTQ_TORCH
|
| Activations and main VPTQ buffers on different CUDA devices | Runtime error |
| At most 4 rows and activation dtype equals centroid dtype | Fused vptq_linear native kernel; input is made contiguous |
| More than 4 rows, or activation/codebook dtype mismatch | Native vptq_dequantize, convert reconstructed weight to input dtype, then F.linear/CUDA GEMM |
The four-row threshold is _VPTQ_CUDA_FUSED_MAX_ROWS = 4; it applies across vector lengths and outlier configurations in the Python dispatcher, not just the vector-length-4 optimized native case. A dtype mismatch selects the reconstruction route instead of the fused route. Optional bias and adapter application are preserved.
The fused route reads packed indices/codebooks without first materializing a persistent full weight. The larger-row route temporarily reconstructs a dense weight once for that forward so GEMM can reuse it across rows. It does not cache the dense result between calls. Do not describe the backend as fused for every batch size or as uniformly avoiding temporary dense allocations.
VPTQ_CUDA is a CUDA backend, not a declared ROCm backend. On other accelerators use VPTQ_TORCH where the required Torch operations are available. The VPTQ Python contract does not specify a QQQ/Marlin-style minimum sm_80 capability check; a working target-compatible build of the native extension is still required.
gptqmodel/utils/vptq_cuda.py defines TorchOpsJitExtension named gptqmodel_vptq_ops, namespace gptqmodel_vptq, with required operations vptq_dequantize and vptq_linear. Its native sources are gptqmodel_ext/vptq/vptq.cpp and vptq_kernel.cu. The central registry exposes vptq with vptq_cuda as an alias.
def clear_vptq_extension_cache() -> None:
def vptq_cuda_runtime_available() -> bool:
def vptq_cuda_runtime_error() -> str:
def prewarm_vptq_extension() -> bool:| Function/control | Behavior |
|---|---|
vptq_cuda_runtime_available() |
Returns False without an available CUDA device; otherwise asks the extension registry |
vptq_cuda_runtime_error() |
Returns a no-CUDA message, empty string when available, or the registry/runtime diagnostic |
prewarm_vptq_extension() |
Returns False without CUDA; otherwise loads registry extension vptq and returns its status |
clear_vptq_extension_cache() |
Clears the extension helper's cache |
GPTQMODEL_VPTQ_BUILD_ROOT |
Overrides build location |
GPTQMODEL_VPTQ_FORCE_REBUILD |
Force-rebuild control |
GPTQMODEL_EXT_VERBOSE |
Verbose extension build output |
The operation wrappers vptq_cuda_dequantize() and vptq_cuda_linear() forward packed state plus keyword-only layout dimensions to the central registry. A representative source signature is:
def vptq_cuda_linear(
x,
indices,
centroids,
res_centroids,
outlier_indices,
outlier_centroids,
inverse_perm,
weight_scale,
weight_bias,
bias,
*,
group_size: int,
in_features: int,
out_features: int,
vector_len: int,
outlier_size: int,
outlier_vector_len: int,
num_centroids: int,
num_res_centroids: int,
index_bits: int,
res_bits: int,
norm_dim: int,
):These are internal kernel wrappers, not codebook-learning APIs. Native validation checks CUDA placement/contiguity, int32 packed indices, consistent optional tensor pairs, dimension/index widths, and matching activation/codebook dtype for the fused op. Prefer the quantized module rather than manually assembling calls.
The repository's dependency baseline is Python >=3.10, PyTorch >=2.14.1, and Transformers >=5.20.2. Portable VPTQ does not require the native CUDA extension; CUDA inference additionally requires an appropriate compiler/toolkit, Ninja, and a successfully built/loaded extension. Refer to Installation and Quickstart for the supported installation procedure.
| Concern | Current behavior |
|---|---|
| Torch versus CUDA | Same FORMAT.VPTQ; choose the runtime at load time |
| Configuration persistence | VPTQ serializes its vector/centroid/group/k-means/normalization/permutation/block/damping fields so the layer can reconstruct the same layout |
| Generic GPTQ loaders | Not compatible: these are codebooks and indices, not GPTQ scalar qweight/qzeros/scales
|
| External VPTQ checkpoints | A compatible per-row index bitstream alone does not establish complete checkpoint compatibility; use the layout and metadata expected by GPT-QModel |
| Stock Transformers/vLLM/SGLang | The repository guide does not expose these as supported VPTQ checkpoint runtimes; load through GPT-QModel's VPTQ backends |
| Mid-quantization checkpoint/resume | Not available for the VPTQ processor chain; it is absent from the supported checkpoint adapter chains |
| Dynamic mixed recipes | Nonempty per-layer overrides rejected; configure codebooks globally or use supported layer exclusion |
| Training | Not supported by either VPTQ kernel contract; adapter application at inference is separate |
The implementation has no separate persistent Hessian-file requirement or external VPTQ calibration phase. Normal model saving after a completed quantization is supported; that is different from resuming an interrupted quantization run.
docs/vptq.md is the repository-local companion guide. It covers the same algorithm/configuration, recipes, module restrictions, backend dispatch, and checkpoint limitations, and documents two benchmark entry points:
| Script | Purpose |
|---|---|
scripts/benchmark_vptq_cuda.py |
Compare dispatch, direct fused native inference, native reconstruction plus GEMM, and portable Torch on identical layer state/activations |
scripts/benchmark_vptq.py |
Model-level dense/VPTQ/GPTQ comparison, including perplexity and optional ARC/speed measurements |
tests/test_vptq.py |
Independent quantization/index-layout oracles and runtime coverage |
For a CUDA checkpoint/backend comparison:
python scripts/benchmark_vptq_cuda.py /path/to/checkpoint --dtype float16For seeded synthetic packed layers without a model download:
python scripts/benchmark_vptq_cuda.py \
--synthetic \
--shapes 2048x2048,4096x4096 \
--batch-sizes 1,8,32,128 \
--dtype bfloat16 \
--output ./vptq-backend-bf16.jsonThe script reports synchronized GPU-event timing and wall-clock latency, including median/minimum/p90 by layer and row count. Run FP16 and BF16 separately when comparing those deployment dtypes; retain the same codebook/activation state for backend comparisons. Synthetic performance is not a model-quality result. For model quality and baseline methodology, see Evaluation.
| Problem | Explanation/action |
|---|---|
Setting only bits does not change compression |
Choose vector length and centroid counts; bits is derived |
| Actual checkpoint is larger than the nominal rate | Include centroid tables, outlier indices, padding, normalization, and bias in storage accounting |
group_num plus group_size fails |
Set only one; leave the other -1
|
| Outliers unexpectedly consume extra channels | Explicit group count with enabled outliers moves remainder columns into the outlier group |
| Nondivisible width creates a remainder | Without outliers, the final main group is padded and trimmed; no implicit outlier codebook is invented |
Packing reports a missing codebook despite group_num <= in_features
|
Explicit group count without outliers can leave an empty trailing group after width rounding; use positive group_size or satisfy the nonempty-group invariant
|
| Outlier residual configuration fails | Keep num_res_centroids[0] = -1
|
nn.Conv1d/nn.Conv2d fails |
Only linear and Transformers Conv1D module semantics are supported |
| Cholesky factorization fails | Increase VPTQ percdamp or improve calibration coverage; inherited GPTQ damping controls are not its retry mechanism |
| Unvisited/rare MoE expert | Inspect counts, fallback stats, and exclusions; Hessian-free vector fallback differs from scalar RTN |
| CUDA path unexpectedly calls GEMM | Expected for more than four flattened rows or mismatched activation/codebook dtypes |
| CUDA device mismatch | Keep all packed state/metadata and activations on the same CUDA device |
| Portable inference uses substantial temporary memory | It reconstructs a dense weight for the current forward; packed checkpoint size is not peak execution memory |
| Codebook count exceeds training-vector count | Duplicate initial centroids preserve configured table size; overhead can dominate small layers |
| Attempting interrupted-run resume | VPTQ does not implement the quantization checkpoint adapter chain |
The behavior above follows the implementation at eb90dbdd. Key source paths:
| Path | Responsibility |
|---|---|
gptqmodel/quantization/config.py |
VPTQConfig, derived bit labels, method/format values, validation and persistence |
gptqmodel/quantization/vptq.py |
VPTQ, VPTQQuantResult, Hessian collection, weighted k-means, error compensation, loss |
gptqmodel/looper/vptq_processor.py |
Calibration, quantization/replay, packed-module replacement |
gptqmodel/looper/stage_subset.py |
Calibration coverage and uncovered-module policy |
gptqmodel/nn_modules/qlinear/vptq_torch.py |
VPTQLinear, codebook/index state, pack/load/dequantize/Torch inference |
gptqmodel/nn_modules/qlinear/vptq_cuda.py |
VPTQCudaLinear, small-row fused/native-GEMM dispatch |
gptqmodel/utils/vptq.py |
Index packing/unpacking, bit counts, input grouping and outlier helpers |
gptqmodel/utils/vptq_cuda.py |
Extension definition, diagnostics, native operation wrappers |
gptqmodel_ext/vptq/vptq.cpp, vptq_kernel.cu, vptq.h
|
Native operation registration, validation, dequantization, and linear kernels |
gptqmodel/extension.py |
Extension registry and vptq_cuda alias |
gptqmodel/utils/backend.py, gptqmodel/utils/importer.py
|
Backend names, alias normalization, kernel selection |
gptqmodel/models/base.py |
VPTQ processor dispatch and quantization entry point |
gptqmodel/looper/gptq_checkpoint.py |
Supported mid-quantization checkpoint processor chains |
docs/vptq.md |
Repository guide and recipes |
scripts/benchmark_vptq.py, scripts/benchmark_vptq_cuda.py
|
Quality and backend benchmarks |
tests/test_vptq.py |
Reference/oracle and integration coverage |
pyproject.toml, requirements.txt
|
Dependency baseline |