Skip to content

Kernel Backends Reference

Qubitium edited this page Nov 21, 2026 · 97 revisions

Kernel backends reference 🔩

🏠 Home · 🧭 Choose a method · 🚀 Quickstart · 🧰 Help

On this page

Runtime tuning below uses KernelConfig: pass kernel=... through the quantization/load API. Kernel-policy fields are not environment variables; build/cache and diagnostic controls remain separate.

Find the exact class, packed layout, and runtime conditions behind a backend name. This reference is for integrators and kernel developers; if you only want to load a model, begin with Inference Backends.

🧭 Choose and understand

Read a kernel’s contract as a conjunction: method + checkpoint format + dimensions + quantization controls + activation dtype + device + available implementation. Matching one column never establishes the rest.

Task Choose this when Consider another path when
Find the class behind a backend You need its layout owner and declared support. You just need a first model load; use Inference Backends.
Diagnose a rejected layer You have its shape, group size, format, dtype, and device. You only know its nominal bit width. Inspect checkpoint metadata first.
Pack or repack a layer You own the raw-weight or packed-source lifecycle and can preserve metadata. You expect the helper to calibrate or discover a full model.
Implement or change a kernel You can check independent numerics, storage lifetime, and exact-device behavior. A declaration/import is your only evidence of correctness.

Before you begin: distinguish raw packing, checkpoint loading, runtime repacking, and forward execution. They can be supported independently. NVIDIA CUDA and AMD ROCm declarations remain separate; Triton is an implementation tool, not a hardware support guarantee.

Reading level Start here
Choose / understand Storage concepts, backend/class/layout map
Use it First compatibility probe, packing utilities
Go deeper Exact contracts, the kernel-family sections, and implementation validation

See Dtypes and Formats before comparing integer codes, FP8, GGUF qtypes, and lossless BF16 encoding as if they were interchangeable weight representations.

Guide source: init.py, packing.py.

Overview

This page describes the quantized linear implementations, packing boundaries, support declarations, and native extension sources in GPT-QModel. Use Inference Backends to choose a backend; use this reference to understand exactly which class executes a layer, what it stores, and why validation can reject a seemingly compatible checkpoint.

The detailed inventory retains public source citations at commit 2e7992a18. Optional dependencies, installed Torch operators, target architecture, tensor qtype, and per-layer dimensions remain part of the executable contract. Declaration tables are not a promise that every declared combination works without runtime validation.

🚀 Use it

🔎 First compatibility probe

This layer-level API demo checks raw-weight packing feasibility for a concrete GPTQ INT4 contract on NVIDIA CUDA. It does not load a model, execute a kernel, benchmark it, or prove that an existing checkpoint can be repacked.

from gptqmodel.utils.packing import hf_check_packing_feasibility

can_pack = hf_check_packing_feasibility(
    bits=4,
    group_size=128,
    desc_act=False,
    sym=True,
    format="gptq_v2",
    quant_method="gptq",
    device="cuda:0",  # NVIDIA CUDA example; preserve the target ordinal.
    in_features=4096,
    out_features=4096,
)
print(can_pack)

False means this contract has no feasible raw packer; invalid arguments/enums can raise ValueError. For AMD ROCm, use a target that resolves to the ROCm family and verify a consumer that explicitly declares DEVICE.ROCM. Do not treat this CUDA example as a cross-vendor qualification. When starting from packed tensors instead of dense weights, inspect the helper’s repack=True path and the source layout restrictions in Packing Formats.

For model loading, use GPTQModel.load(..., backend=BACKEND.AUTO) as shown in Inference Backends; a layer probe is not a full-model loader. Probe source.

Packing utilities and extension packers

Stable layer APIs

def hf_pack_layer(
    module: BaseQuantLinear,
    linear: nn.Module,
    scales: torch.Tensor,
    zeros: Optional[torch.Tensor] = None,
    g_idx: Optional[torch.Tensor] = None,
    *,
    pack_impl: Optional[str] = None,
    device=None,
    block_in: int = 8192,
    workers: int = 1,
    scales_extra: Optional[torch.Tensor] = None,
    post_init: bool = False,
    post_init_kwargs: Optional[Dict[str, Any]] = None,
    checkpoint_format: Optional[Union[str, FORMAT]] = None,
) -> BaseQuantLinear:
    ...

The scales/zero inputs use quantizer-native [out_features, num_groups] layouts. For symmetric modules, omitted zeros can be synthesized; asymmetric modules require zeros. pack_impl=None means original packing; auto prefers GPU packing where implemented, then block, then original. QQQ/AWQ single-packer branches ignore this implementation choice. GPU packing returns CPU-registered packed buffers; final module placement is separate.

def hf_repack_layer(
    module: BaseQuantLinear,
    backend: Optional[Union[str, BACKEND]] = None,
    *,
    device=None,
    dtype: Optional[torch.dtype] = None,
    post_init: bool = True,
    post_init_kwargs: Optional[Dict[str, Any]] = None,
) -> BaseQuantLinear:
    ...

Repacking selects the destination kernel by source method/format, then uses a native repack hook or copies compatible checkpoint tensors and performs destination initialization. Sources already converted to certain private runtime ABIs are rejected. An initialized runtime matrix is not necessarily a valid checkpoint-layout source.

hf_check_packing_feasibility(), hf_check_best_packing_format(), and hf_post_init() support integration checks and initialization. hf_select_quant_linear and its v2 wrapper are in utils/importer.py; converter.py currently has an empty MODULE_CONVERTER_MAP and is not a packing dispatcher.

Planar packing

utils/planar_packing.py implements GPTQ_P split-plane packing. It requires int32 words and aligned groups of 32 logical codes for its row/column helpers.

Logical bits Plane widths, low to high
2 2
3 2 + 1
4 4
5 4 + 1
6 4 + 2
7 4 + 2 + 1
8 8

Each plane width divides 32; a field never straddles a word boundary. For every 32 codes, the plane words are adjacent in low-to-high order. Single-plane 2/4/8-bit layouts are bit-identical to continuous packing; planar 3-bit is not the same as the continuous cross-word 3-bit layout. The helpers are planar_pack_rows, planar_unpack_rows, planar_pack_cols, and planar_unpack_cols.

CPU block packing

qlinear/pack_block_ext.py::pack_block_cpu loads the block-packing extension and calls torch.ops.gptqmodel.pack_block_cpu with weight, scales, zeros, g_idx, bit width, word width, input block size, and worker count. Its native source is gptqmodel_ext/pack_block_cpu.cpp. This wrapper raises if the extension cannot load; higher-level pack selection determines whether an alternative is available.

Workflow at a glance

Packed-layer state lifecycle. Raw packing, repacking, and state changes have different initialization needs.

Reading the diagram: Independent entry routes. Initialization is kernel-specific; scratch is not universal. Dense or expanded weight caches may not survive their consuming operation.

Text version of the workflow
  • Raw packing: Quantizer outputs → hf_pack_layer → Initialize if required → Layer forward.
  • Existing packed source: Checkpoint-layout layer → hf_repack_layer → Initialize target → Layer forward.
  • Changed packed state: State changes → Invalidate owned state → Reinitialize as needed → Next forward.

Open full-size SVG

🔬 Go deeper

Reference area Jump to
Compilation / CUDA graphs Capability declarations
Fused/native integer kernels Marlin, Machete, Swordfish, ExLlama
Portable / compiler-backed paths Torch, Triton, BitBLAS
Other representations GGUF, FP8/FP4, NF12, MACaQ
Alternative execution stacks MLX, Tenstorrent
Build / lifecycle / correctness Native source map, runtime state, validation checklist

Compile and CUDA-graph capabilities

Quantized kernels declare compilation and eager graph-capture capabilities separately from the static-cache generation policy:

Declaration Meaning Base default
SUPPORTS_COMPILE_LEVEL Highest declared torch.compile level for the kernel's forward in its default runtime state on its primary device. COMPILE_LEVEL.EAGER
SUPPORTS_COMPILE_LEVEL_BY_DEVICE Optional override keyed by torch.device.type, e.g. "cpu". Applies only if declared on that class, never via inheritance or for NOT_APPLICABLE kernels. None
SUPPORTS_GRAPH_CAPTURE Whether its eager forward can run inside explicit torch.cuda.graph(...) capture. False
SUPPORTS_COMPILE Whether full-graph tracing is clean and compiled decoding has measured faster on the claimed hardware. Native static-cache generation uses this policy. False
SUPPORTS_COMPILE_EAGER_REASON Reason to keep static-cache generation eager for a kernel that declares INDUCTOR or higher. None

COMPILE_LEVEL is an ordered enum in gptqmodel.nn_modules.qlinear._compile_support:

Level Contract
NOT_APPLICABLE (-1) MLX, TTNN, or XLA owns execution through its own compiler/runtime.
EAGER (0) Eager forward is the declared level; no full-graph compile capability is claimed.
DYNAMO (1) torch.compile(fullgraph=True, backend="eager") traces without graph breaks.
AOT_EAGER (2) Adds backend="aot_eager" with fake/meta tensors and AOT Autograd.
INDUCTOR (3) Adds lowering through backend="inductor".
CUDAGRAPH (4) Adds compiled CUDA-graph capture through mode="reduce-overhead".

Each higher compile level includes the lower levels. Eager graph capture is independent. For example, TorchLinear declares INDUCTOR on CPU and EAGER by default for its CUDA streaming path, with eager CUDA capture enabled. TorchFP8Linear, AwqTorchLinear, BitsAndBytesLinear, MacaqLinear, QQQTorchLinear, and VPTQLinear opt into compiled generation. A kernel that reaches INDUCTOR on CPU or by default can still leave automatic static-cache compilation off, with a recorded reason. See static-cache generation for model-level behavior and overrides.

To inspect an already loaded native quantized model without guessing from backend names:

from gptqmodel.nn_modules.qlinear import BaseQuantLinear
from gptqmodel.nn_modules.qlinear._compile_support import describe_kernel

native_model = getattr(model.model, "_orig_mod", model.model)
for name, layer in native_model.named_modules():
    if isinstance(layer, BaseQuantLinear):
        print(name, describe_kernel(layer))

describe_kernel() returns kernel, module, compile_level, compile_level_cpu, graph_capture, compiled_generation, and compiled_generation_reason. Use compile_level(kernel, device_type="cpu") to query the CPU declaration explicitly. If no device is given, a loaded instance uses its packed-buffer device when available; a class uses its default declaration. A level is not a device support check or a guarantee for every dtype, shape, Torch version, or full-model graph.

For a loaded GGUFTorchLinear or GGUFTritonKernel, graph_capture_safe(layer) checks the stored qtype against that kernel's own GRAPH_CAPTURE_QTYPES through supports_graph_capture(). Passing the class returns its general declaration. An instance hook can only narrow a class declaration of True. The ggml-backed GGUFCppKernel and GGUFCudaKernel declare False. The Torch GGUF CPU Inductor measurement is separate; it does not extend the CUDA-capture qtype list. See capture and warmup for Torch GGUF and Triton GGUF.

The complete kernel matrix lists class-specific declarations. The protocol defines their meaning; registry tests check explicit declarations and matrix consistency, and the test_kernel_compile_levels_*.py suites measure constructed instances on hardware. See generation policy and explicit overrides before enabling compiled decoding.

Current declarations include these measured cases:

Kernel family Compile level Eager graph capture Scope of the recorded measurement
Native Marlin, ExLlamaV2, AllSpark (GPTQ/AWQ), AWQ GEMM/GEMV/GEMV_FAST/LLM_AWQ, Paro CUDA, VPTQ CUDA EAGER True sm_80: eager capture/replay matched eager output; custom-op or tracing limitations still block higher compile levels.
BitsAndBytesLinear CUDAGRAPH True sm_89, sm_120, gfx1201. Permits automatic static-cache compilation.
MacaqLinear CUDAGRAPH False MACAQ_TORCH permits automatic static-cache compilation. The eager forward copies a table from host to device.
QQQTorchLinear CUDAGRAPH False Portable QQQ opts into static-cache compilation; native QQQ and MLX QQQ do not.
VPTQLinear CUDAGRAPH True Portable VPTQ opts into static-cache compilation; VPTQCudaLinear remains opted out even though it supports eager graph capture.
ParoTritonLinear CUDAGRAPH True Non-identity rotation measured in a full-graph Inductor trace and CUDA-graph execution on RTX 5090. Automatic static-cache compilation stays off. Paro runtime
GGUFTritonKernel EAGER True On RTX 5090, replay matched eager bitwise with same and changed inputs for Q1_0, Q1_0_g128, Q4_K, Q5_K, and Q6_K, including wide-scale Q6_K. Automatic static-cache compilation stays off.
AwqTorchLinear CUDAGRAPH True Eager graph capture and replay measured on sm_89 with Torch 2.15.0+cu132. The CPU compile level is INDUCTOR.
AwqGEMVFastTorchLinear, LLMAwqTorchLinear INDUCTOR False CPU full-graph probes on Torch 2.16.0.dev20261021+cpu; automatic static-cache compilation stays off.

These declarations do not certify every shape or device. In particular, FP8 eager capture depends on a native-FP8 execution path; sm_80's host-round-trip fallback is not capturable.

The following CPU paths declare INDUCTOR and were probed with fullgraph=True. Reaching that level alone does not opt a kernel into automatic static-cache compilation.

Kernel CPU path
TorchCpuAvx512Linear, TorchCpuAvx512AwqLinear Inductor calls the native packed AVX512-BF16 INT4 GEMM when the hardware and layer qualify.
NF12Linear post_init() resolves the exponent base before tracing. The compiled decoder handles escape rows with a fixed-shape gather.
ParoLinear Non-identity rotations are traceable. optimize() or an eager warmup prepares the eligible AVX512-BF16 or ATen packed layout; direct cold tracing may need a graph break. AVX512 and ATen can both trace with bias when the prepared layout is eligible. The packed ATen path needs the same FP16/BF16 activation dtype as the prepared layout and adds bias before narrowing the FP32 product. Automatic static-cache compilation stays off.
QQQTorchLinear Eager warmup prepares the packed QQQ operands and native kernel. If that cache is absent when tracing, the portable INT8 path uses torch._int_mm.
VPTQLinear Portable codebook lookup and reconstruction are traceable; the CPU declaration is separate from native CUDA VPTQ.
TorchQuantEmbeddings Tracing supports whole-table lookup and eligible row decoding with metadata from an eager forward. Without the row metadata, tracing dequantizes the table for the current lookup.

See the CPU layer probes and NF12 escape-row decoder test.

After a CPU inference warmup, compiled calls that need an activation gradient on TorchFusedLinear, TorchAtenLinear, TorchFusedAwqLinear, or TorchAtenAwqLinear use a differentiable matmul instead of the no-grad int4pack kernel. The trace decodes a temporary from the live runtime layout without restoring or mutating checkpoint buffers. GPTQ applies its activation order to the input when needed. Subsequent eligible no-grad calls can still use the fused kernel. See the compiled CPU int4pack fallback.

Packed storage and execution concepts

Term Meaning
Checkpoint layout Serialization of quantized codes and metadata. GPTQ, AWQ GEMM, QQQ, VPTQ, GGUF, and EXL3 do not share one universal packed layout.
Runtime ABI A kernel-private representation created after loading: tiled/prepacked matrices, scale permutations, native handles, or specialized INT4 operator inputs.
Group size -1 Channelwise grouping represented by the class's full input dimension rather than a fixed finite group length.
desc_act / g_idx Activation-order grouping and input-to-group mapping. Some kernels reorder operands; others reject activation ordering.
pack_dtype Integer word storage dtype, not the logical quantization bit width or activation dtype.
SUPPORTS_FORMATS priority Positive values enter AUTO ordering; zero retains explicit support without generic AUTO selection. auto_select_priority(format, dtype) can lower the effective rank for a requested dtype.
post_init() Runtime initialization after packed checkpoint buffers have been populated; not a substitute for loading missing weights.
Raw packing Encoding quantizer outputs from an original layer; requires a pack API and is distinct from loading/repacking existing codes.

For standard aligned GPTQ INT4 with int32 words, input width I, output width O, and G groups, qweight is [I/8, O], qzeros is [G, O/8], scales is [G, O], and g_idx is [I]. These are checkpoint-side shapes; a runtime-transformed class may no longer expose that ABI after initialization. GPTQ v1/v2 zero-point interpretation must be preserved when converting; a raw copy of qzeros is not always a semantic conversion.

Backend → class → layout → device

Paths in this table are relative to gptqmodel/nn_modules/qlinear/. Portable classes enumerate their device families; an ALL declaration, where present, does not prove that every runtime implements every required operation. Device-sensitive validators can narrow declarations. MLX rows are checkpoint holders, not directly executable Torch layers. EXL3 classes live outside qlinear and are listed separately afterward. The Tenstorrent kernel table adds XLA/TTNN and the native block-float family.

CUDA in a support declaration means NVIDIA; AMD must be declared as ROCM. The selector normalizes HIP's cuda spellings to DEVICE.ROCM, keeps concrete target ordinals, and translates back to Torch devices only for execution. CUDA availability and HIP's synthetic SM tuples cannot admit a CUDA-only kernel. Thus TritonV2Linear is not a ROCm GPTQ backend; use TorchLinear and its independently gated ROCm fast paths. AllSpark's ROCm validator checks gfx1200/gfx1201 on every requested target (all visible GPUs for an indexless family selector), and warms up on the selected GPU.

These are layer-level contracts, not a model-loader allowlist. For example, AwqMacheteLinear declares Marlin-format support, but the local model loader requires the matching Marlin backend or MLX for a serialized Marlin checkpoint. The same boundary applies to serialized BitBLAS checkpoints; see format-specific loader branches.

Native AWQ/QQQ device metadata also does not establish that CUDA/PTX extension sources run on ROCm. The AWQ and QQQ pages distinguish declarations from the native implementation's portability constraints.

BACKEND Class and qlinear source Checkpoint / runtime packing relationship Declared device families
ALLSPARK AllSparkLinear
allspark.py
GPTQ/GPTQ_V2 W8 → device-specific W8A16 packed layout CUDA, ROCM
ALLSPARK AllSparkAwqLinear
allspark_awq.py
AWQ GEMM W8 → device-specific W8A16 packed layout with zeros CUDA, ROCM
GPTQ_BITBLAS BitblasLinear
bitblas.py
GPTQ/BitBLAS → BitBLAS transformed weight/scales/zeros CUDA
AWQ_BITBLAS AWQBitBlasKernel
bitblas_awq.py
AWQ GEMM/BitBLAS → BitBLAS transformed weight/scales/zeros CUDA
BITSANDBYTES BitsAndBytesLinear
bitsandbytes.py
bitsandbytes codes plus QuantState / INT8 state CPU, CUDA, ROCM
GPTQ_EXLLAMA_V2 ExllamaV2Linear
exllamav2.py
GPTQ packed buffers → extension matrix handle CUDA
AWQ_EXLLAMA_V2 AwqExllamaV2Linear
exllamav2_awq.py
AWQ GEMM → ExLlama v2-compatible packed buffers CUDA
FP8_TORCH TorchFP8Linear
fp8.py
FP8 weight codes plus scales CPU, CUDA, ROCM, XPU, MPS
AWQ_GEMM AwqGEMMLinear
gemm_awq.py
AWQ GEMM packed qweight/qzeros/scales CUDA
AWQ_GEMM_TRITON AwqGEMMTritonLinear
gemm_awq_triton.py
AWQ GEMM packed qweight/qzeros/scales CUDA, ROCM
AWQ_GEMV AwqGEMVLinear
gemv_awq.py
AWQ GEMV layout CUDA
AWQ_GEMV_TRITON AwqGEMVTritonLinear
gemv_awq.py
Same AWQ GEMV layout; explicit-only CUDA, ROCM
AWQ_TORCH AwqGEMVTorchLinear
gemv_awq.py
Portable GEMV reader; CPU-only AUTO fallback CPU
AWQ_GEMV_FAST AwqGEMVFastLinear
gemv_fast_awq.py
GEMV_FAST or LLM_AWQ layout, according to class CUDA
AWQ_GEMV_FAST LLMAwqLinear
gemv_fast_awq.py
GEMV_FAST or LLM_AWQ layout, according to class CUDA
AWQ_TORCH AwqGEMVFastTorchLinear
gemv_fast_awq.py
Portable GEMV_FAST reader and AUTO fallback CPU, CUDA, ROCM
AWQ_TORCH LLMAwqTorchLinear
gemv_fast_awq.py
Portable LLM_AWQ reader and AUTO fallback CPU, CUDA, ROCM
GGUF_TORCH GGUFTorchLinear
gguf.py
Native GGUF block bytes and qtype CPU, CUDA, ROCM, XPU, NPU, MPS
GGUF_CPP_CPU GGUFCppKernel
gguf_cpp.py
Native GGUF block bytes; ggml bridge plans CPU
GGUF_CPP_CUDA GGUFCudaKernel
gguf_cpp.py
Native GGUF block bytes; ggml bridge plans CUDA
GGUF_TRITON GGUFTritonKernel
gguf_triton.py
Native GGUF block bytes and qtype CUDA, ROCM
MACAQ_TORCH MacaqLinear
macaq.py
Native MACaQ MXFP4 with optional transforms ALL
MACAQ_TRITON MacaqTritonLinear
macaq.py
Packed MXFP4 Torch-compatible checkpoint CUDA, ROCM (gfx1201 gate)
MLX MacaqMlxQuantLinear / NF12MlxQuantLinear
mlx.py
Native packed checkpoint holders → MLX Metal modules MPS
MACHETE MacheteLinear
machete.py
GPTQ → Machete prepacked B CUDA
MACHETE AwqMacheteLinear
machete_awq.py
AWQ/Marlin → Machete prepacked B CUDA
MARLIN MarlinLinear
marlin.py
GPTQ → Marlin tiles; permuted scales and activation-order metadata CUDA
MARLIN AwqMarlinLinear
marlin_awq.py
AWQ/Marlin → Marlin tiles; scale/zero permutations CUDA
MLX AWQBitBLASMlxQuantLinear
mlx.py
Torch checkpoint holder → method-specific MLX quantized module MPS
MLX AWQMarlinMlxQuantLinear
mlx.py
Torch checkpoint holder → method-specific MLX quantized module MPS
MLX AwqGemvFastMlxQuantLinear
mlx.py
Torch checkpoint holder → method-specific MLX quantized module MPS
MLX AwqGemvMlxQuantLinear
mlx.py
Torch checkpoint holder → method-specific MLX quantized module MPS
MLX AwqMlxQuantLinear
mlx.py
Torch checkpoint holder → method-specific MLX quantized module MPS
MLX BitsAndBytesMlxQuantLinear
mlx.py
Torch checkpoint holder → method-specific MLX quantized module MPS
MLX GGUFMlxQuantLinear
mlx.py
Torch checkpoint holder → method-specific MLX quantized module MPS
MLX GPTQBitBLASMlxQuantLinear
mlx.py
Torch checkpoint holder → method-specific MLX quantized module MPS
MLX GPTQMarlinMlxQuantLinear
mlx.py
Torch checkpoint holder → method-specific MLX quantized module MPS
MLX LLMAwqMlxQuantLinear
mlx.py
Torch checkpoint holder → method-specific MLX quantized module MPS
MLX MlxQuantLinear
mlx.py
Torch checkpoint holder → method-specific MLX quantized module MPS
MLX ParoMlxQuantLinear
mlx.py
Torch checkpoint holder → method-specific MLX quantized module MPS
MLX QQQMlxQuantLinear
mlx.py
Torch checkpoint holder → method-specific MLX quantized module MPS
PARO_CUDA ParoLinear
paro.py
Packed affine weights plus rotation/scaling metadata CPU, CUDA, ROCM, XPU, NPU, MPS
PARO_TRITON ParoTritonLinear
paro_triton.py
Paro packed affine weights plus rotation metadata CUDA, ROCM
QQQ QQQLinear
qqq.py
QQQ packed B plus group/channel scales CUDA
QQQ_TORCH QQQTorchLinear
qqq.py
QQQ packed B plus group/channel scales CPU, CUDA, ROCM, XPU, NPU, MPS
SWORDFISH AwqSwordfishLinear
swordfish.py
GPTQ or AWQ → Swordfish prepacked B CUDA
SWORDFISH SwordfishLinear
swordfish.py
GPTQ or AWQ → Swordfish prepacked B CUDA
GPTQ_TORCH TorchLinear
torch.py
GPTQ continuous or GPTQ_P planar words CPU, CUDA, ROCM, XPU, NPU, MPS
GPTQ_TORCH TorchQuantEmbeddings
torch.py
GPTQ continuous or GPTQ_P planar words CPU, CUDA, ROCM, XPU, NPU, MPS
GPTQ_TORCH_ATEN TorchAtenLinear
torch_aten_kernel.py
GPTQ → CPU ATen int4pack + affine scale/zero offsets CPU
AWQ_TORCH_ATEN TorchAtenAwqLinear
torch_aten_kernel_awq.py
AWQ → CPU ATen int4pack + affine scale/zero offsets CPU
AWQ_TORCH AwqTorchLinear
torch_awq.py
AWQ GEMM words CPU, CUDA, ROCM, XPU, NPU, MPS
GPTQ_TORCH_CPU_AVX512 TorchCpuAvx512Linear
torch_cpu_avx512.py
GPTQ INT4 → native CPU packed layout CPU
AWQ_TORCH_CPU_AVX512 TorchCpuAvx512AwqLinear
torch_cpu_avx512_awq.py
AWQ INT4 → native CPU packed layout CPU
GPTQ_TORCH_FUSED TorchFusedLinear
torch_fused.py
Checkpoint GPTQ → Torch fused INT4 ABI CPU, XPU
AWQ_TORCH_FUSED TorchFusedAwqLinear
torch_fused_awq.py
AWQ → Torch fused INT4 ABI CPU, XPU
GPTQ_TORCH_INT8 TorchInt8Linear
torch_int8.py
Native 8-bit GPTQ → centered INT8 operands preserving each GPTQ group's scales/offsets CPU
AWQ_TORCH_INT8 TorchInt8AwqLinear
torch_int8_awq.py
Retired; no executable runtime layout CPU
GPTQ_TRITON TritonV2Linear
tritonv2.py
GPTQ packed qweight/qzeros/scales/g_idx CUDA
VPTQ_CUDA VPTQCudaLinear
vptq_cuda.py
Packed vector indices and codebooks CUDA
VPTQ_TORCH VPTQLinear
vptq_torch.py
Packed vector indices, codebooks and optional residual/outlier state CPU, CUDA, ROCM, XPU, NPU, MPS
NF12 NF12Linear
nf12.py
nf12_packed, nf12_escapes, nf12_exp_base, optional bias; method/format string nf12 CPU, CUDA, ROCM, XPU, NPU, MPS; native BF16 CUDA / RDNA4 HIP paths, otherwise Torch codec + matmul
CPTC CPTCLinear
cptc.py
cptc_packed, cptc_table, cptc_coeffs, cptc_scale_bits, optional bias CPU, CUDA, ROCM; fused FP16/BF16 for 1–16 rows on SM80+ or gfx1200/gfx1201, otherwise tiled decode and FP32 matmul. CPTC dispatch
GPTQ_W4AFP8 W4AFP8Linear
w4a_floatx.py
Saved GPTQ INT4 plus runtime E4M3 operand CUDA
GPTQ_W4A_NVFP4 W4ANVFP4Linear
w4a_nvfp4.py
Saved GPTQ INT4 plus runtime FP4/FP8 operands CUDA

Method-specific classes outside the generic qlinear registry

Backend Class / source Packed state Execution boundary
EXL3_EXLLAMA_V3 ExllamaV3Linear, gptqmodel/nn_modules/exllamav3.py trellis, suh, svh, su, sv, optional bias and a codebook marker (mcg, mul1, q4nx, or r4nx) Native LinearEXL3 wrapper on NVIDIA CUDA or eligible ROCm gfx1200/gfx1201. R4NX reconstructs weights for HGEMM. On ROCm, AUTO selects it only when all selected devices qualify and the extension passes the availability probe. Sign vectors suh/svh stay FP16 across model-wide casts.
EXL3_TORCH ExllamaV3TorchLinear, gptqmodel/nn_modules/exllamav3_torch.py EXL3 trellis/codebook and transform metadata Torch decoding/matmul implementation, including chunked execution paths.
Native FP4 weight helper, no separate BACKEND member TorchFP4Linear, qlinear/fp4.py Packed NVFP4 weight, weight scale, block size, original dtype CPU/NVIDIA CUDA declaration only; TorchAO NVFP4Tensor on supported native paths, CPU decodes current weights for F.linear. No validated ROCm route. Not the same as GPTQ W4A NVFP4.

Exact declared quantization contracts

This table transcribes class declarations, resolving inherited constants where applicable. Format names are FORMAT members; numbers after them are priorities. It deliberately separates bit/group/training declarations from the architecture and operator checks in subsequent sections. For example, a GGUF logical bit width does not identify the complete GGML block qtype, and a class declaring training support may use a different forward path in training mode.

Class Format: AUTO priority Bits Group sizes desc_act sym Training
AllSparkLinear GPTQ: 0, GPTQ_V2: 0 8 -1 False True False
AllSparkAwqLinear GEMM: 0 8 -1 True, False True, False False
BitblasLinear BITBLAS: 30, GPTQ: 30, GPTQ_V2: 30 2, 4, 8 -1, 32, 64, 128 False False, True False
AWQBitBlasKernel GEMM: 0, BITBLAS: 30 4 -1, 32, 64, 128 False, True False, True False
BitsAndBytesLinear BITSANDBYTES: 40 4, 8 — — — False
ExllamaV2Linear GPTQ: 80, GPTQ_V2: 80 4 -1, 32, 64, 128 True, False True, False False
AwqExllamaV2Linear GEMM: 80 4 -1, 32, 64, 128 True, False True, False False
TorchFP8Linear FP8: 15 8 — — — True
NF12Linear NF12: 100 12 — — — False
CPTCLinear CPTC: 100 1–6 -1 False True False
MacaqLinear / MacaqTritonLinear MXFP4_MACAQ, MXFP4: 100 4 — — — False
MacaqMlxQuantLinear MXFP4_MACAQ, MXFP4: 0 4 — — — False
NF12MlxQuantLinear NF12: 0 12 — — — False
AwqGEMMLinear GEMM: 60 4 -1, 32, 64, 128 True, False True, False True
AwqGEMMTritonLinear GEMM: 50 4 -1, 32, 64, 128 True, False True, False True
AwqGEMVLinear GEMV: 40 4 -1, 64, 128 True, False True, False False
AwqGEMVTritonLinear GEMV: 0 4 -1, 64, 128 True, False True, False True
AwqGEMVTorchLinear GEMV: 0 (CPU fallback) 4 -1, 64, 128 True, False True, False False
AwqGEMVFastLinear GEMV_FAST: 30 4 128 True, False True, False False
LLMAwqLinear LLM_AWQ: 100 4 128 True, False True, False False
AwqGEMVFastTorchLinear GEMV_FAST: 0 (portable fallback) 4 128 True, False True, False False
LLMAwqTorchLinear LLM_AWQ: 0 (portable fallback) 4 128 True, False True, False False
GGUFTorchLinear GGUF: 15 1, 2, 3, 4, 5, 6, 8 — — — True
GGUFCppKernel GGUF: 25 1, 2, 3, 4, 5, 6, 8 — — — False
GGUFCudaKernel GGUF: 35 1, 2, 3, 4, 5, 6, 8 — — — False
GGUFTritonKernel GGUF: 45 1, 4, 5, 6 — — — False
MacheteLinear GPTQ: 100 4, 8 -1, 64, 128 True, False True, False False
AwqMacheteLinear GEMM: 100, MARLIN: 100 4, 8 -1, 64, 128 False True, False False
MarlinLinear GPTQ: 90, GPTQ_V2: 90, MARLIN: 90 4, 8 -1, 32, 64, 128 True, False True, False (asymmetric only without effective activation order) False
AwqMarlinLinear GEMM: 90, MARLIN: 90 4, 8 -1, 32, 64, 128 True, False True, False False
AWQBitBLASMlxQuantLinear BITBLAS: 0 2, 3, 4, 5, 6, 7, 8 -1, 32, 64, 128 True, False True, False False
AWQMarlinMlxQuantLinear MARLIN: 0 4, 8 -1, 32, 64, 128 True, False True, False False
AwqGemvFastMlxQuantLinear GEMV_FAST: 0 4 -1, 16, 32, 64, 128 True, False True, False False
AwqGemvMlxQuantLinear GEMV: 0 4 -1, 64, 128 True, False True, False False
AwqMlxQuantLinear GEMM: 0 2, 3, 4, 5, 6, 7, 8 -1, 16, 32, 64, 128 False True, False False
BitsAndBytesMlxQuantLinear BITSANDBYTES: 0 4, 8 — — — False
GGUFMlxQuantLinear GGUF: 0 1, 2, 4, 5, 6, 8 — — — False
GPTQBitBLASMlxQuantLinear BITBLAS: 0 2, 4, 8 -1, 32, 64, 128 False True, False False
GPTQMarlinMlxQuantLinear MARLIN: 0 4, 8 -1, 32, 64, 128 False True False
LLMAwqMlxQuantLinear LLM_AWQ: 0 4 -1, 16, 32, 64, 128 True, False True, False False
MlxQuantLinear GPTQ: 0, GPTQ_V2: 0, GPTQ_P: 0 2, 3, 4, 5, 6, 7, 8 -1, 16, 32, 64, 128, 256, 512, 1024 False True, False False
ParoMlxQuantLinear PARO: 0 4 -1, 16, 32, 64, 128 True, False True False
QQQMlxQuantLinear QQQ: 0 4 -1, 128 True, False True False
ParoLinear PARO: 55 (0 on ROCm) 4 -1, 16, 32, 64, 128 True, False True True
ParoTritonLinear PARO: 10 on ROCm, 0 on CUDA 4 -1, 16, 32, 64, 128 True, False True True
QQQLinear QQQ: 100 4 -1, 128 False True False
QQQTorchLinear QQQ: 90 4 -1, 128 False True False
AwqSwordfishLinear GEMM: 101 4 -1, 32, 64, 128 False True, False False
SwordfishLinear GPTQ: 101 4, 8 -1, 32, 64, 128 True, False True, False False
TorchLinear GPTQ: 20, GPTQ_V2: 20, GPTQ_P: 20 2, 3, 4, 5, 6, 7, 8 -1, 16, 32, 64, 128, 256, 512, 1024 True, False True, False True
TorchQuantEmbeddings GPTQ: 20, GPTQ_V2: 20, GPTQ_P: 20 2, 3, 4, 5, 6, 7, 8 -1, 16, 32, 64, 128, 256, 512, 1024 True, False True, False False
TorchAtenLinear GPTQ: 110, GPTQ_V2: 110 4 32, 64, 128, 256 True, False True, False True
TorchAtenAwqLinear GEMM: 110 4 32, 64, 128, 256 True, False True, False True
AwqTorchLinear GEMM: 10 2, 3, 4, 5, 6, 7, 8 -1, 16, 32, 64, 128 True, False True, False True
TorchCpuAvx512Linear GPTQ: 120, GPTQ_V2: 120 4 16, 32, 64, 128, 256, 512 True, False True, False True
TorchCpuAvx512AwqLinear GEMM: 120 4 16, 32, 64, 128, 256, 512 True, False True, False True
TorchFusedLinear GPTQ: 50, GPTQ_V2: 50 4 -1, 16, 32, 64, 128 True, False True, False True
TorchFusedAwqLinear GEMM: 20 4 -1, 16, 32, 64, 128 True, False True, False True
TorchInt8Linear GPTQ: 0, GPTQ_V2: 0 8 -1, 16, 32, 64, 128 True, False True, False False
TorchInt8AwqLinear GEMM: 0 4 -1, 16, 32, 64, 128 True, False True, False False
TritonV2Linear GPTQ: 40, GPTQ_V2: 40 2, 4, 8 -1, 16, 32, 64, 128, 256, 512, 1024 True, False True, False True
VPTQCudaLinear VPTQ: 110 1, 2, 3, 4, 5, 6, 7, 8 — — — False
VPTQLinear VPTQ: 100 1, 2, 3, 4, 5, 6, 7, 8 — — — False
W4AFP8Linear GPTQ: 0, GPTQ_V2: 0 4 128 False True False
W4ANVFP4Linear GPTQ: 0, GPTQ_V2: 0 4 128 False True False

The base implementation also validates platform, activation dtype, packing dtype, adapters, automatic padding, and input/output divisibility. AUTO passes the adapter into candidate validation; the explicit-backend selector branch currently does not, so explicit selection alone is not proof of adapter compatibility. Sharded-loading support uses SUPPORTS_SHARDED_LOAD when present, otherwise SUPPORTS_SHARDS. Swordfish explicitly disables sharded loading. Inspect these additional fields when adding a format or diagnosing a shape that passes the bit/group columns but fails selection.

Trainable kernels and device declarations

The training support contract lists the classes that declare SUPPORTS_TRAINING=True. The devices below are SUPPORTS_DEVICES declarations, not proof of every dtype, shape, operator, or adapter combination. CUDA here means NVIDIA, not ROCm.

Method Trainable classes Declared devices
GPTQ TorchLinear CPU, CUDA, ROCm, XPU, NPU, MPS
GPTQ TritonV2Linear CUDA
GPTQ TorchFusedLinear CPU, XPU
GPTQ TorchAtenLinear, TorchCpuAvx512Linear CPU
AWQ AwqTorchLinear CPU, CUDA, ROCm, XPU, NPU, MPS
AWQ AwqGEMMLinear CUDA
AWQ AwqGEMMTritonLinear, AwqGEMVTritonLinear CUDA, ROCm
AWQ TorchFusedAwqLinear CPU, XPU
AWQ TorchAtenAwqLinear, TorchCpuAvx512AwqLinear CPU
FP8 TorchFP8Linear CPU, CUDA, ROCm, XPU, MPS
GGUF GGUFTorchLinear CPU, CUDA, ROCm, XPU, NPU, MPS
Paro ParoLinear CPU, CUDA, ROCm, XPU, NPU, MPS
Paro ParoTritonLinear CUDA, ROCm

The native CUDA and portable CPU AWQ GEMV readers are inference-only, unlike Triton GEMV. An eval-mode gradient through a fallback does not make a kernel trainable. Use backend selection to choose a compatible training path.

Operand dtypes, dimension declarations, and sharding

AllSpark's GPTQ and AWQ classes both declare FP16/BF16 activations, int32 packing, K and N divisible by 16, no automatic padding, no sharding, and no adapters. GPTQ declares Linux/Windows; AWQ declares Linux. Both have priority zero (explicit selection only); AllSparkLinear.SUPPORTS_RAW_PACKING=False distinguishes loading/repacking from raw quantization. CUDA requires capability 8.0–12.0 inclusive; ROCm uses a separate HIP extension restricted to gfx1200/gfx1201. Its packed layout pads N to 128, unlike CUDA's N32K16 order, so the runtime layouts are not interchangeable. Select BACKEND.ALLSPARK; method/format choose the class. See Inference Backends for the full contract.

AllSpark's CUDA JIT linker resolves cuBLAS to an absolute toolkit/wheel library path, accepting a versioned file such as libcublas.so.13 when no development symlink exists. If resolution fails it retains -lcublas. This fixes linking on CUDA-wheel hosts without adding a compiler or changing the hardware/layout contract.

The following are declarations, including inherited values. A dash means no concrete declaration was found for that field, not unrestricted support. Automatic padding and specialized post-initialization can impose further constraints or adapt the checkpoint dimensions. Private Torch operations can have a narrower actual compute dtype than the supported interface, as documented in the family sections. A declared sharding flag does not override method-specific loader restrictions.

Class Activation dtypes Pack word dtypes Input divisibility Output divisibility Auto padding Sharded load
AllSparkLinear float16, bfloat16 int32 16 16 False False
AllSparkAwqLinear float16, bfloat16 int32 16 16 False False
BitblasLinear float16, bfloat16 int32 16 16 False True
AWQBitBlasKernel float16, bfloat16 int32 16 16 False True
BitsAndBytesLinear float16, bfloat16, float32 int8, int16, int32, int64 1 1 False True
ExllamaV2Linear float16, bfloat16, float32 int32 32 32 False True
AwqExllamaV2Linear float16, bfloat16, float32 int32 32 32 False True
TorchFP8Linear float16, bfloat16, float32 int8, int16, int32, int64 1 1 False True
NF12Linear float16, bfloat16, float32 int8, int16, int32, int64 64 16 False True
CPTCLinear float16, bfloat16, float32 int8, int16, int32, int64 1 1 False True
AwqGEMMLinear float16, bfloat16 int32 1 64 False True
AwqGEMMTritonLinear float16 int32 32 8 False True
AwqGEMVLinear float16 int32 1 64 False True
AwqGEMVTritonLinear float16 int32 1 8 False True
AwqGEMVTorchLinear float16 int32 1 64 False True
AwqGEMVFastLinear float16 int16 64 128 False True
LLMAwqLinear float16 int16 64 128 False True
GGUFTorchLinear float16, bfloat16, float32 int8, int16, int32 1 1 True True
GGUFCppKernel float16, bfloat16, float32 int8, int16, int32 1 1 True True
GGUFCudaKernel float16, bfloat16, float32 int8, int16, int32 1 1 True True
GGUFTritonKernel float16, bfloat16, float32 int8, int16, int32 1 1 True True
MacheteLinear float16, bfloat16 int32 64 128 False True
AwqMacheteLinear float16, bfloat16 int32 64 128 False True
MarlinLinear float16, bfloat16 int32 1 1 False True
AwqMarlinLinear float16, bfloat16 int32 1 1 False True
AWQBitBLASMlxQuantLinear float16 int32 16 16 False True
AWQMarlinMlxQuantLinear float16 int32 8 8 False True
AwqGemvFastMlxQuantLinear float16 int16 64 8 False True
AwqGemvMlxQuantLinear float16 int32 32 64 False True
AwqMlxQuantLinear float16 int32 8 8 False True
BitsAndBytesMlxQuantLinear float16 int8, int16, int32, int64 1 1 False True
GGUFMlxQuantLinear float16 int8, int16, int32 32 8 False True
GPTQBitBLASMlxQuantLinear float16 int32 32 32 False True
GPTQMarlinMlxQuantLinear float16 int32 32 32 False True
LLMAwqMlxQuantLinear float16 int16 64 8 False True
MlxQuantLinear float16 int32 32 32 False True
ParoMlxQuantLinear float16 int32 32 8 False True
QQQMlxQuantLinear float16 int32 64 64 False True
ParoLinear float16, bfloat16 int32 1 1 False True
ParoTritonLinear float16, bfloat16 int32 1 1 False True
QQQLinear float16 int32 64 64 False True
QQQTorchLinear float16, bfloat16 int32 64 64 False True
AwqSwordfishLinear float16, bfloat16 int32 64 64 False False
SwordfishLinear float16, bfloat16 int32 64 64 False False
TorchLinear float16, bfloat16 int8, int16, int32 1 1 True True
TorchQuantEmbeddings float16, bfloat16 int8, int16, int32 1 1 True True
TorchAtenLinear float16, bfloat16 int32 1 16 True True
TorchAtenAwqLinear float16, bfloat16 int32 1 16 True True
AwqTorchLinear float16, bfloat16 int32 1 8 False True
TorchCpuAvx512Linear bfloat16 int32 2 32 False True
TorchCpuAvx512AwqLinear bfloat16 int32 2 32 False True
TorchFusedLinear float32, float16, bfloat16 int32 1 8 True True
TorchFusedAwqLinear float32, float16, bfloat16 int32 1 8 True True
TorchInt8Linear float16, bfloat16, float32 int32 1 1 True True
TorchInt8AwqLinear float16, bfloat16, float32 int32 1 1 False True
TritonV2Linear float16, bfloat16 int32, int16, int8 32 32 True True
VPTQCudaLinear float16, bfloat16, float32 int32 1 1 True True
VPTQLinear float16, bfloat16, float32 int32 1 1 True True
W4AFP8Linear float16, bfloat16 int32 128 32 False True
W4ANVFP4Linear float16, bfloat16 int32 128 32 False True

Marlin and caller-owned scratch

qlinear/marlin.py and marlin_awq.py adapt GPTQ/AWQ tensors to Marlin's tile layout. utils/marlin.py owns extension access, device checks, padding helpers, repacking, scale/zero permutations, workspace allocation, and GEMM wrappers. Both Marlin classes require NVIDIA CUDA compute capability 8.0 or newer. Input/output packing divisibility and activation-order alignment are validated even when broader declarations permit padding.

GPTQ Marlin uses KernelConfig.marlin_fp32 with default True. Disabling it changes reduction precision; this is not a lossless memory-only toggle. The low-level helper defaults and the module's configured value must not be confused: the module forwards its own reduction choice to the GEMM wrapper.

Native CUDA GPTQ Marlin supports asymmetric (sym=False) 4- and 8-bit weights for group sizes -1, 32, 64, and 128, subject to normal validation and without effective activation order. A single group makes desc_act=True a no-op. For an ordinary GPTQ checkpoint with grouped activation order, AUTO tries the next compatible backend; explicit MARLIN rejects it. If an asymmetric checkpoint has a noncanonical g_idx despite desc_act=False, Marlin rejects it during repacking rather than silently ignoring the index. See asymmetric Marlin selection.

gptqmodel.utils.marlin.marlin_device_profile(device=None, dtype=torch.float16) returns the cached exact-device profile used by launch selection: integer fields sms, max_shared_mem, capability_major, capability_minor, l2_cache_size, max_threads_per_sm, and has_tensor_memory. Both FP16 and BF16 extension libraries expose it. An omitted/index-less CUDA device means the current device; explicit ordinal zero remains zero, and raw integer/string ordinals are checked before Torch can narrow/wrap them. CPU, ROCm and out-of-range devices fail rather than reporting another GPU's profile. Tensor-memory support and narrow split-K architecture decisions come from this profile, not a blanket Blackwell assumption.

Negative integer ordinals and strings such as "cuda:-1" raise Marlin's explicit out-of-range ValueError before torch.device() parsing. See the device-profile validation flow.

Marlin post_init repacks weights into the runtime layout. Native loader selection rejects Marlin for CPU-offload maps because initialization requires CUDA-resident tensors. GPTQ Triton and AWQ GEMM Triton declare SUPPORTS_CPU_OFFLOAD=True, which describes initialization for an offloaded layer. Under AUTO the Hugging Face bridge applies the check to CPU and disk placements, including maps resolved from device_map="auto". When explicitly selected through the bridge, GPTQ and AWQ Marlin can repack and refresh a writable in-memory AlignDevicesHook weights map. See Hugging Face integration.

Both MarlinLinear and AwqMarlinLinear keep non-quantized bias in its checkpoint dtype and order: MODEL_DTYPE_BUFFERS=() exempts it from model-dtype narrowing. post_init records padding without rewriting that buffer. Each forward derives a temporary FP32, padded, Marlin-permuted bias operand; the native kernel adds it to the FP32 accumulator before the visible output cast. Repeated FP16/BF16 calls leave the stored bias unchanged, preserving finite cancellation when an FP32 bias such as -65536 would overflow FP16. Fused Marlin sources contribute individually padded bias slices before concatenation, so later launches do not double-pad. This bias contract does not make repacked runtime weights portable checkpoints; existing export restrictions still apply. See bias storage and epilogue.

Marlin's launch memory accounting now includes the full FP32 bias stage and act-order index bytes. The bias stage sits after the shared B/reduction region, so split-K reduction cannot overwrite it before the epilogue. This fixes incorrect shared-memory sizing and biased reductions without adding a kernel option or changing the packed format.

GPTQ and AWQ Marlin forwards use a separate zeroed split-K lock workspace for each CUDA stream that executes the layer. Each stream reuses its own counters after the kernel resets them; overlapping streams no longer borrow one shared lock array, which could cause hangs or incorrect reductions. The registered workspace remains available for checkpoint/fusion bookkeeping. This automatic lock isolation is separate from the optional scratch context below and does not establish general concurrent-forward support for every mutable layer state. Workspace selection source.

When a GPTQ or AWQ quantization group spans multiple Marlin pipeline stages (for example, group_size=128 with a K tile of 64), the first stage loads its scales and any zero points from shared memory. Later stages reuse those values from registers. Earlier kernels could read a refilled slot during GPU time-slicing and silently return incorrect output blocks. The fix is in the FP16 and BF16 runtime kernels and works with existing checkpoints. See the Marlin kernel.

The low-level gptq_marlin_gemm also accepts packed MXFP8 (E4M3) and MXFP4 (E2M1) weight operands with torch.float8_e8m0fnu scales per 32 input values. Both paths require BF16 activations and read the scale bytes in pairs of adjacent groups. For MXFP8, gptqmodel.utils.marlin.mxfp8_marlin_prepare(weight, scales) converts a torch.float8_e4m3fn weight of shape (N, K) and an E8M0 scale tensor of shape (N, K // 32) into the packed GEMM operands. The interleave requires an even number of groups along K. These are low-level CUDA kernel entry points. The BACKEND.MARLIN loader selects kernels for GPTQ and AWQ checkpoints. See the packing helper and CUDA dispatch.

For automatic tile selection, Marlin can use a narrower tile (K=128, N=64, 128 threads) to increase output parallelism. The switch requires at most one output tile per four SMs under the original schedule and a valid compiled kernel for the narrower tile. On GPUs with at most 128 SMs, the narrow tile must also split K across at least two blocks per output tile, or at least four for a 16-row tile with K greater than 4,096. Otherwise the wider tile stays in use. The full-SM sm_80 and Ada narrow FP32 AWQ schedules retain their tuned selection. See Marlin thread-tile selection.

Marlin uses a separate hybrid schedule for launches with 64-row tiles when the full GEMM has at least 512 flattened activation rows. When enough output tiles are available, each block processes whole tiles across K before splitting the remaining tiles across blocks. This avoids cross-block reduction for the whole tiles. Smaller calls keep the original split-K schedule. The choice is automatic and does not change the checkpoint format. See Marlin's hybrid work schedule.

Scratch API

gptq_marlin_gemm accepts caller-provided c_tmp and a_tmp; without them the native implementation retains its normal allocation behavior. marlin_scratch_sizes() returns element counts, not bytes. The query and GEMM share the native size calculation, including executed/padded K, activation-order permutation requirements, and reduction mode.

def __init__(
    self,
    device: torch.device | str | int,
    *,
    max_cached_bytes: int = 64 << 20,
) -> None:
    ...
import torch
from gptqmodel.utils.marlin_scratch import MarlinScratchContext

with torch.cuda.device("cuda:0"):
    with MarlinScratchContext("cuda:0", max_cached_bytes=64 << 20):
        # model must already be loaded with a compatible Marlin kernel.
        result = model(**inputs)
Scratch property Contract
Ownership One CUDA device, host thread, and CUDA stream; no nested/reentrant contexts or simultaneous leases.
Retained budget Default 64 MiB total including the lock workspace; oversized requests use temporary storage. This is not a bound on outputs or allocator-reserved memory.
Buffers Growing reduction and activation buffers plus a separate zero-initialized lock workspace.
Supplied native scratch Checked for device, dtype, contiguity, capacity, 16-byte alignment, and shared storage with GEMM operands/other scratch. Invalid buffers fail rather than silently falling back.
CUDA graphs The eager context rejects capture. A low-level graph caller must preallocate sufficient scratch and locks and retain exclusive ownership for the whole graph lifetime.
Lifetime No scratch is saved in module checkpoints. Leaving a with scope deactivates the context but does not clear a context object kept by the caller; bounded buffers can be reused on later compatible entries. clear() releases retained ownership with stream-lifetime tracking, without synchronizing.

The feature is opt-in and does not claim an established latency improvement or concurrent-forward support. docs/marlin_scratch.md explains capacity formulas and graph ownership in detail.

Machete

MacheteLinear and AwqMacheteLinear use CUTLASS-based prepacked matrices. GPTQ consumes GPTQ format; AWQ declares GEMM and MARLIN inputs. utils/machete.py manages extension generation/loading, prepacking, schedules, and supported quantization types.

Both group-size declarations are -1, 64, and 128. AWQ group size 32 is excluded before launch because native Machete requires K-group alignment to 64; it must use another compatible backend.

The runtime is Hopper-only, capability exactly 9.0, with at least 204800 bytes of opt-in shared memory per block. Its sm_90a build is not forward-compatible evidence for Blackwell. The helper pins CUTLASS 4.7.1, validates the archive checksum/version, and caches generated sources. An import/JIT cache is separate from per-device runtime validation.

Loading a supported checkpoint still requires satisfying the class's shape, scale dtype, group size, symmetry, and activation-order rules. Repacking to Machete is a layout transform, not a new quantization pass. Do not serialize the private runtime B layout as ordinary GPTQ qweight without a matching format contract.

Asymmetric Machete repacks keep logical zero points. The CUDA mainloop subtracts the zero point from the code before multiplying by the scale, so weights round once from (code - zero) * scale. Repacking rejects affine products outside the stored scale dtype's finite range before replacing the checkpoint layout. See Machete asymmetric dequantization.

For both GPTQ and AWQ, a biased Machete projection requests an FP32 result from the native matmul, adds the bias in FP32, then rounds once to the FP16 or BF16 activation dtype. This preserves cancellation when the matmul result alone exceeds the activation dtype's finite range. Bias-free projections use the native output in the activation dtype. GPTQ implementation · AWQ implementation

Swordfish

SwordfishLinear and AwqSwordfishLinear are in qlinear/swordfish.py; utils/swordfish.py exposes runtime checks, prepacking, quant-type queries, and swordfish_mm. Native sources are under gptqmodel_ext/swordfish/, including stable-libtorch bindings.

Swordfish's vendored code is licensed GNU AGPL v3.0+, as stated in its extension loader and gptqmodel_ext/swordfish/LICENSE / licenses/SWORDFISH. Review that dependency's license when distributing a build; it should not be inferred from another kernel's license.

Property Current implementation
Architectures Exact native build targets include sm_100a, sm_103a, sm_110a; runtime support is derived from architecture flags, including their PTX rules. Not a generic SM100+ predicate.
Torch requirement Stable ABI target Torch 2.10; older or unparseable versions fail validation.
Device/platform CUDA on Linux; ROCm rejected.
Shape Prepacked block shape (64, 64); both feature dimensions divisible by 64.
Groups -1, 32, 64, 128.
Activations FP16 or BF16.
Weight quant types With zero points: uint4; without zero points: uint4b8 or uint8b128. Zero-point INT8 is rejected.
Sharded load / training Both disabled by the classes.

GPTQ accepts 4/8-bit and AWQ accepts 4-bit. Generic declaration support is followed by shape, extension, quant-type, and exact-device checks. Swordfish's Blackwell targets must not be conflated with W4A NVFP4's separate capability 12.1 floor.

ExLlama v2 and v3

ExLlama v2

ExllamaV2Linear consumes 4-bit GPTQ/GPTQ_V2, with group sizes -1, 32, 64, or 128. Group size 16 is deliberately absent. Both feature dimensions must be divisible by 32. The class declares CUDA rather than ROCm and does not support training.

Both GPTQ and AWQ validate_once() reject ROCm with NotImplementedError before probing/building the native extension. Runtime-availability helpers and the central extension registry also gate exllamav2/exllamav2_awq before JIT loading: these CUDA sources cannot be built with hipcc, and ROCm support is disabled because of known accuracy regressions. Use BACKEND.AUTO for compatible selection, BACKEND.TORCH for GPTQ, or BACKEND.AWQ_TORCH for AWQ. This is not an implicit fallback when an explicit unsupported backend is requested.

post_init() prepares an extension matrix handle and FP16 scales, with scratch supplied by the loader. GPTQ and AWQ forward paths accept FP16/BF16/FP32 activations and preserve their dtype in the output instead of unconditionally narrowing inputs to FP16. The CUDA GEMM accumulates in FP32; scales remain FP16, so this is not an FP32 scale-storage format. AwqExllamaV2Linear first adapts AWQ GEMM tensors. Do not reuse an initialized native handle after replacing its underlying buffers without rebuilding runtime state.

If an initialized AWQ module receives replacement checkpoint-layout words, the handle builder recognizes their non-native qweight shape, discards the old descriptor/shuffle state, and repacks before rebinding. Clearing the handle alone does not make raw AWQ checkpoint words a native ExLlama layout. See the descriptor lifecycle.

The GPTQ handle cache records its source signature after native act-order construction has inserted/populated q_perm and q_invperm. Unchanged forwards therefore reuse the completed handle instead of treating the newly created permutation metadata as a buffer replacement on every call. Actual source changes still invalidate the handle.

The GPTQ/AWQ handle retains no dense-weight reconstruction scratch: temp_dq_size() returns zero, and any row-count fallback allocates its reconstructed weight only for the current call. Other forward workspace and temporary peak memory still matter.

For biased FP16 prefill above 50 flattened rows, the GPTQ and AWQ row-count fallback feeds FP16 activations and temporary FP16 decoded weights to cuBLAS, with FP32 accumulation and FP32 output. The wrapper adds bias before casting the output to FP16. This avoids making FP32 copies of both operands; it does not remove temporary dense reconstruction or change the non-FP16 path. See the GPTQ binding and AWQ binding.

Native v2 bindings and CUDA sources live under gptqmodel_ext/exllamav2/; utility setup is in gptqmodel/utils/exllamav2.py.

ExLlama v3 / EXL3

EXL3 is not GPTQ INT4 with a different GEMM. ExllamaV3Linear wraps the bundled EXL3 runtime and its trellis/transformation tensors. ExllamaV3TorchLinear provides a Torch implementation with its own codebook decoding and chunked matmul paths. They are loaded by the EXL3-specific branch rather than generic BaseQuantLinear discovery.

Native EXL3 inference also uses per-call writable scratch: split-K locks, the HGEMM cuBLAS workspace, and the batch-one Hadamard input buffer are no longer shared across streams or modules. The allocator can recycle them after the consuming work without a process-wide scratch cache. Quantization has a separate run-lifetime tile workspace: workers share one quantization stream per device and reuse its scratch pair for a fixed K/tile batch, instead of allocating a pair per worker. See EXL3 workspace and native scratch ownership.

The bundled Python runtime is under gptqmodel/exllamav3/; native code is under gptqmodel_ext/exllamav3/. On ROCm, AUTO selects native EXL3 only when all selected devices pass the gfx120x gate and the non-compiling extension probe; otherwise it uses EXL3_TORCH. On gfx1200/gfx1201, the native backend can use HIP GEMV and WMMA GEMM, with hipBLAS reconstruction when required. mlx_exl3.py implements a separate packed-trellis MLX path. See EXL3, including KernelConfig.exl3_tile_batch for per-device quantization scratch control.

The native EXL3 wrapper uses LinearEXL3 for FP16 and for BF16 on eligible NVIDIA CUDA or HIP devices once the module passes its initial finiteness probe. After warmup, eligible contiguous tensors use fused, on-device pre-scale and rescale launches, one per step, when the extension exposes those operators. Ineligible layouts or builds use eager scaling for the corresponding step. FP32 inputs and BF16 inputs that fail the probe use a bounded ExllamaV3TorchLinear helper borrowing the same packed buffers. Both routes restore the caller dtype. The helper is not a registered child module and adds no checkpoint keys. A per-module lock serializes handle publication and teardown; copies discard inherited handles. Native signatures track sign/codebook metadata edits, and clear_weight_cache() releases derived handles/helpers, including after unversioned inference-trellis edits. GGUF CUDA has separate ggml operand and bias-epilogue rules; do not infer its arithmetic from EXL3 dispatch. See EXL3 inference and GGUF native execution.

Triton v2 and Triton utilities

TritonV2Linear supports 2/4/8-bit GPTQ/GPTQ_V2 with CUDA declarations and both inference and training support. It consumes quantized words, scales, zeros, and group indexes directly through Triton kernels rather than maintaining a permanently dense weight matrix. The validator requires Triton 2.0.0+ and, when the GIL is disabled, 3.4.0+.

These are embedded validator thresholds, not the current packaging requirement: the triton extra in pyproject.toml requests Triton 3.8.0 or newer on Python below 3.15. See Inference Backends for the distinction between optional dependency metadata and runtime validation.

File under nn_modules/triton_utils/ Responsibility
kernels.py Packed 2/4/8-bit quantized matmul, transpose matmul, autograd wrappers, and inference-only wrapper.
dequant.py Packed-weight decoding, dtype mapping, and quantized matmul helper/autograd wrapper. A dequantization helper is not itself permission to retain expanded weights.
gptq_gemv.py Specialized packed GPTQ matrix-vector execution.
gptq_gemm.py Specialized small-M packed GPTQ matrix multiplication and shape limits.
dequant.py Triton dequantization used by the packed GPTQ paths.
custom_autotune.py CustomizedTritonAutoTuner, autotune decoration, and matmul configuration pruning.
mixin.py TritonModuleMixin; the current warmup(..., transpose=False, seqlen=2048) body is a no-op.

Triton helpers are also used by selected portable/ROCm paths. A helper's device context code does not expand the selectable devices of every class importing it. The declaration table and runtime validator jointly determine selection.

dequant.py::quant_matmul() routes both normal and transposed multiplication through matmul_fp32_reduce(). CUDA FP16/BF16 operands request FP32 reduction output before casting back; HIP uses its ordinary matmul path. This preserves activation output dtype without retaining an expanded weight cache or changing global cuBLAS settings.

In evaluation mode, when a decoded weight would overflow the activation dtype and the fused paths are unavailable, TritonV2Linear delegates to the range-safe Torch forward instead of materializing the whole weight in the activation dtype. The fallback keeps decode and reduction wide until the output cast; it does not guarantee that an out-of-range final result becomes finite. See Triton dispatch.

dequant.py::dequant() makes qweight, scales, qzeros, and g_idx contiguous before launching its dense-offset decoder. Strided packed buffers therefore decode by logical values rather than storage gaps; the copy does not change the packed representation or retain decoded weights. Other helpers have their own contracts; see the strided operand inventory.

On ROCm the fused GPTQ GEMV/GEMM/dequant kernels take their launch constants from a per-architecture table in gptqmodel/utils/arch.py rather than one module-level constant set: is_rocm_fused_arch() registers gfx950 (CDNA4) and the gfx120*/gfx125* RDNA4 family, gate_limits() returns (small_m_max, max_in_features) — (16, 5120) on CDNA4, (16, 17408) measured on gfx1201 — and gemm_fused_dtypes() gates the fused small-M GEMM to FP16 on RDNA4 while CDNA4 keeps FP16 and BF16. Ada (SM 8.9) now shares these Torch-backend fused paths: GEMV for a single flattened row and GEMM for 2–16 rows with K <= 2048 and N <= 2048, in FP16/BF16. max_out_features() supplies Ada's output-width cap. Both require the common INT32 2/4/8-bit, GPTQ-v2-zero, alignment, device, and no-gradient checks. The existing KernelConfig.torch_rocm_gemv/KernelConfig.torch_rocm_gemm controls also apply to Ada; other CUDA families and unregistered ROCm architectures keep the dequantize + matmul fallback.

GPTQ dequant/GEMV/GEMM, AWQ GEMV, and Paro GEMM have separate Ada/RDNA4 launch tables. AWQ GEMM uses exact device/shape/dtype scheduler rules; GGUF uses capability-specific choices on NVIDIA and generic/autotuned choices on HIP. This is measured dispatch coverage, not a guarantee of equal speed on every GPU. See Inference Backends.

Torch families

Activation layout and AWQ tile correctness

Updated native activation-consuming entry points pack strided inputs contiguously, select the activation tensor's device, use widened indexing for large offsets, and return correctly shaped empty results for M=0. This fixes views/permutations and non-current-device calls without requiring callers to change the public backend. AWQ split-K assigns every K tile even when the split does not divide K evenly; GEMV2 Triton bounds its K tile to the logical width rather than reading beyond a narrow matrix. See input contract and AWQ tile contract.

Portable GPTQ and AWQ

TorchLinear supports 2–8-bit GPTQ-family weights, including GPTQ_P planar packing. AwqTorchLinear is the portable AWQ GEMM implementation. TorchQuantEmbeddings is a GPTQ packed embedding implementation, not another linear GEMM backend: it sets SUPPORTS_BACKEND_SELECTION=False and is excluded from general selector discovery despite sharing the GPTQ_TORCH declaration. Its rows above describe the role-specific class, not an additional AUTO candidate. These classes provide format-aware decoding and fallback math; "Torch" does not mean weights are permanently stored as a normal dense nn.Linear.weight.

torch.py includes a metadata shim for integrations that inspect weight attributes, device-aware unpacking (including NPU shift handling), and optional Triton paths. Runtime-cache keys must include the owning packed tensor identities and versions; see _runtime_cache.py and Inference Backends.

Eligible Torch GPTQ fused GEMV and small-M GEMM on Ada and supported ROCm architectures can consume FP16 checkpoint scales with BF16 activations. The kernels form the decoded product in FP32 before rounding it to the activation dtype. Architecture, shape, dtype, and range gates still apply; when the dense fallback can safely decode into the activation dtype, it avoids an intermediate cast through the scale dtype. See BF16 fused GPTQ dispatch.

For CPU FP16/BF16 GPTQ calls with bias, portable TorchLinear uses a fused addmm epilogue with CPU autocast disabled. Matching low-precision operands use addmm directly; wider decoded weights or checkpoint bias promote the operation to FP32 before the final output cast. This prevents a separately rounded matmul from losing bias cancellation without changing packed checkpoint storage.

The eager and streaming GPTQ fallback paths also preserve a stored wide bias on CUDA and ROCm. When linear_needs_wide_bias() requires it, bias is added in FP32 before the output cast; autocast is disabled around the corrective wide computation. Streaming applies this per output tile without narrowing the bias first or retaining a dense weight. See the low-precision bias epilogue.

Quantized token embeddings

In eval mode, TorchQuantEmbeddings can decode only the requested token rows. On CPU it uses chunks of up to 8,192 IDs when the query is no larger than the vocabulary. On accelerators it uses row decode for at most 1024 IDs and an equivalent dense table of at least 64 MiB (in_features * out_features * scales.element_size()). Both paths require int32/int64 IDs on the packed weight's device and eligible packed metadata, including a valid g_idx entry for every vocabulary row. Continuous GPTQ/GPTQ_V2 packing supports 2/3/4/8 bits; GPTQ_P split-plane packing supports every integer width from 2 through 8. These are automatic dispatch thresholds, not config fields.

For continuous 2/4/8-bit packing, row lookup derives the word row and intra-word shift from the actual storage width: pack_factor = pack_dtype_bits // bits, word row token // pack_factor, and shift (token % pack_factor) * bits. This fixes incorrect row selection with INT8/INT16 packed words; it no longer assumes 32-bit storage. Continuous 3-bit packing retains its three-INT32-word layout per 32 tokens, and split-plane decoding is separate. No checkpoint conversion or threshold change is required.

Training, batches outside the applicable thresholds, and ineligible metadata use whole-table dequantization followed by F.embedding. Empty IDs return shape ids.shape + (out_features,). The output dtype (including for empty IDs) follows the embedding/model dtype when set, and falls back to the scale dtype otherwise. Whole-model casts update that output dtype without changing the checkpoint schema; choose the intended dtype before the first forward. Row decoding retains indexing and zero-point metadata only; decoded weights remain temporary. Checkpoint loads, device/dtype moves, and ordinary packed-tensor mutations invalidate derived metadata; explicitly clear after unversioned source edits. See the row-decode implementation.

Fused INT4

TorchFusedLinear / TorchFusedAwqLinear declare CPU and XPU, 4-bit weights, and training support. The fused operator ABI is not the same on CPU and XPU.

CPU selection validates resolved group sizes against 32/64/128/256 even though the shared declaration also includes XPU's -1 and 16; the class's declared group-size set must also admit the request. A per-channel -1 resolves to in_features and is CPU-valid only at one of those four widths. CPU int4pack additionally requires out_features divisible by 16; the shared fused declaration is 8 for the separate XPU path. AUTO selects another compatible kernel otherwise; an explicit incompatible fused request raises before forward. XPU's separate support is unchanged.

Biased FP16 CPU inference stays on the packed fused path: it requests an FP32 matmul output, adds bias in FP32, and casts once. The ATen GPTQ path uses the same epilogue; FP16 bias alone no longer forces full-weight reconstruction.

On CPU, fused GPTQ and AWQ keep FP32 activations in FP32 rather than narrowing them to BF16 before int4pack. FP32 inputs return FP32 outputs, including with bias. The separate XPU fused path still uses BF16 operands for FP32 inputs.

CPU ATen/fused int4pack also requires in_features divisible by the resolved group size; a partial final group fails validation before packing or forward. Use backend="gptq_torch" for a GPTQ checkpoint with incompatible geometry, or let AUTO find a compatible kernel. The separate GPTQ AVX512 kernel supports ragged final groups and is exempt from this int4pack divisibility rule. CPU first-forward transforms for GPTQ/AWQ ATen, fused, and AVX512 AWQ are serialized per module, so concurrent initial forwards cannot snapshot or repack half-transformed weights.

docs/torch_fused_int4_transformations.md explains the checkpoint starting layout, XPU nibble unpack/reorder/transposition, and CPU int4pack conversion. Activation-order indices can require input permutation. AWQ first adapts its GEMM packing into the appropriate fused representation. Runtime code remains authoritative if historical line references in the note have shifted.

The fused/ATen GPTQ classes key activation-order indices to the live g_idx tensor and refresh derived scale/zero metadata after checkpoint loads, device moves, or observable source-buffer changes. Explicit CPU cache clears preserve the live packed operands and refresh derived metadata. Choose the dtype before the first eager forward. For a different activation dtype afterward, reload or use a new module. Forward dtype contract

Fused GPTQ CPU conversion unpacks the checkpoint's actual zero codes and computes affine offsets (2^(bits-1) - zero_code) * scale; it no longer substitutes zero offsets for asymmetric groups. The shared scale/offset dtype selection retains FP32 metadata when narrowing would lose range or affine identity. This correction applies to CPU conversion, not the separate XPU ABI. See INT4 transformations.

Training can use the Torch path and switch qzero format state; therefore a training-support declaration is not proof that the same fused inference operator has a backward implementation. Use the class's train() and normal initialization rather than manually toggling ABI buffers.

CPU ATen INT4

The CPU overrides (TorchAtenLinear, TorchAtenAwqLinear, TorchFusedLinear, TorchFusedAwqLinear, TorchInt8Linear) share optimize(backend="inductor", mode=None, fullgraph=False). Inductor can lower the packed op directly when operands match the compute dtype. When a cast is needed, as for BF16 or biased FP16 paths that compute in FP32, a traceable custom op now keeps the cast and packed GEMM in the graph. This does not change the packed-weight layout. See CPU layer probes.

Model-level optimize() installs the inference layout before tracing for eligible eval-mode CPU GPTQ/AWQ ATen and fused int4pack layers, including the supported AWQ AVX512 path. When calling torch.compile directly on a cold layer, layout setup can run outside the trace. Warm up eagerly before a direct fullgraph=True trace that cannot graph-break. Unsupported layouts and forwards requiring activation gradients still use their normal fallback paths. See the cold-compile regression.

In eager eval(), input-gradient calls on GPTQ/AWQ CPU ATen, fused, and AVX512 layers also use temporary dequantization without replacing a live packed inference layout. This avoids rewriting buffers while another forward reads them. It does not give the packed GEMM a backward or change the separate XPU layout. See the gradient-after-inference regression.

TorchAtenLinear and TorchAtenAwqLinear call installed ATen CPU int4pack conversion/matmul operators. They declare 4-bit, groups 32/64/128/256, output widths divisible by 16, automatic padding, and training support. The CPU validators check the effective group size even when it equals in_features; a full-width group of 512 does not bypass the packed operator's limit. AUTO falls through to a compatible kernel rather than failing on the first forward. validate_once() requires both _convert_weight_to_int4pack_for_cpu and _weight_int4pack_mm_for_cpu.

The in-memory GPTQ ATen/fused dequantize_model(model, dtype=None) helpers honor an explicit dtype; otherwise they prefer surrounding non-quantized FP16/BF16 state, then other floating state, and fall back to layer bias/scales and FP16 when necessary. They keep bias-less layers at bias=None and preserve the replaced layer's device. These helpers are separate from checkpoint-level utils.model_dequant.dequantize_model(); the former GPTQModel.export() API has been removed.

CPU GPTQ and AWQ repacking also detaches small operands from checkpoint mmap storage. GPTQ ATen handles scales and g_idx; AWQ ATen handles scales, bias, and g_idx; both AVX512 variants ensure private FP32 scales/bias and detach g_idx. The shared detach_cpu_operand() helper copies CPU tensors only when their storage reports non-resizable, as file-backed checkpoint views do; ordinary resizable allocations retain their identity, and non-CPU tensors pass through. This prevents retained metadata from pinning a whole safetensors shard after its packed weights were replaced without needlessly replacing ordinary loader-owned tensors. Values and checkpoint-facing dtypes/order follow each kernel's contract; other surviving views can still keep a shard mapped. See operand ownership, AWQ ATen, and AWQ AVX512.

The affine zero-offset conversion is important. The CPU operator's signed-code form uses scale * (signed_code - 2^(bits-1)) + zero_offset; the adapter computes (2^(bits-1) - zero_code) * scale so it reproduces the checkpoint's scale * (code - zero_code). Supplying raw zero codes where the operator expects offsets changes the math.

For transformed CPU GPTQ/AWQ ATen and fused int4pack layers, eager gradient forwards refresh affine offsets after a scale-only update, as does clear_weight_cache(). If the caller also changed runtime qzeros, the eager refresh treats those values as explicit offsets. A compiled gradient forward computes offsets from the current scales and retained checkpoint zero codes when available, without writing to the module. See the scale-update regression.

CPU INT4 dense fallbacks select a decode dtype that can represent the weights before multiplication by the input. This includes the temporary decode from a warmed int4pack layout on a gradient forward. It can widen to FP32 or FP64 when the stored scales would overflow a half-precision weight reconstruction. The result still uses the activation dtype. This lets a small FP16 activation produce a finite FP16 output even when the intermediate weight exceeds FP16 range. See the FP16 overflow regressions.

For BF16 activations, the shared _cpu_int4pack_mm() adapter converts the current activation and affine scale/zero operands to FP32 for ATen's CPU int4pack matmul, then casts the result back to the input dtype. This reduces numerical drift observed with BF16 operands on ARM CPU int4pack. It applies to both GPTQ/AWQ ATen classes (including their fused-projection calls) and the CPU Int4PackedOp used by fused GPTQ/AWQ. Packed weights remain INT4; this is per-call operand promotion, not a dense-weight cache or a new backend/config flag. XPU fused execution is a separate path.

CPU AVX512-BF16

TorchCpuAvx512Linear and TorchCpuAvx512AwqLinear use int4_qzero_group_scaled_cpu. Their validators check AVX512-BF16 and resolve native pack/GEMM ops. Declared GPTQ dimensional constraints include input width divisible by 2 and output width divisible by 32; other padding/packing rules still apply. The specialized fused input dtype is BF16, with the class controlling other interface behavior.

Both classes declare SUPPORTS_TRAINING. The fused kernel has no backward path, so it serves only eval/no-grad inference; in training mode forward() routes through a dequantized matmul with gradients intact — for the GPTQ class this is the inherited TorchAtenLinear training path. The cached fused-vs-dequantized dispatch resets when train()/eval() switches modes, so an eval forward cannot pin later training forwards to the fused kernel. The AWQ class stores its bias in FP32 for the fused path and adds it in place in the training fallback, rounding once to the activation dtype like the ATen path. On CPU-only GPTQ/AWQ loads, AUTO prefers these trainable kernels and falls back to an inference-only CPU kernel only when no trainable candidate validates; a training-mode switch on an inference-only kernel raises and points at backend="auto_trainable". This matters for HF/PEFT loading, which uses BACKEND.AUTO and then calls model.train() for LoRA fine-tuning.

When the GPTQ layer already holds AVX512-packed weights and needs a dense fallback, for example for gradient-enabled evaluation, it decodes the live AVX512 layout rather than treating it as ATen int4pack. The temporary decode can widen for range safety and does not restore or overwrite the checkpoint layout.

The transformed AWQ class preserves exact FP32 scales and bias across module device/dtype moves instead of rounding those operands through BF16/FP16. Its cache-clear hook restores the checkpoint snapshot when present, or re-establishes FP32 affine operands for a live transformed layer without one. The fused operator still requires BF16 activations. Projection fusion uses the concatenated int4pack operands and suppresses checkpoint restoration while binding source views; the combined executor discards the first source's inherited snapshot rather than shrinking back to that source's width.

Native INT8 and the retired AWQ expansion

TorchInt8Linear is an 8-bit-only, CPU, explicit-only backend. post_init() centers each source GPTQ group's codes into signed INT8 while preserving that group's original scale and zero offset for _weight_int8pack_mm. It rejects raw packing and training. It is not a path for persistent INT4-to-INT8 expansion.

Cold source dequantization crops decoded input rows to the logical in_features before looking up group scales and zeros. For example, a 17-channel input stored as 20 packed rows decodes only its 17 real channels. Explicit dequantization after dropping the derived runtime layout, including after a dtype/device apply, therefore handles a partial final packing word consistently with the initialized runtime view. Buffers that decode fewer than in_features rows raise instead of silently truncating the model width.

Native INT8 initialization and cold dequantization both crop unpacked qzeros to out_features before applying group scales and zero offsets. For example, a five-output projection can store eight zero-point lanes in two INT32 words. The extra three lanes are padding. This applies to both symmetric and asymmetric 8-bit GPTQ checkpoints. INT8 source and runtime decode

After post_init(), the backend releases the original qweight buffer after verifying that the derived layout exactly reproduces the checkpoint words, one tile of output columns at a time. This bounds the widened INT32 scratch used for verification. A mismatch or reconstruction error retains the source. qzeros, scales, and g_idx remain checkpoint buffers. state_dict() and the export path assemble the original qweight from the same column tiles as needed, and strict loading restores the source slot before installing new weights. The derived INT8 buffers are non-persistent. Forward restores activation ordering as needed and accumulates group results in FP32 before adding bias and casting to the input dtype. Symmetric groups use native INT8 reductions; asymmetric groups apply their zero points before reduction. Source signatures refresh the layout after versioned edits, replacement, state load, and moves; call clear_weight_cache() for in-place edits to unversioned inference tensors. Cold rebuilds are serialized while unchanged forwards reuse the layout. See the INT8 checkpoint lifecycle.

Compiled, unfused INT8 projections use the live checkpoint scales and applicable zero points, so affine edits are visible without recompiling. A strict state load repacks an already initialized projection before the next compiled forward. This does not make arbitrary in-place writes to packed codes traceable.

Each nonempty native INT8 group is padded to its own multiple of 8, with zero padding in the matching activation columns. Groups with the same aligned width keep the dense runtime layout. Unequal widths use a flat int8_weight_gnk buffer indexed by int8_group_offsets; empty groups need no panel or kernel launch. This reduces runtime storage for ragged groups without changing checkpoint shapes. Native projection fusion requires the dense layout and matching group and input-permutation contracts; projections using the flat layout remain separate.

TorchInt8AwqLinear is a retired compatibility class. It has priority zero and SUPPORTS_RAW_PACKING=False; construction and validation raise a migration error explaining that persistent AWQ INT4-to-INT8 expansion is disabled. Its continued presence in the enum is not runtime support.

Lookahead

qlinear/lookahead.py retains configure_lookahead_chain() and configure_default_lookahead() for compatibility. They route through deprecated lookahead controls; dense lookahead is disabled with a warning, and no dense weight chain is retained. Do not document these helpers as a current inference acceleration feature.

BitBLAS and target detection

bitblas.py and bitblas_awq.py create BitBLAS operators and transform packed weights/scales/zeros into the runtime layout. The GPTQ class is BitblasLinear; the module also exposes a compatibility BitBLASLinear spelling used by utilities. Declared bits are 2/4/8 for GPTQ and 4 for AWQ. The module sets MINIMUM_BITBLAS_VERSION = "0.1.0.post1".

bitblas_target_detector.py obtains NVIDIA target tags from TVM, fuzzy-matches the GPU model with threshold 25, verifies target architecture compatibility, and falls back to "cuda" when appropriate. OEM model names pg506-230 and pg506-232 map to NVIDIA A100. The main module includes an SM90 target fallback limit for compatibility; target-string fallback is not independent validation that generated code supports every later device.

GPTQ BitBLAS target detection passes torch.cuda.current_device() as a logical CUDA index. Device-SMI resolves that index through CUDA_VISIBLE_DEVICES; the detector no longer parses its first entry as an integer physical device ID. Reordered visibility lists, GPU UUIDs, and MIG identifiers therefore reach the logical-device resolver.

utils/bitblas.py distinguishes already-serialized BITBLAS from GPTQ-to-BitBLAS repacking. BITBLAS_ENABLE_TUNING can override tuning; absent that variable, repacking does not enable tuning by default, while non-repack conversion does. For the documented BitBLAS version, BF16 is an interface dtype with FP16 operator execution and boundary casting because the bundled low-bit BF16 code generation fails; accumulation remains FP32. Do not promise native BF16 compute from the support dtype list alone.

GGUF family

When autograd is enabled and the activation or bias needs gradients, GGUFTritonKernel uses the differentiable Torch parent rather than a raw no-backward Triton launch. Its declared inference-only training policy is unchanged. BitsAndBytes' optional four-bit Triton path similarly defers input-gradient calls to the vendor implementation.

GGUF stores typed GGML quantization blocks, not generic GPTQ groups. GGUFTorchLinear reads native packed GGUF buffers and selects fused/tiled or current-call decoding paths according to qtype and workload. GGUFTritonKernel supports its own subset of qtypes and executes with FP16 work inputs before restoring the interface dtype. GGUFCppKernel uses the ggml CPU bridge with a Torch CPU alternative where its dispatch permits; GGUFCudaKernel keeps native packed buffers and caches CUDA execution plans.

The CPU bridge defaults to native matmul for supported layouts without benchmarking it against floating-activation Torch; execution failures fall back to Torch for that rows/dtype/qtype key. Native ggml may quantize activations to Q8, so this is a numerical policy. Its CPU weight path can wrap aligned contiguous packed UINT8 storage, with a copied-buffer fallback; initialization aligns unaligned checkpoint views once. CUDA plans share one packed ggml weight allocation per layer's current device/weight identity and one activation/output/intermediate scratch arena per device across all layers and row counts. Arena growth invalidates stale plan generations; it does not upload another packed weight for each batch size. The arena is freed when the last live native GGUF CUDA weight on that device is released. See GGUF runtime details.

Native QQQ eager and compiled calls allocate their own INT32 reduction scratch with torch.empty and return it to the Torch caching allocator after use. No process-wide device/stream cache retains it after the model is released. Each forward also allocates fresh zeroed split-K locks rather than using the registered compatibility workspace buffer, preventing shared-counter corruption between overlapping calls on one module. The portable QQQ path keeps nibble-unpack and scale-product operands temporary and avoids INT64-expanded lane arithmetic. See QQQ storage and scratch.

Class Important execution boundary
GGUFTorchLinear Broad logical bit declaration is not a blanket qtype guarantee; inspect qtype-specific implementation.
GGUFTritonKernel Declares logical bits 1/4/5/6 and CUDA; requires Triton and its free-threaded-Python version gate.
GGUFCppKernel Requires the bridge's CPU availability; CPU input required.
GGUFCudaKernel Requires the bridge's CUDA availability and CUDA input; reuses compatible native matmul plans. Rejects Q8_1, Q8_K, TQ1_0, and TQ2_0 at construction and before native graph creation; use GGUF_TORCH. AUTO can retry after construction-time rejection. CPU native execution still supports the ternary pair.

The model loader imposes additional native Q1_0/Q2_0 restrictions described in Inference Backends. These local kernel descriptions do not apply to the dense Q1_0 FAST import path: in from_pretrained() with no quantization configuration, native Q1_0 AUTO resolves to FAST and can bypass local packed execution entirely. LOW_MEMORY uses the quantized-loading redirection instead. Plan or byte-buffer caching is different from retaining decoded dense weights. See GGUF for the format itself.

FP8, FP4, and W4A activation families

Weight-format FP8 and FP4

TorchFP8Linear defaults to weight-only W8A16 (activation_quant_method="none"). Eligible CUDA row/tensor layouts apply scales in the FP32 output epilogue. Eligible block layouts on NVIDIA CUDA use a fused small-row product or a single-pass Triton decode before matmul. Both optimized block routes bypass an unsafe FP16 decoded-weight range in favor of FP32 decode and matmul before the final cast. W8A8 requires a layer-level "per_token" opt-in, eligible tensor/row scales, and a passing _scaled_mm row-scale probe for the GPU and output dtype. Biased ROCm layers use W8A16. A failed production _scaled_mm call disables that path for the module until reset. Compiled CPU execution has an eager fallback. See FP8 runtime dispatch.

TorchFP4Linear holds NVFP4-packed checkpoint weights and constructs a TorchAO NVFP4Tensor view for native execution. CPU execution decodes the live FP4 buffers for the current F.linear. This helper has no independent BACKEND enum member and should not be confused with W4A, which starts from GPTQ INT4 weights and quantizes activations.

For FP16/BF16 input, a stored wider bias that cannot narrow exactly selects a call-local decoded FP32-reduced matmul/bias epilogue, including on NVIDIA where it bypasses TorchAO. Exactly representable bias can keep the native route. See FP4 bias precision.

W4A kernels and constraints

W4AFP8Linear and W4ANVFP4Linear both have zero AUTO priority. They require explicit/activation-policy-directed selection, GPTQ/GPTQ_V2 INT4, group size 128, desc_act=False, symmetry, int32 packing, input width divisible by 128, output width divisible by 32, and no adapters. The centered-code path checks contiguous groups and zero point 8 after the appropriate GPTQ zero-format interpretation.

Both are NVIDIA-only. Shared validation and direct FP8 GEMM/INT4 packing/NVFP4 packing entry points reject ROCm before compiling NVIDIA MMA/PTX operations. FP8 weights being representable on AMD does not make this W4A activation runtime portable. Load the native GPTQ checkpoint on ROCm with BACKEND.GPTQ_TORCH and without a W4A activation policy instead.

Implementation Role
w4a_floatx.py W4AFP8Linear: FP8 hardware floor (8, 9); constructs an E4M3 runtime operand from centered INT4 codes; calls group-aware FP8 execution.
w4a_triton.py Per-token FP8 packing, E4M3 rounding, grouped FP8 GEMM, and prepacked activation consumption. Launches bind to the operand device. The Gluon pieces import lazily, so the module loads on any Triton build; fp8_linear_prepacked raises native_fp8_gemm_error() when mma_v2 is absent (first shipped in Triton 3.6.0).
w4a_nvfp4.py W4ANVFP4Linear: floor (12, 1); native FP4/FP8 operands, integer-plane encoding, scale swizzling, block-scaled GEMM, and mixed FP8 consumers. Pure NVFP4 needs no Gluon; the mixed FP8 consumer calls fp8_linear_prepacked and reports the Triton 3.6.0 floor at run time.
w4a_nvfp4_triton.py NVFP4 pack/swizzle/decode, supported recipe variants, and exact low/high INT4-plane accumulation with GPTQ group scales.
w4a_activation.py Immutable W4AActivation carrier: mode, packed codes, scales, logical shape, model dtype, optional global/token scales, recipe, rotation flag, and exact-reference tensor.
w4a_boundary.py Producer-boundary policy, mixed MLP precision policy, scale completeness checks, and NVFP4BoundaryQuantizer.
w4a_llama_replay.py Installation/removal of quantization replay hooks; reference QDQ and precision matching for the operands that deployed consumers actually see.
w4a_llama_stream.py Encoded stream installation on complete selected Llama decoder layers, residual/norm/attention/MLP boundaries, head transition, and data-parallel rebinding.

W4AActivation.to() moves codes and all scale/reference tensors together. decode() reconstructs the quantized hardware operand; exact() returns the source compute-dtype reference when present. Residual arithmetic, norms, and the unquantized head must not accidentally consume only a low-bit approximation when the stream policy requires the reference value. Packing scales independently of their codes, or dropping token_scale, changes semantics.

NVFP4 boundary scales must be finite positive FP32 values and match the exact expected producer key set. Runtime copies use INT32 bit buffers to survive model dtype conversion without rounding the FP32 calibration scalar. Replay, producer calibration, and runtime installation must use the same mixed-precision policy. The Llama helpers are model-structure-specific, not universal hooks for arbitrary architectures.

Weight-storage lifetime: W4AFP8Linear and NVFP4's mixed FP8 attention lane materialize E4M3 weights per forward through fp8_weight_operand(); neither retains an expanded _weight_e4m3 plane. NVFP4 retains its two packed E2M1 planes in _weight_both (together 1 byte per logical weight) and _unit_weight_scales, in addition to native GPTQ checkpoint buffers. Its native-weight token check requires post_init() rebuilding after observable changes rather than accepting stale packed planes. persistent=False excludes those packed buffers from the state dict, not runtime memory. Account for retained packed operands and temporary expanded scratch rather than estimating resident memory from checkpoint INT4 bytes alone.

AWQ native and portable families

AWQ GEMM, GEMV, GEMV_FAST, and LLM_AWQ are different packing contracts. The model loader, class declaration, and native entry point must agree.

Family Runtime behavior
AwqGEMMLinear Native CUDA extension; rejects ROCm in validation. Its forward handles device/dtype/contiguity checks. The autograd wrapper uses dense-current-call matmul from 640 flattened rows with fp32_accum=True, or above 1024 rows regardless. Biased native calls and the dense branch keep output in FP32 through bias addition, then cast once.
AwqGEMMTritonLinear AWQ GEMM via Triton; checks versions and packed tensor co-location, converts work input to FP16, and restores the interface boundary.
AwqGEMVLinear CUDA-only GEMV layout; uses the GEMMv2 forward for more than 8 flattened rows, otherwise GEMV.
AwqGEMVTorchLinear CPU reader of the same GEMV layout under AWQ_TORCH; priority zero with a CPU-only AUTO fallback. FP32 reconstruction/matmul is call-local, with bias before output conversion. Output width must be divisible by 64; does not read fast/LLM layouts.
AwqGEMVTritonLinear Explicit-only Linux CUDA/ROCm reader of the same GEMV layout, with FP16 work inputs, FP32 output/bias addition for biased calls, then a cast back to input dtype before adapter processing. Gradient-requiring input/bias calls use temporary differentiable dequantization/matmul. No persistent dense weights. Output width must be divisible by 8 and normalized group size must be 64 or 128; AUTO does not opt into it.
AwqGEMVFastLinear / LLMAwqLinear Fast GEMV and legacy LLM_AWQ layouts; runtime zero/scales preparation is layout-specific. AUTO resolves inferred dtypes to FP16 for GEMV, GEMV_FAST, and LLM_AWQ before calibration/loading; explicit incompatible dtypes raise. Explicit backends use their declared dtype contract.
AwqTorchLinear Portable packed GEMM fallback, including bit widths outside the native INT4 kernels' declarations.
AWQ Marlin/Machete/Swordfish/ExLlama/BitBLAS Repacking/adaptation classes described above; not new AWQ quantizers.
AWQ fused/ATen/AVX512 CPU/XPU ABI adapters where their declarations/operators permit.

The native AWQ GEMM/GEMV/GEMV_FAST/LLM_AWQ classes require NVIDIA CUDA. On ROCm, GEMM can use portable Torch or Triton, ordinary GEMV uses explicit AWQ_GEMV_TRITON, and GEMV_FAST/LLM_AWQ use their layout-specific AWQ_TORCH readers under AUTO or explicit selection. CPU and ROCm pack GEMV_FAST/LLM_AWQ with those same portable classes. See AWQ layout and backend selection.

Native AWQ GEMV and GEMV_FAST/LLM_AWQ keep split-K partials in FP32. Biased calls request an FP32 kernel result for both decode and prefill, add the stored bias, and cast once to the activation dtype. This does not change their FP16 work operands or checkpoint layouts. See AWQ GEMV, GEMV_FAST, and LLM-AWQ.

Portable AwqTorchLinear preserves source scales/bias dtypes during packing and initialization, and the loader restores these buffers after Accelerate's model-dtype cast. Forward uses temporary compute-dtype scales and a bias moved without narrowing; matmul_fp32_reduce() keeps a wider or incomparable-dtype stored bias wide through the epilogue. Repeated forwards at the module's fixed activation dtype do not overwrite checkpoint precision. The shared helper also dequantizes directly in FP32 for Paro's FP32 dense fallback instead of passing through FP16. Dense weights remain call-local. See Torch AWQ precision and the source flow.

Native GEMM now validates output widths divisible by 64 and groups -1, 32, 64 or 128 (-1 is normalized to the input width for the native check). GEMV_FAST/LLM_AWQ require group size 128, input width divisible by 64, and output width divisible by 128. Unsupported geometry is rejected during selection, allowing AUTO to try another compatible backend instead of failing in packing/prefill.

QQQ, VPTQ, Paro, and bitsandbytes

Family Representation and execution
QQQ QQQLinear uses QQQ-packed B, group/channel scales, dynamic activation quantization, and native multiply. It declares 4-bit symmetric weights, no activation ordering, group size -1 or 128, feature widths divisible by 64, and no training. QQQTorchLinear uses the same QQQ format with portable execution. Native availability/capability validation still applies; on ROCm the native kernel is rejected up front and AUTO selects QQQTorchLinear.
VPTQ VPTQLinear stores vector codebooks/packed indices plus configured residual/outlier state; Torch forward decodes the current weight for F.linear. VPTQCudaLinear uses a fused native path for eligible row counts/dtypes and checks that activation/indices/codebooks share an NVIDIA CUDA device; other work follows its fallback path. Registry loading, availability checks, and prewarming reject ROCm/CPU before JIT build; use VPTQ_TORCH there. This is not scalar GPTQ packing.
Paro ParoLinear stores packed affine codes plus pairwise rotation/scaling metadata. It rotates inputs before quantized matmul and retains separate adapter semantics. PARO_CUDA covers device-dependent native, Torch and CPU paths. On ROCm, AUTO ranks ParoTritonLinear ahead of the portable dense fallback when the Triton kernel passes validation.
bitsandbytes BitsAndBytesLinear represents NF4/FP4 codes plus QuantState or INT8 state, with CPU/CUDA-facing (including HIP) declarations. Eligible RDNA4 four-bit calls use fused packed Triton at M >= 16 with FP16/BF16 compute; Ada keeps vendor CUDA unless KernelConfig.bitsandbytes_triton=True enables A/B use. Smaller/unsupported calls and INT8 keep the existing paths. Native/packed CPU and current-call fallback logic remain; see bitsandbytes_triton.py for gates.

Use QQQ, VPTQ, Paro, and BitsAndBytes for quantization-specific configuration and checkpoint details.

VPTQ stores linear bias in FP32 independently of centroid dtype. Both wrappers keep inexact-to-narrow bias wide through the final epilogue; VPTQ_CUDA uses temporary native reconstruction instead of the fused small-row route in that case. An exactly representable bias can retain fused execution. See VPTQ dispatch.

NF12 lossless BF16 encoding

NF12 also has a native MLX route: NF12MlxQuantLinear transfers canonical packed bytes/escape metadata into MlxNF12Linear. Single-row calls fuse decode into the Metal dot; multirow calls decode only for the current multiplication and never retain a dense weight. Geometry remains K divisible by 64 and N divisible by 16. See MLX native inference.

qlinear/nf12.py declares bits=[12], K divisible by 64, N divisible by 16, no automatic padding, sharding enabled, training disabled, and FP16/BF16/FP32 activations. Only BF16 activations enter the optional native paths. The CUDA loader delegates to the wheel's lightweight gptqmodel_nf12 package, reuses a complete registered operator set, or loads/builds vendored libraries with CUDA 13.0+, CMake 3.26+, Ninja, and SM80+ visible GPUs. The lightweight API adds packed linear, embedding, MoE, and FP32-output projections; it is NVIDIA-only, not a portable/HIP entry point. RDNA4 uses a separate HIP build for gfx1200/gfx1201/gfx1250; other device declarations do not imply native support. Torch fallback decodes only for the current multiplication. FP16 or narrowing-autocast fallback keeps BF16 weights and the epilogue wide instead of converting weights to FP16; no dense weights persist. See NF12 for the API, configuration, checkpoint geometry, cache location, dispatch, and validation flags.

The HIP wrapper flattens a contiguous copy of strided/higher-rank activations and restores the caller's leading dimensions. Its small-M route handles up to 15 rows; WMMA handles eligible 16–256-row calls on gfx1200/gfx1201, not gfx1250. Split-K adds bias once after FP32 partial reduction. The documented validated stack is ROCm SDK 10.1.0 with HIP 7.16.26385, not a minimum-version declaration. Native geometry/workspace/routing checks and expanded WMMA launch bounds do not change checkpoint bytes or expose the compiled MoE helpers through the registered Torch API.

The separate source-storage policy auto_dtype_cast="auto_nf12" (default) can use NF12 during quantization forwards without selecting NF12 as the output method. Its eligible linear sources use canonical packed bytes; selected embeddings use native row-major packing and lookup on NVIDIA SM80+ or supported RDNA4 HIP. Canonical HIP embedding lookup is also exposed. Current-call FP32 reconstruction can use fused widening decode, and GPTQ + EoRA can fuse decode/subtract before releasing its packed source. The removed experimental_rowmajor Python module is not the production API. See automatic source storage and row-major embedding helpers.

MACaQ packed kernels

qlinear/macaq.py defines MacaqLinear (MACAQ_TORCH) and MacaqTritonLinear (MACAQ_TRITON) for FORMAT.MXFP4_MACAQ and transform-free FORMAT.MXFP4 with METHOD.MACAQ. Both declare FP16/BF16/FP32, fixed 4-bit weights, sharding, no training or auto-padding, K divisible by 32 (and runtime had_block), and unrestricted N. Torch supports all declared device families; Triton admits NVIDIA CUDA and ROCm gfx1201 after selected-device validation. The Triton W4A4 path consumes packed E2M1 operands; fallback reconstructs weights only for the current call. rotation="none" omits BH; pre_mode="none" omits pre_scale. W4A16 skips disabled activation passes, while Triton W4A4 can use a temporary identity transform for its fused pack. MacaqMlxQuantLinear transfers transform-free W4A16 to stock MLX QuantizedLinear; other MACaQ layers use MlxMACaQLinear. See MACaQ.

MLX checkpoint holders and executable modules

qlinear/mlx.py declares method/format-specific Torch checkpoint holders with priority zero and MPS device declarations. Their forward() raises until the model is converted into MLX modules. The holder hierarchy covers GPTQ/GPTQ_P, AWQ variants, MARLIN, BITBLAS, Paro, QQQ, GGUF, FP8, and bitsandbytes; the loader also has EXL3-specific conversion.

Source Executable MLX implementation
mlx_gptq.py MlxGPTQLinear, MlxGPTQGroup16Linear; packed affine/group-16 execution and output cast to input dtype.
mlx_awq.py MlxAWQLinear, MlxAWQGroup16Linear; AWQ affine/group-16 execution.
mlx_group16.py MlxGroup16Linear; exact alternating half-group scales through two group-32 affine matmuls; validates input alignment and supported affine bits.
mlx_exl3.py MlxEXL3Linear; packed trellis matmul and block Hadamard transforms without a dense-weight fallback.
mlx_gguf.py MlxGGUFLinear / MlxGGUFQ6KLinear; affine GGUF and specialized packed Q6_K execution.
mlx_paro.py MlxParoLinear; pairwise rotations before quantized matmul.
mlx_qqq.py MlxQQQLinear; per-token activation and per-channel output-scale semantics.
mlx_bitsandbytes.py MlxBitsAndBytesLinear; packed 4-bit / INT8 execution and exact signed-to-affine INT8 repacking helper.

An MLX implementation is not a Torch MPS kernel. Install/use the MLX-compatible model path and verify the method-specific conversion rather than calling a Torch holder directly. See MLX and Apple Silicon.

Tenstorrent kernels and native block floats

Class / backend Format priority Bits / groups Activation / packing Device and shape contract
XLALinear / GPTQ_TT_XLA GPTQ/GPTQ_V2: 20 4; -1,16,32,64,128,256,512,1024 BF16 / int32 XLA, Linux, K/N divisible by 8, no padding; TT PJRT packed decode and BF16 matmul
TTNNLinear / GPTQ_TTNN GPTQ/GPTQ_V2: 15 Same BF16 / int32 XLA selector, native TTNN; same shape gates; explicit selection avoids PJRT initialization
TTNNBlockFloat8Linear / GPTQ_TT_BF8 GPTQ/GPTQ_V2: 15 Same BF16 / int32 input layout Experimental native TTNN GPTQ-to-bf8 runtime conversion; avoids PJRT probes, with HiFi4 and FP32 accumulation
TTBlockfloatLinear / TT_BLOCKFLOAT TT_BLOCKFLOAT: 100 4/8; blocks of 16 FP16/BF16/FP32 / uint8 codes CPU, CUDA, ROCm, XPU, NPU, MPS; K divisible by 16, N by 1; no auto-padding

TTNNLinear uses FP32 weight decode and matmul operands if any checkpoint scale loses precision in BF16. The public input and output stay BF16. See packed GPTQ on the cards.

The three GPTQ classes require symmetric weights, allow either desc_act value, and declare sharded loading but no training or adapters. They consume existing GPTQ words; a TTNN bf8 runtime conversion does not read native block-float checkpoint files. TTBlockfloatLinear declares sharded loading, no training, and LoRA adapters, but its dedicated checkpoint loader bypasses generic backend selection. These declarations do not establish a full TTNN deployment path for TT_BLOCKFLOAT.

Native block-float modules retain packed codes, exponents, and code_shape, with one exponent per 16 values. Eligible HIP and SM80+ CUDA kernels compute from packed storage; unavailable/unsupported calls use current-operation Torch dequantization. The standalone CPU ttbfp library is a separate build and is not the CPU module's automatic dispatch path. See Tenstorrent for the conversion command, dtype-preserving load, runtime ownership, installation, and source links.

🧰 Keep the standalone ttbfp helper separate

The public ttbfp README documents standalone CPU support for bf8_b, bf4_b, and bf2_b, a Makefile build, and a Python ctypes binding. That helper is separate from package JIT extensions and from TTNN/XLA runtime integration. Its optional CUDA build is not evidence of a completed CUDA runtime qualification. Do not infer 2-bit TTBlockfloatConfig model support from bf2_b CPU helper support: the configuration permits decoder bits 4 or 8.

Hooked modules during quantization

gptqmodel/nn_modules/hooked_linear.py provides HookedLinear plus Conv1D/Conv1d/Conv2d-compatible wrappers, StopForward, and legacy/tree-based replacement helpers. These intercept forwards for quantization processing and can materialize meta weights through the shell/lazy source and restore output placement. They are not entries in the inference BACKEND enum and do not replace a qlinear kernel's packing contract.

See Model Definitions and Module Tree, Looper Processors, and Memory, Offload, and Multi-GPU for their place in the quantization lifecycle.

Native C++/CUDA source map

gptqmodel_ext/__init__.py is only a support package; its empty __all__ is not the runtime kernel registry. Python loaders/utilities bind the relevant extension and torch.ops entry points.

Source path under gptqmodel_ext/ Role
marlin/ GPTQ/AWQ repack kernels; FP16/BF16 GEMM sources and Torch bindings; shared kernel/core code.
machete/ CUTLASS Machete core and PyTorch CUDA interface; generation/build driven by utils/machete.py.
swordfish/ Prepacked Blackwell implementation, bindings, core, and stable-libtorch support.
allspark/ CUDA W8A16 fused GEMM and N32K16 repacking, loaded by utils/allspark.py.
allspark_hip/ RDNA4 gfx1200/gfx1201 W8A16 WMMA GEMM and distinct HIP repacking, loaded by utils/allspark_hip.py.
exllamav2/ Native GPTQ/AWQ matrix creation, GEMM bindings, CUDA/CPU support code.
exllamav3/ EXL3 bindings, Hadamard, GEMM, trellis/quantization native implementation.
awq/ AWQ Torch bindings, GEMM/GEMV/fast quantization kernel sources.
qqq/ QQQ binding and quantized GEMM source.
vptq/ Native vector-quantized inference binding/kernel.
paro/ Native rotation implementation.
int4_qzero_group_scaled_cpu/ Native INT4 packing/GEMM, AVX512 implementation, QQQ CPU support, and binding code.
pack_block_cpu.cpp Blockwise GPTQ CPU packing.
floatx_cpu.cpp Native packed/floatx CPU support.
gptq_block/ Native GPTQ block-processing implementation used during quantization, not a selectable inference backend.
hadamard/ Hadamard transform bindings and CUDA implementation.
cutlass_extensions/ Shared CUTLASS support utilities.

Architecture-specific source compilation requires an appropriate Torch build, compiler/toolkit, headers, and the extension's dependencies. A successful Python import cannot substitute for an exact-device capability check. Do not copy one extension's build flags into another backend merely because both use CUDA.

ROCm quantization BLAS is not an inference BACKEND

utils/rocm.py::quant_blas_backend() scopes the process-wide preferred Torch BLAS library during quantization and restores the prior selection after the last participating run exits, on a best-effort basis. The default tuning request is "rocblas", translated to Torch's "hipblas" spelling, but the run plan decides whether a deterministic quantization pin is needed. The switch requires ROCm, the preferred-BLAS API, and an eligible current selection; an explicit different caller-selected library is preserved.

Run-plan processor Scoped policy
GPTQ, QQQ, VPTQ, MACaQ, EXL3, EoRA Pin rocBLAS for reproducible Hessian/Gram accumulation; EoRA's input Gram and eigh also require this scope.
Paro Pin rocBLAS for reproducible rotation-optimization GEMMs, including FP16/BF16; no Hessian is required to select this scope.
AWQ, RTN without a processor listed above Retain/guard hipBLASLt for forward/replay work.

The decision considers secondary processors too: AWQ + EoRA pins rocBLAS for the whole run, including calibration forwards that supply the adapter's Gram, instead of keeping the AWQ-only hipBLASLt policy.

Same-backend runs share the guard. Runs needing different backends wait instead of changing the process-wide selection under one another. If the 1800-second wait expires, the waiting run warns and continues on the active backend, so serial-equivalent reproducibility is no longer promised. This is not a global deterministic-algorithms guarantee; VPTQ also scopes its k-means segment reductions separately.

KernelConfig.rocm_quant_blas value Effect
None (default) Enable the run-plan policy above when runtime checks permit.
"hipblaslt", "cublaslt", "default", "off", "0", or an empty string Disable this override.
"rocblas" Resolve to the default run-plan policy.
Any unrecognized name Raise ValueError at configuration construction instead of silently falling back.

Values are stripped and lowercased, so " ROCBLAS " is accepted; a typo such as "hipblas_lt" is rejected.

The ROCm default and reproducibility reference separates the tuning default from the per-method pin; docs/rocm-quant-blas-backend.md covers the original gfx950 cold-start investigation. Disabling the guard warns because Hessian reproducibility can be lost. Measurements are method/hardware-specific, not inference-backend benchmarks or proof of cross-platform bitwise equality. Changing this kernel-policy field does not make a CUDA-only Marlin, Machete, Swordfish, or AWQ GEMM extension ROCm-compatible.

Availability and runtime-state lifecycle

CPU fused GPTQ/AWQ and Paro INT4 repacking extend checkpoint-mapping detachment to the operands retained after layout conversion: fused GPTQ scales/g_idx, fused AWQ scales/bias, and Paro scales/bias/theta/pairs/channel_scales. The shared helper only copies CPU storage reported as non-resizable; ordinary allocations keep identity. This avoids pinning a safetensors shard through small surviving views while keeping operand values/dtypes/order and packed-weight contracts unchanged. See Paro CPU ownership and concurrency.

GPTQ ExllamaV2Linear has native quantized GEMM for FP16. Its BF16/FP32 extension path reconstructs transient dense weights each forward, so auto_select_priority lowers the effective AUTO rank to at most 1 for those requested dtypes. The static priority table above lists declarations; explicit selection is still supported and equal effective ranks keep the prior order. See AUTO discovery and selection.

AUTO checks cached runtime availability before shape/dtype validation, including Machete; an unavailable native extension cannot pass merely because its shape fits. On ROCm, extension.load("all") skips CUDA-only AWQ/QQQ/Marlin entries, while explicit unsupported requests report actionable errors. RDNA4 family-wide source compilation is not runtime support for every target: AllSpark still rejects gfx1250 because its WMMA path is not implemented.

Derived packed/native caches follow their source tensor identity/version and are invalidated on movement, state load or mutation. GGUF CUDA keeps at most eight execution plans in an LRU and closes evictions.

GPTQ ExLlamaV2 serializes the cold path that creates/rebuilds its native q-matrix under the lifecycle lock, preventing concurrent first forwards from racing native reshuffling or descriptor publication. The completed cache key includes act-order permutation tensors inserted by matrix creation, avoiding a needless rebuild on the next call. Warm validated forwards retain the fast cached route.

ExLlamaV2 GPTQ/AWQ rebuilds stale matrix handles without reshuffling already finalized packed words. This preserves outputs across explicit cache clears, moves, and projection-fusion install/removal. Raw checkpoint loads detach fused views before copying; GPTQ runtime state dictionaries carry layout metadata, while AWQ recognizes its repacked runtime shapes. After AWQ finalization, raw checkpoint qweight and qzeros must be supplied together; incomplete raw pairs raise RuntimeError. See the descriptor lifecycle and native validation.

CPU INT4 layers capture non-persistent checkpoint state before inference or fusion transforms their layout; AVX512 also preserves the bias widened to FP32. Eligible CPU ATen/fused and AVX512 INT4 layers for GPTQ and AWQ replace the duplicate checkpoint qweight clone with a shape/dtype descriptor once the inverse has reconstructed the original words bit for bit for that layout in the current process. Matching layers reuse the verification rather than rebuilding the full checkpoint weight during initialization. ATen/fused and AVX512 GPTQ both undo non-identity ret_idx channel reordering for desc_act checkpoints. AVX512 AWQ rebuilds the original nibble ordering with Torch operations. Unsupported checkpoint dtypes, unknown or mismatching inverses, and XPU layouts retain the packed snapshot. Scales are not cloned when unchanged; compatible runtime zero offsets share the packed affine-metadata view. Export reconstructs checkpoint words on demand, not as a persistent dense or expanded cache. Explicit cache clears refresh metadata from the live packed tensors. Checkpoint loads, export through get_state_dict_for_save(), training restoration, and fusion removal restore checkpoint layout when the snapshot remains valid. ATen/fused device changes or changes to FP16/BF16 scale dtype also require reconstruction; same-device FP32 widening keeps the runtime layout. Partial state loads classify only supplied operands, preserve scales/bias-only updates in the live layout, and reject mixed or invalid layout shapes. Copy-based loads materialize inference-mode buffers as normal tensors in both checkpoint and runtime layouts, including g_idx, so a post-forward reload does not fail on in-place writes to inference tensors. Changed runtime-state loads make the old snapshot non-exportable until checkpoint-layout weights are reloaded. Snapshots are cloned outside inference mode and never appear in state_dict(). See checkpoint export and fusion storage. No dense reconstructed weights are retained.

Source: docs/kernel-validation-and-cache-lifecycle.md and docs/rocm-rdna4.md.

CPU Torch, ATen, fused, and eligible AVX512 INT4 layers support a checkpoint-state round trip through meta, to_empty(device="cpu"), and load_state_dict(). The GPTQ kernels rebuild their non-persistent unpack shift tables on the move. Transformed AWQ AVX512 restores checkpoint buffer layout before reloading, and the next eligible forward repacks it. Save the checkpoint state before moving to meta and reload it before forwarding. See the round-trip regression.

GGUF Triton's one-time packed-weight relayout serializes the check/copy/publication under a process-wide lock and waits for the calling CUDA/HIP stream outside graph capture before exposing the transposed qweight view. Concurrent cold builds therefore cannot derive persistent metadata from unfinished copied bytes; the wait is not part of steady-state forwards. This is separate from a completed cache's cross-stream readiness events. See Triton cache ownership.

Kernel implementation and validation checklist

  1. Declare method, backend, format/priority, bit widths, groups, desc_act, symmetry, training, shards, devices/platforms, packing/activation dtypes, divisibility, and adapters accurately. A positive priority opts the kernel into AUTO.
  2. Keep exact-device validation separate from process-global extension availability. Multi-GPU selection checks every target ordinal; do not freeze the current GPU's capability in an import cache.
  3. Preserve checkpoint zero-point conventions and layout semantics. Validate raw packing separately from loading/repacking; do not claim pack support solely because post_init() can transform loaded tensors.
  4. Keep weights quantized between operations. Temporary decode is allowed for the consuming operation; dense or expanded caches must not persist. Nonpersistent Torch buffers still consume memory and are not an exemption.
  5. Invalidate caches/handles on state changes, placement/dtype transitions, or mutation as required. _runtime_cache.tensor_cache_key() covers identity/version/device/dtype/shape, but inference-mode mutation can require explicit invalidation.
  6. Follow the repository's independent Torch-oracle and persistent-storage requirements for kernel/packer/math changes. Test supported FP16/BF16 paths independently, check output dtype, separate arithmetic error from output-rounding error, and include repeated forwards/mutation/lifetime cases rather than one successful call.

Pitfalls

  • Do not equate enum presence, format declaration, extension importability, or a broad logical-bit list with successful end-to-end execution.
  • Do not save private runtime packed tensors under a standard checkpoint format without restoring its serialization contract.
  • Do not assume a training-support declaration means the inference-native operator supplies backward; classes may dispatch differently in training.
  • Do not confuse CUDA/HIP's shared Torch device namespace with binary compatibility of native CUDA kernels.
  • Do not benchmark JIT compilation/repacking as steady-state GEMM, or present a declaration priority as a measured performance ranking.
  • Do not estimate peak memory from on-disk weight bytes; repacking, native workspaces, scratch, activations, and temporary decode can coexist.

Source files

See also

Clone this wiki locally