Repository navigation
GGUF
Documentation home · Quickstart · Lifecycle
On this page
GPT-QModel supports GGUF in two related but distinct forms:
-
GGML block-quantized linear weights in a GPT-QModel checkpoint, configured with
GGUFConfig, saved through the normal checkpoint writer, and executed by GGUF-aware Torch, native ggml, Triton, or MLX kernels. -
A
.ggufmodel file, resolved by the Hugging Face integration and imported either into a packed GPT-QModel runtime or into a dense Transformers model, subject to the loader restrictions below.
These are not interchangeable file formats. FORMAT.GGUF identifies the quantization family; it does not mean that model.save(...) writes a standalone llama.cpp model file. GGUF block compatibility, model-file compatibility, architecture support, and execution-kernel support are separate checks.
This page describes the implementation at source revision c1976918. Package requirements and format coverage can change; use the source for your installed revision when diagnosing differences.
Reading the diagram: Lanes are separate routes. Packing needs no calibration; .gguf import depends on architecture, tensor type, processor and profile. Q1_0 auto/fast chooses dense import.
Text version of the workflow
- Create a GPT-QModel checkpoint: Choose a GGUF subtype → Pack source matrices → Save checkpoint directory.
- Eligible packed .gguf import: Check import eligibility → Install packed blocks → Execute block kernels.
- Dense HF .gguf import: Resolve the model file → Decode GGUF tensors → Run the dense model.
| Concept | Value | Purpose |
|---|---|---|
| Quantization method |
METHOD.GGUF, value "gguf"
|
Selects GGUF weight-only quantization |
| Checkpoint family |
FORMAT.GGUF, value "gguf"
|
Selects compatible loaders and linear kernels |
| Public subtype | For example, "q_0", "q_k", "q_k_m", "iq_xs"
|
Combined with bits to identify a block representation |
| Full alias | For example, "q4_0", "q4_k_m", "iq2_xs"
|
Width plus subtype, accepted as a bits specification |
| Runtime specification | GGUFBits |
Structured width/version/variant/quality representation |
| Actual tensor type | For example, Q4_0, Q4_K
|
Determines block size, byte layout, and kernel dispatch |
GGUFConfig(bits=4, format="q_k_m") and GGUFConfig(bits="q4_k_m") normalize to the same runtime specification. The public config stores an integer width and a subtype string, not a GPTQ-style integer bit width alone.
The dataclass fields are:
bits: int
version: str
variant: str
quality: Optional[str] = None| Field | Validation |
|---|---|
bits |
Positive integer; GGUFConfig further restricts widths to 1, 2, 3, 4, 5, 6, 8
|
version |
q, iq, tq, mxfp, nvfp
|
variant |
Empty string, 0, 1, k, xxs, xs, s, m, nl
|
quality |
None, xs, s, m, l, g128
|
Individual component validation does not establish that a combination is supported. The alias registry and GGUFConfig combination validation are authoritative.
| API | Behavior |
|---|---|
GGUFBits.from_string(value: str) -> "GGUFBits" |
Strips surrounding whitespace, lowercases, replaces hyphens with underscores, and looks up a registered full alias |
GGUFBits.from_alias(value: str) -> "GGUFBits" |
Compatibility alias for from_string
|
to_string() -> str |
Full alias such as q4_k_m
|
serialize() -> str |
Same canonical full alias |
to_public_format() -> str |
Drops the width from the prefix: q4_k_m becomes q_k_m; width-in-name formats mxfp4 and nvfp4 remain unchanged |
__repr__() |
Constructor-style debug form, for example GGUFBits('q4_k_m')
|
QuantBits = GGUFBits remains a backward-compatible alias in the configuration module.
from gptqmodel.quantization import GGUFBits, GGUFConfig
spec = GGUFBits.from_string(" Q4-K-M ")
assert spec.to_string() == "q4_k_m"
assert spec.to_public_format() == "q_k_m"
config = GGUFConfig(bits=spec)
assert config.bits == 4
assert config.format == "q_k_m"
assert config.runtime_bits == specThe GGUF-specific dataclass declarations are:
bits: Union[int, str, GGUFBits] = field(default=4, metadata={"choices": [1, 2, 3, 4, 5, 6, 8]})
format: Optional[str] = field(default=None)
method: METHOD = field(default=METHOD.GGUF, init=False)
group_size: int = field(default=-1, init=False, repr=False)
desc_act: Optional[bool] = field(default=False, init=False, repr=False)
sym: bool = field(default=True, init=False, repr=False)These are field declarations, not the complete generated constructor: GGUFConfig also inherits preprocessing and base configuration fields. See QuantizeConfig Reference and Quantization Lifecycle.
| Parameter/property | Default | Meaning |
|---|---|---|
bits |
4 |
Integer, numeric string, full alias, or GGUFBits; normalized public value is an integer |
format |
None at construction |
Resolved to a public GGUF subtype; default with four bits is q_0
|
method |
METHOD.GGUF |
Fixed; not a constructor argument |
group_size |
-1 |
Fixed; GGUF block/subblock sizes belong to the tensor layout, not a GPTQ grouping option |
desc_act |
False |
Fixed; no GPTQ activation ordering |
sym |
True |
Fixed config convention; does not imply that every GGML block is mathematically symmetric |
runtime_bits |
Derived GGUFBits
|
Used by kernel selection and packing |
uses_weight_only_lifecycle() |
True |
Packs source weights directly rather than running GPTQ Hessian optimization |
supported_export_formats() |
(FORMAT.GGUF,) |
Supported checkpoint quantization family, not a .gguf file-writer promise |
format=None, the empty string, and the family token "gguf" request the width-specific default. Full aliases in format normalize to a subtype. A full alias in bits must agree with an explicit format; incompatible combinations raise ValueError.
Integer bits
|
Default full alias | Default public format
|
|---|---|---|
| 1 | q1_0 |
q_0 |
| 2 | q2_0 |
q_0 |
| 3 | q3_k |
q_k |
| 4 | q4_0 |
q_0 |
| 5 | q5_k_m |
q_k_m |
| 6 | q6_k |
q_k |
| 8 | q8_0 |
q_0 |
Dynamic rules normalize bits and format through the same resolver. For example:
config = GGUFConfig(
bits=4,
format="q_k_m",
dynamic={
r".*\.mlp\..*": {"bits": "q5_k_m", "format": "q_k_m"},
},
)A negative dynamic rule can exclude a module. Kernel/layout validation still applies separately to each selected module. An omitted dynamic format inherits the base subtype, even when bits contains a full alias; the inherited subtype and alias must agree. When changing representations, explicitly supply both bits and format to avoid a conflict or an unintended inherited layout.
to_dict() serializes the public subtype and removes group_size, desc_act, and pack_dtype. It also strips runtime-only metadata such as weight-only/offload/packing/VRAM-management settings. The config is not a dump of every runtime attribute.
The following table combines the alias registry with _GGUF_TYPE_INFO. Bytes include block metadata. Effective bits per weight are 8 * bytes / elements, not just the nominal width in the name; these values exclude model-level metadata, non-quantized tensors, and padding.
| Accepted full alias(es) | Tensor layout | Elements/block | Bytes/block | Effective bits/weight |
|---|---|---|---|---|
q1_0, q1_0_g128
|
Q1_0 / compatibility name Q1_0_g128
|
128 | 18 | 1.125 |
q2_0 |
Q2_0 |
64 | 18 | 2.25 |
q4_0 |
Q4_0 |
32 | 18 | 4.5 |
q4_1 |
Q4_1 |
32 | 20 | 5 |
q5_0 |
Q5_0 |
32 | 22 | 5.5 |
q5_1 |
Q5_1 |
32 | 24 | 6 |
q8_0 |
Q8_0 |
32 | 34 | 8.5 |
q8_1 |
Q8_1 |
32 | 36 | 9 |
q2_k, q2_k_s, q2_k_m
|
Q2_K |
256 | 84 | 2.625 |
q3_k, q3_k_s, q3_k_m, q3_k_l
|
Q3_K |
256 | 110 | 3.4375 |
q4_k, q4_k_s, q4_k_m
|
Q4_K |
256 | 144 | 4.5 |
q5_k, q5_k_s, q5_k_m
|
Q5_K |
256 | 176 | 5.5 |
q6_k |
Q6_K |
256 | 210 | 6.5625 |
q8_k |
Q8_K |
256 | 292 | 9.125 |
iq1_s |
IQ1_S |
256 | 50 | 1.5625 |
iq1_m |
IQ1_M |
256 | 56 | 1.75 |
iq2_xxs |
IQ2_XXS |
256 | 66 | 2.0625 |
iq2_xs |
IQ2_XS |
256 | 74 | 2.3125 |
iq2_s |
IQ2_S |
256 | 82 | 2.5625 |
iq3_xxs |
IQ3_XXS |
256 | 98 | 3.0625 |
iq3_s |
IQ3_S |
256 | 110 | 3.4375 |
iq4_nl |
IQ4_NL |
32 | 18 | 4.5 |
iq4_xs |
IQ4_XS |
256 | 136 | 4.25 |
tq1_0 |
TQ1_0 |
256 | 54 | 1.6875 |
tq2_0 |
TQ2_0 |
256 | 66 | 2.0625 |
mxfp4 |
MXFP4 |
32 | 17 | 4.25 |
nvfp4 |
NVFP4 |
64 | 36 | 4.5 |
Quality suffixes such as _s and _m are preserved in the bit specification but map to the same base tensor layout here. They do not, by themselves, implement llama.cpp's whole-model mixed-type quantization recipes. Use explicit dynamic rules for per-module choices; do not infer a mixed-precision model from a filename-like suffix.
Selected layout details:
| Type | Reconstruction/storage detail |
|---|---|
Q1_0 |
FP16 scale plus 128 sign bits; local packer uses mean absolute weight for the scale |
Q2_0 |
FP16 scale plus packed two-bit codes; local quantizer uses a ternary decision at normalized ±0.5 |
Q4_0 |
FP16 scale and 16 bytes of nibbles; reconstruction uses d * (code - 8)
|
Q4_K |
256-weight superblock; FP16 bases, packed subblock scales/minima, and four-bit codes; reconstruction is scale-times-code minus minimum |
Q5_K |
Related scale/minimum layout with an additional high-bit plane |
Q6_K |
Low four-bit and high two-bit planes, signed subblock scales, and an FP16 base; codes are centered by subtracting 32 |
Q8_0 |
FP16 scale plus 32 signed INT8 values |
Q8_1 |
FP16 scale and sum plus 32 signed INT8 values |
Q8_K |
FP32 scale, 256 signed INT8 values, and 16 signed INT16 subblock sums |
MXFP4 / NVFP4
|
GGUF float4 layouts; not aliases for arbitrary FP4 checkpoints or BitsAndBytes FP4 |
The internal enum also recognizes unquantized/raw F32, F16, BF16, F64, and integer types. Enum recognition is not evidence that a type is a supported GGUFConfig quantization target.
| Path | Source-declared requirements and boundaries |
|---|---|
| Project Python requirement | >=3.10 |
| Project Transformers dependency |
transformers>=5.20.2 in requirements.txt at the documented revision |
| GGUF extra |
pip install 'gptqmodel[gguf]'; declares gguf>=0.19.0 and llama-cpp-python>=0.3.36
|
| Internal reader / documented Bonsai import | Does not require the external gguf PyPI package |
| Native ggml CPU |
llama-cpp-python with loadable ggml-base and ggml-cpu libraries and required symbols |
| Native ggml CUDA | Additionally needs loadable ggml-cuda, available ggml CUDA devices, and a working CUDA build; installing a CPU-only wheel does not supply this |
| GGUF Triton | Installed Triton >=2.0.0; free-threaded/GIL-disabled Python requires Triton >=3.4.0; class advertises CUDA devices on Linux/Windows |
| Triton package extra |
pip install 'gptqmodel[triton]'; current metadata requests triton>=3.8.0 on Python <3.15, a higher floor than the kernel's import-time compatibility check |
| MLX | Apple Silicon (Darwin, arm64), available Metal, mlx and mlx_lm; gptqmodel[mlx] declares mlx>=0.32.4, and mlx_lm>=0.31.3 on Python <3.11 or >=0.32.2 on Python ≥3.11 |
The portable Torch class advertises all device/platform families, but actual operators, dtypes, and optional dequantizers must be supported by the installed stack. Neither the native GGUF_CPP_CUDA class nor the GGUF Triton class declares a separate ROCm backend; do not assume ROCm availability from generic Triton or PyTorch HIP support. See Installation and Quickstart and Inference Backends.
from gptqmodel import GPTQModel
from gptqmodel.quantization import GGUFConfig
from gptqmodel.utils.backend import BACKEND
config = GGUFConfig(bits=4, format="q_k")
model = GPTQModel.load("/models/source-hf", quantize_config=config)
model.quantize(backend=BACKEND.GGUF_TORCH)
model.save("/models/gguf-block-checkpoint")
loaded = GPTQModel.load(
"/models/gguf-block-checkpoint",
backend=BACKEND.GGUF_TORCH,
)The source directory must contain a supported, unquantized HF model. This weight-only lifecycle ignores any supplied calibration dataset: it performs no activation capture, Hessian solve, or calibration replay. Configured preprocessing is planned without forward passes; optional smoothing acts on weight blocks. Adapter/EoRA generation, GPTAQ, and FOEM are rejected in this lifecycle. GGUFTorchLinear.pack_original(...) ignores GPTQ scales, zeros, and g_idx, applies an optional smoother, pads the source width to a complete GGML block, and packs the source weight matrix.
The output is a GPT-QModel checkpoint directory, not a new llama.cpp .gguf artifact. Native C++ inference classes deliberately disable pack, pack_block, pack_gpu, and pack_original; use a packing-capable backend for quantization and select C++ at load time.
from gptqmodel import GPTQModel
# Supply the artifact itself as model_id_or_path.
model = GPTQModel.load("/models/bonsai/model.gguf", profile="low_memory")The public API rejects gguf_file=.... Pass the local file itself, or a local directory / Hub repository with a single unambiguous .gguf file. Automatic single-file inference requires exactly one GGUF and no dense checkpoint files recognized by the resolver. Repositories with several GGUF choices should be downloaded/selected explicitly; do not rely on an arbitrary first file.
The HF adapter resolves the file into its enclosing model location plus an internal gguf_file argument. Transformers still handles the model/config/tokenizer interpretation; the internal GGUF reader is not a universal model-architecture implementation.
internal_gguf.inspect_quantized_checkpoint(...) currently recognizes a packed native-import candidate only when:
-
general.architectureisqwen3; - the recognized attention/MLP projection tensors have exactly one common supported quantization type;
- that type maps to a registered bits alias.
It also records whether output.weight uses that type, so the loader can configure lm_head. Native name mapping additionally requires Transformers' default tensor processor. Unsupported architectures, mixed projection types, or specialized tensor processors must not be assumed to retain packed weights through this path.
For a native packed candidate, the loader creates GGUF linear modules and copies the GGUF byte rows into qweight. It verifies packed shapes, dequantizes tensors that are not being installed as packed linear weights, and checks that required quantized modules were populated. The other HF import path produces dense weights through GGUF dequantization; it is not packed GGUF inference.
| Native import case | Behavior |
|---|---|
Q1_0, profile="auto"
|
Resolves to PROFILE.FAST; the source loader uses dense HF GGUF import |
Q1_0, profile="low_memory"
|
Packed/native-loader route rather than the fast dense route |
Native Q1_0 backend validation |
Allows AUTO, GGUF_TORCH, GGUF_TRITON, MLX; rejects native C++ backend selections |
Native Q2_0
|
AUTO becomes GGUF_TORCH; explicit backend must be GGUF_TORCH or MLX
|
These are model-file loader restrictions, separate from a class's generic bit-width capability table.
The repository's Prism/Bonsai guide documents inference-only support:
from gptqmodel import GPTQModel
model = GPTQModel.load("prism-ml/Bonsai-1.7B-gguf")
# To request the packed low-memory route instead:
# model = GPTQModel.load("prism-ml/Bonsai-1.7B-gguf", profile="low_memory")
tokens = model.generate(
"Who wrote Romeo and Juliet?",
max_new_tokens=128,
)[0]
print(model.tokenizer.decode(tokens, skip_special_tokens=True))Q1_0_g128 remains a compatibility alias for the official 128-element Q1_0 layout. The HF compatibility code supplies internal runtime/Prism hooks when the installed Transformers lacks them. This does not add Prism model quantization support.
| Direction | What the implementation provides |
|---|---|
| Dense HF → GGUF blocks |
GGUFConfig plus the weight-only lifecycle and normal GPT-QModel checkpoint writer |
.gguf → HF model/runtime |
HF integration with internal GGUF reader/dequantizers; packed native import only under the restrictions above |
| GGUF byte rows → Torch dense tensor |
internal_gguf.dequantize_to_torch(...) and linear-layer dequantize_weight()
|
| GGUF byte rows → MLX | Exact packed-layout transcoding for the supported subset, not quantizing already-dequantized weights again |
| GPT-QModel checkpoint → standalone llama.cpp file | No general .gguf writer branch in the reviewed GPTQModel.export implementation; a real model-level converter is required |
GGUF linear model → GPTQModel.export(..., format="hf")
|
Not a general GGUF export route: this branch calls the Torch GPTQ dequantizer, which rejects non-TorchLinear quantized modules |
Do not rename a safetensors file to .gguf, assume every llama.cpp architecture has an HF mapping, or assume quality-suffix presets and packed bytes alone supply tokenizer and architecture metadata.
The native bridge uses llama.cpp's ggml libraries through ctypes; it runs linear operations inside GPT-QModel, not llama.cpp's full tokenization, KV-cache, scheduler, or generation engine.
| Component | Role |
|---|---|
gguf.py::_quantize_gguf_tensor_numpy |
Tries an available external gguf quantizer, then built-in packers, then native ggml quantization |
gguf.py::_fallback_gguf_quantize |
Built-in Q1_0/compatibility, Q2_0, Q4_0, Q8_0, Q4_K, Q5_K, Q6_K, TQ1_0, TQ2_0, MXFP4, Q8_1, Q8_K packing |
gguf_cpp.py::_GGMLBridge.quantize_chunk |
Uses ggml_quantize_chunk; availability depends on installed ggml types/functions |
quantization/mlx_gguf.py::gguf_quantize_weight_mlx(weight, qtype: str) |
Metal packing into GGUF bytes for its explicit supported subset |
utils/mlx_gguf_packing.py::repack_gguf_affine(qweight, qtype, in_features) |
Converts existing GGUF codes/scales/minima into an equivalent packed affine representation |
utils/mlx_gguf_packing.py::repack_gguf_float4(qweight, qtype, in_features) |
Reorders MXFP4/NVFP4 nibbles and scale bytes into native MLX layouts |
utils/internal_gguf.py |
Internal enum/size registry, memory-mapped reader, quantization/dequantization adapter, native checkpoint inspection, minimal Qwen3 name mapping |
Recognized types are broader than the built-in packer. In particular, the built-in NumPy fallback does not pack every IQ/K format or NVFP4. Those cases may require external gguf/native ggml, or the separately supported Metal packer. An unsupported target on the NumPy/native route ultimately raises NotImplementedError; the Metal helper rejects unsupported types with ValueError. Neither route silently changes the quantization type.
If native ggml reports that a quantizer requires an importance matrix, the bridge supplies uniform importance values. That is a valid non-activation-aware fallback, not a calibration-derived importance matrix or a claim of parity with an importance-tuned llama.cpp quantization run.
The Metal packer accepts Q1_0, Q1_0_G128, Q2_0, Q4_0, Q4_K/_S/_M, Q5_K/_S/_M, Q6_K, TQ1_0, TQ2_0, MXFP4, NVFP4, and Q8_0 (case-normalized with upper()). It requires a nonempty rank-two FP16/BF16/FP32 matrix with complete blocks and returns evaluated mx.uint8 byte rows. This low-level function rejects unaligned widths; the higher-level Torch packer can pad them first.
On Darwin arm64 with mlx.core importable, GGUFTorchLinear._pack_weight_tensor(...) automatically selects this Metal packer when the padded weight has at least 2 * 1024 * 1024 elements. Smaller weights use the NumPy/native route. The dispatch does not first filter by tensor type and does not catch an unsupported-Metal-type error to retry another packer. Thus an alias accepted by GGUFConfig can still fail during large-layer packing on Apple Silicon even if an external/native quantizer could handle it elsewhere. Metal packing availability and exact MLX inference transfer support are separate checks.
GGUFReader(path, mode="r") memory-maps the file, validates magic 0x46554747, accepts file versions 2 and 3, reads scalar/string/array metadata, and exposes ReaderTensor byte/data views. Default alignment is 32 bytes; an explicit general.alignment must be a nonzero power of two. Duplicate fields/tensor names and rows not divisible by the quantization block size are rejected. GGUF_VERSION is 3. install_runtime() registers this module as gguf for the HF adapter; it is not the complete external package or a writer.
| Object | Shape/layout |
|---|---|
GGUFTorchLinear.qweight |
torch.uint8 rows: (out_features, padded_in_features // gguf_block_size * gguf_type_size)
|
GGUFTorchLinear.dequantize_weight(...) |
Contiguous (in_features, out_features) matrix, with input padding removed; this is the orientation for x @ weight, not nn.Linear.weight
|
internal_gguf.dequantize_to_torch(...) |
Reconstructs the supplied tensor data without the linear wrapper's final transpose or input-width crop |
The source signatures are:
# GGUFTorchLinear
def dequantize_weight(
self,
*,
device: torch.device | str | None = None,
dtype: torch.dtype | None = None,
) -> torch.Tensor:
...
# gptqmodel.utils.internal_gguf
def dequantize_to_torch(
data: np.ndarray,
qtype: GGMLQuantizationType,
*,
device: torch.device | str | None = None,
dtype: torch.dtype = torch.float32,
) -> torch.Tensor:
...The internal helper defaults to CPU/FP32. The linear wrapper resolves its target from the requested device/dtype and module state; on CPU it reconstructs in FP32 before the final output cast. These helpers explicitly materialize dense values for a consumer; calling one does not change the checkpoint into a dense model or authorize retaining an expanded inference cache.
| Backend enum / string | Class | Scope and restrictions |
|---|---|---|
GGUF_TORCH / gguf_torch
|
GGUFTorchLinear |
General GGUF reference/fallback; uint8 packed weights; optional chunked paths; advertises all devices/platforms |
GGUF_CPP_CPU / gguf_cpp_cpu
|
GGUFCppKernel |
CPU; selects native ggml versus Torch per shape/dtype/type through benchmarking or an explicit override |
GGUF_CPP_CUDA / gguf_cpp_cuda
|
GGUFCudaKernel |
CUDA ggml graph/plan execution; requires matching native libraries and supported tensor types |
GGUF_TRITON / gguf_triton
|
GGUFTritonKernel |
CUDA; only Q1_0/compatibility, Q4_K, Q5_K, Q6_K
|
MLX / mlx
|
GGUFMlxQuantLinear checkpoint holder; MlxGGUFLinear / MlxGGUFQ6KLinear runtime |
Apple Silicon/Metal; exact packed transfer subset and alignment constraints |
The Torch, C++ CPU/CUDA, and Triton classes accept FP16, BF16, and FP32 activations in their declarations. This does not imply identical internal precision: Triton converts non-FP16 inputs to FP16 for its fused matmul, then casts the output back to the original dtype. Bias and supported adapters are applied by the wrappers.
| Class | Advertised training support | Advertised adapters |
|---|---|---|
GGUFTorchLinear |
True |
Lora |
GGUFCppKernel, GGUFCudaKernel
|
False |
Lora |
GGUFTritonKernel |
False |
Lora |
GGUFMlxQuantLinear |
False |
None |
These declarations describe kernel validation contracts, not a general promise that every GGUF format supports every training workflow.
GGUFTorchLinear rounds input width up to the tensor block size, stores weights as byte rows, and pads activations for computation. Its portable forward reconstructs weights for the consuming matmul. Chunked fused paths reconstruct bounded blocks and accumulate their contributions rather than requiring a persistent dense weight cache. The packed input and its scale/index metadata may persist; a dequantized full-weight cache is not the intended storage model.
The class has CPU/CUDA row/matrix-size gates and autotuning. Performance depends on row count, input/output widths, dtype, quantization type, threading, and installed kernels; a backend name alone is not a speed guarantee.
CPU native ggml quantized dot-product/SIMD paths can quantize the activation operand to a Q8 dot-product representation. This adds activation quantization error beyond GGUF weight quantization. Native ggml output therefore need not match x @ dequantized_weight.T with floating activations, even for identical packed weights. Treat that arithmetic difference separately from packer/dequantizer correctness; report weight-only reference error separately from native activation-quantized error. The actual SIMD implementation depends on the installed llama.cpp build and host hardware.
GGUFCppKernel benchmarks native versus Torch on first use of a (rows, dtype, qtype) key; failures select Torch. GPTQMODEL_GGUF_CPP_NATIVE_MATMUL=0 forces the Torch arithmetic path within this backend. Set it explicitly when a weight-only oracle is required; a forced native path is not an accuracy-equivalent optimization.
Q8_1 and Q8_K are special safety cases. They are valid stored layouts with built-in packing/dequantization, but upstream ggml does not expose the corresponding safe weight quantizers/vec-dot kernels used by this bridge. Native quantization excludes both to avoid GGML_ASSERT aborts. The native CPU matmul entry point rejects both to avoid missing-function-pointer failures, and the CPU dispatcher returns Torch even if native matmul is forced. Do not assume this CPU guard protects CUDA bridge/plan calls: explicitly choose GGUF_TORCH for these weights, and never force them through unsupported native quantization or matmul calls.
Bridge initialization is locked; CPU execution also locks shared mutable ggml graph scratch. CUDA plans retain packed weights and execution buffers, are keyed by device/rows/dtype/weight pointer, and are closed by clear_weight_cache(). CUDA BF16 input is converted to FP16 for the ggml operand. These graph mechanics do not remove the need to validate arithmetic on the target build.
Triton fuses packed code decoding with matmul for the explicitly supported types. It uses FP32 accumulators and FP16 matmul operands, then stores an FP16 kernel result before the wrapper restores the original activation dtype. Returning FP32/BF16 therefore does not remove the intermediate FP16 rounding. Its caches hold packed planes and scale/minimum metadata, not a dense floating matrix. Q1_0 has shape/capability-specific launch choices for CUDA capabilities (8, 0) and (8, 9), including packed-word and K=2048 decode specializations. Other supported configurations use the general/autotuned choices; these specializations are not a claim that every CUDA device has the same performance.
Exact transfer supports Q1_0/compatibility, Q2_0, Q4_0, Q8_0, Q4_K, Q5_K, Q6_K, TQ1_0, TQ2_0, MXFP4, and NVFP4. The holder requires 32-aligned input width, output width divisible by eight, complete unpadded source blocks, and no adapter. Its loader dtype declaration is FP16; runtime wrappers preserve the input activation dtype, including FP16/BF16 operation where supported.
| Source layout | MLX representation |
|---|---|
Q1_0 |
Two-bit packed affine codes, group 128, scale 2*d, bias -d
|
Q2_0 |
Two-bit affine, group 64 |
Q4_0 |
Four-bit affine, group 32, bias -8*d
|
Q8_0 |
Eight-bit affine after signed-to-unsigned code shift, group 32 |
Q4_K, Q5_K
|
Packed affine codes with group-32 scales/minima |
Q6_K |
Exact group-16 affine representation and a dedicated packed Metal execution option |
TQ1_0, TQ2_0
|
Two-bit affine representation with group-128 scales |
MXFP4, NVFP4
|
Native MLX float4 modes; groups 32 and 16 respectively |
MlxGGUFQ6KLinear uses a packed Q6_K kernel for decode and selected small BF16 prefill cases, and MlxGroup16Linear for other cases. MlxGGUFLinear wraps the general quantized linear and restores activation output dtype. Unsupported layouts or padding are not silently requantized to another type.
| Environment variable | Default / effect |
|---|---|
GPTQMODEL_GGUF_CPP_THREADS |
Otherwise max(1, torch.get_num_threads()); controls native CPU thread count |
GPTQMODEL_GGUF_CPP_NATIVE_MATMUL |
Unset: benchmark selection; 0, false, off: Torch; other values request native except excluded types |
GPTQMODEL_GGUF_FUSED_CUDA_MAX_ROWS |
32 |
GPTQMODEL_GGUF_FUSED_CUDA_MIN_MATRIX_ELEMENTS |
8388608 |
GPTQMODEL_GGUF_FUSED_CPU_MAX_ROWS |
64 |
GPTQMODEL_GGUF_FUSED_CPU_MIN_MATRIX_ELEMENTS |
0 |
GPTQMODEL_GGUF_FUSED_CHUNK_BLOCKS |
8, clamped to at least one |
GPTQMODEL_GGUF_FUSED_AUTOTUNE_WARMUP |
1 |
GPTQMODEL_GGUF_FUSED_AUTOTUNE_ITERS |
2, clamped to at least one |
GPTQMODEL_GGUF_FUSED_AUTOTUNE_MARGIN |
0.05 |
GPTQMODEL_INTERNAL_GGUF_QUANTIZED_LOADER |
Enabled if unset; empty/0/false/off/no disables packed candidate detection |
GPTQMODEL_INTERNAL_GGUF_DEQUANT_DEVICE |
Optional CPU/CUDA device for the sign-only NumPy-facing dequantization acceleration; invalid/unavailable devices fall back |
GPTQMODEL_INTERNAL_GGUF_DEQUANT_MAX_BYTES |
268435456 bytes (256 MiB) of FP32 output per sign-only chunk, with at least one row |
GPTQMODEL_INTERNAL_GGUF_TORCH_LOADER |
Controls the direct Torch HF import helper; if unset, a nonempty internal dequant-device setting enables that helper |
The fused class defaults are read at module import. Set environment controls before importing/loading. These internal tuning variables do not widen format or architecture coverage.
Common mistakes:
- Selecting a backend by nominal width while ignoring the exact tensor type.
- Assuming a successfully parsed alias guarantees a built-in packer, native kernel, or packed model-file importer.
- Treating
group_size=-1as a single GGUF block spanning the entire row. - Comparing native activation-Q8 arithmetic to a floating-activation weight-only oracle without separating the errors.
- Expecting MLX to accept padded GGUF rows merely because Torch can pad them.
- Expecting
Q1_0default fast loading to have the same memory footprint as explicit low-memory packed loading. - Expecting dequantization to recover pre-quantization HF weights; it only reconstructs the stored quantized values.
Paths below are relative to the repository and link to the documented revision.
| Source | Responsibility |
|---|---|
| gptqmodel/quantization/config.py | Enums, aliases, GGUFBits, config normalization and serialization |
| gptqmodel/utils/backend.py | Backend and profile enums |
| gptqmodel/utils/internal_gguf.py | Internal reader, types, dequantization, native checkpoint inspection |
| gptqmodel/utils/mlx_gguf_packing.py | Exact MLX packed transcodes |
| gptqmodel/quantization/mlx_gguf.py | Metal GGUF packers |
| gptqmodel/nn_modules/qlinear/gguf.py | Layout registry, pack/dequantize logic, Torch linear |
| gptqmodel/nn_modules/qlinear/gguf_cpp.py | Native ggml SIMD/dequantization and CPU/CUDA graph execution |
| gptqmodel/nn_modules/qlinear/gguf_triton.py | Fused Triton inference |
| gptqmodel/nn_modules/qlinear/mlx_gguf.py | MLX runtime wrappers and packed Q6_K kernel |
| gptqmodel/nn_modules/qlinear/mlx.py | MLX source-layout capability validation |
| gptqmodel/utils/hf.py | Artifact discovery, public argument handling, Transformers compatibility |
| gptqmodel/models/loader.py | Native packed import, profiles, tensor mapping |
| gptqmodel/models/auto.py | Public load/export entry points |
| gptqmodel/nn_modules/qlinear/torch.py | Generic HF export dequantizer's TorchLinear restriction |
| docs/prism-bonsai-gguf.md | Prism inference and Q1_0 compatibility |
| pyproject.toml, requirements.txt | Optional extras and dependency floors |