Repository navigation
EXL3
Documentation home · Quickstart · Lifecycle
On this page
EXL3 is GPT-QModel's integration of the ExLlamaV3 trellis-quantized weight format. It is a separate quantization method and checkpoint format, not a GPTQ packing option:
| Identifier | Value | Meaning |
|---|---|---|
METHOD.EXL3 |
"exl3" |
Select the EXL3 quantization algorithm |
FORMAT.EXL3 |
"exl3" |
Serialize EXL3 trellises and their accompanying tensors |
EXL3Config |
Configuration class | EXL3-specific settings, plus inherited model/looper controls |
BACKEND.EXL3_EXLLAMA_V3 |
"exl3_exllama_v3" |
Vendored CUDA/HIP extension runtime |
BACKEND.EXL3_TORCH |
"exl3_torch" |
Torch reconstruction and multiplication runtime |
BACKEND.MLX |
"mlx" |
Explicit Apple Silicon/Metal inference route |
Portions of gptqmodel/exllamav3/, the processor, and the runtime utilities are adapted from TurboDerp / ExLlamaV3. The native MLX kernel headers also identify the upstream EXL3 format and its MIT provenance. GPT-QModel supplies model traversal, calibration capture, module replacement, configuration, saving/loading, and additional Torch/MLX implementations. An EXL3 checkpoint should not be described as an ordinary GPTQ checkpoint that happens to use ExLlama kernels: its tensors and loader are different.
This page describes the implementation in source revision eb90dbdd5416c0c8e50d1b64f1ac562012afe4aa. In particular, distinguish available low-level MLX components from the public quantization entry point, which currently requires CUDA/HIP.
Reading the diagram: The public quantization entry point requires CUDA/HIP and calibration. Torch/MLX inference does not enable CPU/MLX quantization through that entry point.
Text version of the workflow
- Per-layer, per-subset solve: Capture selected modules → Solve EXL3 trellises → Replay solved weights → Finalize EXL3 modules.
- Load an EXL3 artifact: Read EXL3 configuration → Rebuild packed modules → Run the selected runtime.
EXL3 does not store an integer code for every weight together with GPTQ-style group scales and zero points. It stores a packed trellis representation and input/output sign-scale tensors. Quantization works in a transformed weight space using randomized signs, block Hadamard transforms, regularization, and Hessian-aware error compensation. The decoder reconstructs codebook values from trellis states.
| Component | Purpose |
|---|---|
trellis |
Packed state information, stored as torch.int16 with shape (in_features // 16, out_features // 16, 16 * K) for integer bitrate K
|
suh, svh
|
FP16 input/output sign-scale vectors used with Hadamard transforms |
su, sv
|
Legacy packed-sign alternatives; runtimes can unpack these to sign vectors |
mcg |
Optional marker for the mcg decoder family |
mul1 |
Optional marker for the mul1 decoder family |
bias |
Optional linear bias |
The codebook names accepted by EXL3Config are "mcg", "mul1", and "3inst"; the default is "mcg". These are decoder families, not user-provided learned centroid tables. The mcg and mul1 choices are conveyed by marker tensors. The no-marker case uses the 3inst decoder. The Torch implementation has matching decoder arithmetic for all three.
The block-128 Hadamard layout requires compatible input and output dimensions. In particular, the Torch transform helpers and the native MLX pipeline require both feature dimensions to be divisible by 128. Module-shape discovery alone does not establish that a convolution or a small projection is quantizable with this layout; do not assume automatic padding.
The configuration accepts fractional bits per weight. The normalization signature is:
def _normalize_exl3_bits(bits: Union[int, float, str]) -> float:| Input | Normalization or error |
|---|---|
Integer, such as 3
|
Converted to 3.0
|
Float, such as 3.5
|
Retained as a float |
String, such as " 3.5 "
|
Stripped and parsed with float()
|
| Non-finite float | Rejected |
Value outside [1.0, 8.0]
|
Rejected |
| Other object type | Rejected |
The integer test uses Python's isinstance(bits, int), so True also normalizes to 1.0; False normalizes to zero and fails the range check. Use explicit numeric targets rather than booleans.
Acceptance of a fractional value does not mean the current processor packs a single matrix at that fractional bitrate. EXL3Config.runtime_bits calls quant_bits_width(), whose float path is max(1, int(math.floor(bits))). EXL3Processor._target_bits() uses that integer as quantizer argument K. A configured bits=3.5 therefore produces K=3 for ordinary modules in this implementation. A head-specific head_bits value is converted with max(1, int(head_bits)), also truncating these positive values.
The original float is serialized in the config, while each module's tensor_storage describes its actual integer bits_per_weight. This matters when comparing configuration files, packed tensor sizes, and measured model-wide storage. Scale vectors, optional bias, metadata, skipped modules, and unquantized parameters also prevent total checkpoint size from being simply the parameter count multiplied by bits.
There is no automatic mixed-integer allocation implied by a fractional bits value in this processor. Use explicit module overrides when different modules need different integer targets.
from gptqmodel import GPTQModel
from gptqmodel.quantization import EXL3Config
qcfg = EXL3Config(
bits=3.0,
codebook="mcg",
out_scales="auto",
device="cuda:0",
)
model = GPTQModel.load("/path/to/dense-model", quantize_config=qcfg)
# Supply representative text or the dataset form supported by prepare_dataset.
calibration = [
"This example document contains several complete sentences about the kinds "
"of questions and explanations that the model will encounter during deployment.",
"Use many real documents from the intended domain for an actual run, keeping "
"the calibration corpus separate from the data used to evaluate quality.",
]
model.quantize(calibration, batch_size=1)
model.save("/path/to/exl3-model")The short dataset above illustrates the call shape, not a calibration-quality recommendation. Dataset preparation removes samples with at most calibration_data_min_length tokens (default 10); use sufficiently long, representative inputs. EXL3 requires calibration; the processor rejects a missing dataset and also rejects a module for which no activation samples were collected. See Quantization Lifecycle for dataset preparation and sequencing.
On ROCm, PyTorch uses the cuda device type for HIP tensors too. A ROCm device is consequently specified as "cuda:0" in this example. Both the PyTorch runtime and the EXL3 extension must support the installed accelerator stack.
The following are the fields declared by EXL3Config itself. Shared controls such as device, dynamic, lm_head, offload, and metadata are inherited from BaseQuantizeConfig; see QuantizeConfig Reference. Unlike FP8Config, GGUFConfig, and BitsAndBytesConfig, EXL3 does not inherit the PreProcessorConfig constructor fields (preprocessors, smoother, smooth). Keep the inherited rotation field unset: the loader accepts shared model-rotation metadata only for GPTQ/GPTQ_V2 checkpoints. EXL3's internal sign/Hadamard transforms are part of its own format, not that shared rotation workflow.
| Field | Declared type | Default | Behavior |
|---|---|---|---|
bits |
float |
3.0 |
Normalized to finite float in [1.0, 8.0]; runtime K is floored |
method |
METHOD |
METHOD.EXL3 |
Only exl3 accepted |
format |
FORMAT |
FORMAT.EXL3 |
Only exl3 accepted |
group_size |
int |
-1 |
Forced to -1; not a GPTQ group-size setting |
desc_act |
Optional[bool] |
False |
Forced to False
|
sym |
bool |
True |
Forced to True
|
head_bits |
Optional[float] |
None |
Optional target for a selected lm_head; otherwise uses the module's effective bits after dynamic overrides |
out_scales |
Optional[str] |
"auto" |
Controls output-scale decision passed to the quantizer |
codebook |
str |
"mcg" |
mcg, mul1, or 3inst, case/whitespace normalized |
tensor_storage |
Optional[Dict[str, Any]] |
None |
Per-module storage metadata; populated on save and required on load |
calibration |
Optional[Dict[str, int]] |
None |
Dictionary normalized to string keys and integer values; not the calibration dataset argument |
EXL3Config.allowed_quant_methods() returns (METHOD.EXL3,); supported export formats are the EXL3 export-format tuple. Changing method or format does not convert the config into another method.
out_scales input |
Canonical config value | Quantizer argument |
|---|---|---|
"always", "true"
|
"always" |
True |
"never", "false"
|
"never" |
False |
"auto", "none"
|
"auto" |
None |
Python None
|
None |
None |
Other strings raise ValueError. These are normalized requests to the regularization stage; they are not a separate checkpoint scale format. The quantizer's regularization code makes the final output-scale decision, including handling exceptional zero-scale cases.
from gptqmodel.quantization import EXL3Config
qcfg = EXL3Config(
bits=3.0,
device="cuda:0",
lm_head=True,
head_bits=6.0,
dynamic={
r"^lm_head$": {"bits": 6.0, "group_size": -1},
r".*\.self_attn\.(q_proj|k_proj)$": {"bits": 4.0},
r".*\.mlp\.down_proj$": {"bits": 4.0, "out_scales": "always"},
},
)The patterns must match the model's actual module names. head_bits only affects a head that is actually selected for quantization; it does not enable head quantization by itself.
Keep the explicit head override in this revision. The shared ModuleLooper.loop() inserts GPTQ-oriented head defaults (bits=8, group_size=32, sym=True, desc_act=False, mse=2.4) when lm_head=True and no dynamic entry matches the head. EXL3's cloned configuration rejects that group_size=32. head_bits alone does not suppress this injection. A matching entry such as the ^lm_head$ example prevents it; use the actual fully qualified head name for composite models. The shared looper also rejects head quantization while the head weight is tied to another parameter.
clone_exllamav3_config_for_module(qcfg, module_full_name) resolves dynamic selection, returns None for a disabled module, deep-copies the base config, applies overrides for bits, head_bits, out_scales, codebook, and calibration, then reruns __post_init__(). This avoids leaking one module's override into another. The dynamic normalizer validates bits and head_bits; a dynamic group_size must be -1 or None. It is not an escape hatch for GPTQ-style grouped EXL3.
from gptqmodel import GPTQModel, BACKEND
# AUTO selects the extension for a CUDA target when CUDA is available; on
# ROCm AUTO resolves to EXL3_TORCH (the native GEMM table is nullptr for HIP).
# This does not test whether the EXL3 extension can build or load.
model = GPTQModel.load("/path/to/exl3-model", device="cuda:0")
# Explicit Torch fallback on CPU.
cpu_model = GPTQModel.load(
"/path/to/exl3-model", device="cpu", backend=BACKEND.EXL3_TORCH,
)
# Explicit MLX route on a supported Apple Silicon installation.
mlx_model = GPTQModel.load(
"/path/to/exl3-model", device="mps", backend=BACKEND.MLX,
)These are alternative loading examples, not instructions to keep three model copies resident. Standard generation usage is covered in Loading and Inference.
| Operation/runtime | Requirements | Important behavior |
|---|---|---|
Public model.quantize() for EXL3 |
CUDA/HIP available; quantization device type cuda; backend AUTO or EXL3_EXLLAMA_V3
|
Public validation currently rejects CPU/MPS EXL3 quantization |
EXL3Processor low-level MLX branch |
Darwin arm64, importable mlx.core, target CPU or MPS |
Implemented internally, but the public EXL3 quantization guard above runs first |
ExllamaV3Linear |
Materialized CUDA/HIP packed tensors and EXL3 extension | Lazily constructs the vendored inner runtime; rejects meta/CPU tensors |
ExllamaV3TorchLinear |
Torch operations supported by the selected device | Reconstructs for the current operation without retaining a dense weight cache |
| Native MLX inference | Apple Silicon/Metal, MLX and MLX-LM integration | Uses packed trellises and Metal kernels; explicit BACKEND.MLX
|
The loader accepts only AUTO, EXL3_EXLLAMA_V3, EXL3_TORCH, or MLX for FORMAT.EXL3 after backend alias normalization. AUTO chooses the extension only when CUDA is available and the selected target family is CUDA; on ROCm — CDNA and RDNA4 alike — AUTO resolves to EXL3_TORCH because the extension's tensor-core GEMM dispatch table is nullptr for HIP. It otherwise chooses Torch. This selection does not probe extension availability: a subsequent extension build/import failure does not automatically retry with Torch. Select BACKEND.EXL3_TORCH explicitly to avoid the extension. The general Apple Silicon auto-selector deliberately does not auto-select MLX for EXL3 in this revision.
gptqmodel/exllamav3/ext.py exposes exllamav3_runtime_available(), exllamav3_runtime_error(), and prewarm_exllamav3_extension(). The extension is named gptqmodel_exllamav3_ops; it is built from the vendored sources under gptqmodel_ext/exllamav3/ using Torch's extension machinery. A compatible GPU-enabled PyTorch installation, compiler/toolkit, and supported architecture are needed. Merely having the Python files is not enough.
| Environment variable | Purpose |
|---|---|
GPTQMODEL_EXT_VERBOSE |
Extension build verbosity |
GPTQMODEL_EXT_DEBUG |
Debug build controls |
GPTQMODEL_EXLLAMAV3_BUILD_ROOT |
Override extension build/cache root |
GPTQMODEL_EXLLAMAV3_FORCE_REBUILD |
Force rebuilding instead of reusing the build |
TORCH_CUDA_ARCH_LIST |
CUDA architecture selection; the extension can derive this when unset |
The extension checks for required operations including had_paley, had_paley2, quantize_tiles, pack_trellis, unpack_trellis, pack_signs, reconstruct, had_r_128, hgemm, and bc_linear_exl3_run. Build/import failures should be diagnosed through the runtime error helper rather than assuming a model-format error. See Installation and Quickstart and Troubleshooting and FAQ.
The processor constructor is:
def __init__(
self,
tokenizer,
qcfg: QuantizeConfig,
calibration,
prepare_dataset_func,
calibration_concat_size: Optional[int],
calibration_sort: Optional[str],
batch_size: int,
require_fwd: bool = True,
calibration_concat_separator: Optional[str] = None,
lm_head_name: str = "lm_head",
):EXL3Processor subclasses LoopProcessor, reports the name "exl3", and requests forward capture, fwd_replay_after_process=True, and subset_forward_early_stop=True.
The calibration dataset is fixed at construction. set_calibration_dataset() raises NotImplementedError rather than replacing a capture in progress.
| Stage | Implementation behavior |
|---|---|
| Module selection | Clone the EXL3 config for the module; honor dynamic exclusion |
| Capture setup | Construct a GPTQ capture task with GPTQConfig(bits=max(1, runtime_bits), group_size=-1, desc_act=False, sym=True, act_group_aware=False) and the selected device/pack dtype |
| Forward hooks | Feed activation batches into the capture task; the GPTQ object is used for activation/Hessian collection, not GPTQ weight export |
| Hessian preparation | Finalize the captured Hessian on the target device; require a nonempty capture and positive sample count |
| Weight layout | Present the EXL3 quantizer with (in_features, out_features) weights; transpose/restore ordinary linear weights as needed |
| Quantization | Call native EXL3 with one device, integer K, sigma_reg=0.025, seed 787, output-scale policy, and codebook flags; alternatively use the internal MLX bridge |
| Replay preparation | Stage packed tensors to CPU through the module stream-state payload and install reconstructed weights for the current sequential replay |
| Submodule finalization | Collect EXL3 buffers and replace the dense module with create_exllamav3_module()
|
| Model finalization | Set quantized=True, method/format to EXL3, and qlinear_kernel=ExllamaV3Linear
|
The captured Hessian payload passed to the CUDA/HIP quantizer is marked single_use=True. It is not retained for cross-module factor caching by this processor. The sequential replay lets later subsets observe already-quantized earlier projections; it is not an inference-time dense-weight cache.
The public EXL3 branch rejects adapter/EoRA generation, GPTAQ/native activation capture, and FOEM/native activation capture with NotImplementedError.
The main lower-level entry point is:
def quantize_exl3(
weight: torch.Tensor,
H_data: dict,
quant_args: dict,
return_weight_q: bool,
progress_str: str | None = None,
verbose: bool = False,
swap_to_device: torch.device | None = None,
save_reg: str = None
):It returns reconstructed quantized weights when requested, a proxy error, and packed output tensors. The main stages are:
- Normalize the Hessian by sample count, inspect its diagonal, and choose fallback for unusable capture statistics.
- Apply diagonal regularization, random signs, and Hadamard transforms; compute a block LDL factor for Hessian-aware quantization when applicable.
- Regularize transformed weight ranges and choose whether to use output scales.
- Select a global scale with golden-section search over sampled tiles.
- Quantize tensor-core-layout tiles through trellis/Viterbi kernels, using LDLQ error compensation or the fallback path.
- Reconstruct for the processor's replay/error reporting, then pack the trellis and emit scales/signs and codebook markers.
The tile encoder uses 256 encoded states per tile. pack_trellis() converts a three-dimensional encoded int16 tensor with last dimension 256 into packed shape (shape[0], shape[1], 256 * K // 16). The runtime reads K back as trellis.shape[-1] // 16.
Per-thread quantization streams and per-stream temporary workspaces are used to avoid accidental serialization across worker threads. The quantizer also contains multi-GPU tile-distribution machinery, but EXL3Processor._build_quant_args() supplies a one-device list per module. Do not interpret those lower-level helpers as a promise that one module is automatically split across every visible GPU.
| Implementation | Execution and storage |
|---|---|
gptqmodel/nn_modules/exllamav3.py — ExllamaV3Linear
|
Torch module holding EXL3 buffers, a meta placeholder weight, and a lazy LinearEXL3 runtime; forward() calls the inner implementation on x.half() and casts the result back to the input dtype |
gptqmodel/exllamav3/modules/quant/exl3.py — LinearEXL3
|
NVIDIA small-batch BC_LinearEXL3 route, or reconstruct + Hadamard + GEMM route; default reconstruct threshold on NVIDIA is batch size greater than 32 |
| Same vendored runtime on ROCm | Always uses reconstruction and hgemm/hipBLAS because the small-batch tensor-core binding is NVIDIA-only |
gptqmodel/nn_modules/exllamav3_torch.py — ExllamaV3TorchLinear
|
Decodes the current trellis in bounded output chunks, applies block-128 Hadamard transforms, multiplies, adds bias, and casts to input dtype; no persistent dense weight cache |
gptqmodel/nn_modules/qlinear/mlx_exl3.py — MlxEXL3Linear
|
Native Metal row/matrix kernels over packed trellises, including codebook state decoding and optional output/bias fusion |
The EXL3 wrapper files are under nn_modules/, not nn_modules/qlinear/exllamav3.py or nn_modules/qlinear/exllamav3_torch.py in this revision.
The extension reconstruction path allocates an inner FP16 weight for the consuming multiplication; get_inner_weight_tensor() does not retain it on the module. The Torch fallback also reconstructs for the current operation rather than caching a dense model. Its CPU arithmetic uses FP32, with a conditional oneDNN BF16 path when the CPU and problem size qualify. That path requires AVX512 BF16 and MKLDNN support, is enabled by default through GPTQMODEL_EXL3_CPU_BF16X3, and uses a size threshold of either 1 << 19 weight elements or 1 << 23 multiply-accumulates. Output decoding is chunked in groups of 32 output tiles.
ExllamaV3TorchLinear.optimize(backend="inductor", mode=None, fullgraph=False) is an explicit CPU compilation opt-in. A compiled-forward error falls back to eager execution. It should not be confused with a different checkpoint format or an unconditional compilation step during loading.
The lower-level native quantization signature is:
def exl3_quantize_weight_mlx(
weight,
hessian,
input_signs,
output_signs,
*,
sample_count: int,
bits: int,
codebook: str = "3inst",
force_output_scales=None,
sigma_reg: float = 0.025,
workspace_bytes: int = 256 << 20,
):Note that this low-level function's codebook="3inst" default differs from EXL3Config.codebook="mcg"; the processor passes its configured choice explicitly.
| Input/setting | Native MLX contract |
|---|---|
weight |
Nonempty rank-2 FP32 array in (in_features, out_features) layout; both dimensions divisible by 128 |
hessian |
FP32 square matrix matching the input-feature dimension |
input_signs |
FP32 (in_features, 1)
|
output_signs |
FP32 (1, out_features)
|
bits |
Integer 1 through 8; booleans rejected |
codebook |
3inst, mcg, or mul1
|
workspace_bytes |
Positive integer, default 256 << 20
|
The return tuple contains reconstructed weight, proxy error, packed trellis, FP16 input scales, FP16 output scales, fallback decision, selected global scale, and output-scale decision. The Torch bridge returns the reconstructed weight, a Python proxy-error value, and a dictionary of tensors; it seeds input/output signs on CPU and uses DLPack transfers.
| Module family | Responsibility |
|---|---|
mlx_exl3.py |
Viterbi tile encoding, state decode, trellis packing |
mlx_exl3_tiles.py |
Tensor-core tile layout transforms |
mlx_exl3_hadamard.py |
Block-128 Hadamard transforms |
mlx_exl3_hessian.py, mlx_exl3_block_ldl.py
|
Hessian finalization, transforms, block LDL |
mlx_exl3_rms.py, mlx_exl3_regularize.py
|
RMS/range regularization and scale selection |
mlx_exl3_gss.py |
Global-scale search |
mlx_exl3_ldlq.py, mlx_exl3_fallback.py
|
Hessian-aware or fallback tile quantization |
mlx_exl3_pipeline.py |
End-to-end orchestration and Torch bridge |
MlxEXL3Linear(in_features, out_features, bits, codebook="3inst", bias=False) requires positive input/output dimensions divisible by 128 and integer bitrates 1–8. Its activation inputs may be FP16, BF16, or FP32; the result is cast back to the activation dtype. gptqmodel/utils/mlx.py transfers packed trellises, scales, and codebook information and can decode legacy packed signs. MLX inference availability does not bypass the public quantization guard documented above.
gptqmodel/utils/exllamav3.py owns these operations:
| Function | Role |
|---|---|
_resolve_linear_shape(submodule) |
Resolve features from NamedModule state, nn.Linear, convolution/Transformers Conv1D, or feature attributes |
create_exllamav3_module(*, module_root, name, submodule, tensors, module_cls=ExllamaV3Linear) |
Build from packed tensors and replace the named module |
build_exllamav3_tensor_storage(model) |
Collect tensor_storage_entry() from EXL3 modules |
replace_exllamav3_placeholders(*, model, module_names, tensor_storage=None, module_cls=ExllamaV3Linear) |
Replace supported dense placeholders before loading tensor values |
On save, the writer rebuilds tensor_storage from the actual modules. Entries contain stored_tensors keyed by full tensor names with shape and torch_dtype, quant_format="exl3", actual integer bits_per_weight, and optional mcg_multiplier/mul1_multiplier. This metadata allows the loader to allocate the correct packed buffers before loading checkpoint shards.
On load, a nonempty quantization_config.tensor_storage dictionary is mandatory. The loader chooses ExllamaV3TorchLinear placeholders for EXL3_TORCH or MLX and ExllamaV3Linear for the extension route. Keeping only a bits value and removing the tensor metadata is insufficient to recreate an EXL3 checkpoint.
The vendored util/ helpers provide architecture-list handling (arch_list.py), Hadamard support (hadamard.py), memory/tensor workspaces (memory.py, tensor.py), miscellaneous helpers (misc.py), and progress reporting (progress.py). These support the integration; they are not alternate public quantization methods.
EXL3 and VPTQ both use calibration/Hessian information and nonuniform weight representations, but they have separate configurations, processors, tensor layouts, and runtimes. VPTQ's implementation learns vector codebooks using weighted k-means and optionally residual codebooks; EXL3 uses its trellis/codebook decoder families. The EXL3 path does not call VPTQ's quantizer or interpret VPTQ index tensors. A VPTQ checkpoint is not an EXL3 checkpoint, and selecting an EXL3 backend is not a VPTQ conversion.
docs/exl3-mi350x-quant-lifecycle.md is a profiling record for a specific ROCm run. It explicitly excludes inference kernels. Its figures are not general throughput guarantees.
| Measured item | Recorded value |
|---|---|
| Accelerator | One MI350X, gfx950, 256 CUs |
| Model |
tiny-qwen2-base, hidden size 896, six layers, 42 quantized modules |
| Quantization | EXL3 4-bit, K=4, mcg
|
| Calibration | Eight samples, 136 tokens |
| Steady-state wall time | 7.97–8.02 seconds |
| Warm run | 14.6 seconds |
| Dominant GPU kernel |
quantize_tiles_kernel<4, 1>, 91.9% of profiled GPU time |
| Mean concurrent tile kernels in the trace | Approximately 1.155 |
The record explains why a latency-bound trellis kernel can leave a large GPU underoccupied, why isolated multi-stream launches scale better than the complete layer pipeline, and how sequential subsets/replay create synchronization barriers. Per-layer attention and MLP subsets have different numbers of independent chains. GPU-time sums include overlapping kernels and must not be equated with wall time. Optimization conclusions in that document concern scheduling, stream overlap, workspace, and occupancy for that workload—not EXL3 inference performance or model quality.
| Symptom or assumption | What to check |
|---|---|
Fractional bits apparently produces an integer-sized trellis |
Expected in this processor: runtime K is floored; inspect per-module metadata |
Setting group_size, desc_act, or sym changes nothing |
EXL3 forces -1, False, and True; these are not GPTQ tuning knobs here |
head_bits has no effect |
Ensure the head is actually selected (lm_head and model definition/module selection) |
Enabling lm_head raises an EXL3 group_size error |
Add an explicit matching dynamic head entry with group_size=-1; the shared looper's injected GPTQ head defaults use 32 |
| Head quantization fails on a tied-weight model | Untie the head weight first; the shared looper rejects tied head parameters |
| CPU/MPS EXL3 quantization fails despite MLX pipeline files | Public quantize() still requires CUDA/HIP in this revision |
| CPU inference raises a CUDA/HIP error | Load with EXL3_TORCH, not the extension wrapper |
AUTO chooses an extension that cannot build |
AUTO checks the target device, not extension build success; request EXL3_TORCH explicitly |
EXL3 tensors are on meta
|
Materialize checkpoint tensors before forward/post-initialization |
| Load fails before weights are read | Preserve nonempty tensor_storage metadata and valid EXL3 method/format |
| Missing calibration capture | Supply a dataset and ensure selected modules execute during capture |
| CUDA works but ROCm takes a different inference path | ROCm intentionally uses reconstruct + GEMM rather than NVIDIA's small-batch binding |
| BF16 input is assumed to imply BF16 extension arithmetic | The extension wrapper converts input to FP16 and converts output back |
| Adapter, GPTAQ, or FOEM requested during EXL3 quantization | Unsupported in the public EXL3 processor branch |
| An upstream ExLlamaV3 artifact is assumed interchangeable solely by its name | Validate GPT-QModel's expected tensor layout and tensor_storage; format provenance alone is not a loader guarantee |
-
gptqmodel/quantization/config.py— EXL3 enums, normalization, config, serialization, integer runtime width. -
gptqmodel/looper/exllamav3_processor.py— capture, per-module config, quantizer dispatch, replay and replacement. -
gptqmodel/looper/module_looper.py— shared head selection, injected dynamic defaults, and tied-head validation. -
gptqmodel/utils/exllamav3.py— module construction and tensor-storage metadata. -
gptqmodel/nn_modules/exllamav3.py— extension-backed Torch module. -
gptqmodel/nn_modules/exllamav3_torch.py— Torch runtime and CPU optimization. -
gptqmodel/exllamav3/ext.py— extension loading/building and runtime diagnostics. -
gptqmodel/exllamav3/modules/quant/exl3.py— vendored runtime dispatch. -
gptqmodel/exllamav3/modules/quant/exl3_lib/quantize.py— Hessian finalization, regularization, LDLQ, scale search, packing, streams. -
gptqmodel/exllamav3/util/{arch_list,hadamard,memory,misc,progress,tensor}.py— runtime/quantization support. -
gptqmodel_ext/exllamav3/— native extension sources. -
gptqmodel/quantization/mlx_exl3*.py— native MLX quantization stages listed above. -
gptqmodel/nn_modules/qlinear/mlx_exl3.py— Metal inference. -
gptqmodel/utils/mlx.py— MLX model transfer and packed payload handling. -
gptqmodel/models/base.py,loader.py,writer.py— public guards, processor selection, loading and saving. -
gptqmodel/quantization/vptq.py— separate VPTQ algorithm, for comparison. -
docs/exl3-mi350x-quant-lifecycle.md— MI350X measurement record.