Repository navigation
MLX Apple Silicon
Documentation home · Quickstart · Lifecycle
On this page
- Overview
- Workflow at a glance
- Requirements and installation
- Concepts: method, checkpoint, runtime, and tensor layout
- Quantize a model directly in MLX
- Native weight-level quantization APIs
- Native ParoQuant path
- Native EXL3 path
- Automatic MLX acceleration inside Torch workflows
- Loading existing checkpoints for MLX inference
- Packed layouts and exact transcoding
- Executing MLX qlinear kernels
- Benchmarks and numerical validation
- Pitfalls and edge cases
- Source files
- See also
The contracts below describe source revision 08efb7c7; installed MLX-LM behavior and supported architectures can also vary by dependency version.
GPT-QModel's MLX support has three separate entry points:
| Workflow | Entry point | Result |
|---|---|---|
| Quantize a dense MLX-LM model without loading a Torch model | GPTQModel.quantize_mlx(...) |
Saved MLX-LM checkpoint; supports method="gptq" or "awq"
|
| Load an existing GPT-QModel checkpoint for MLX inference | GPTQModel.load(..., backend="mlx") |
Converts supported packed layers into an MLX model and uses MLX-LM generation |
| Use individual native quantization primitives | Functions in gptqmodel/quantization/mlx_*.py
|
Weight-level pseudo-quantized tensors, checkpoint payloads, optimization results, or processor bridges, depending on the function |
The third row is not an extended method list for the first row. In particular, quantize_mlx(method="exl3"), method="rtn", method="fp8", method="qqq", and method="paroquant" are not accepted by the high-level native MLX-LM factory in this source tree. The existence of a native weight primitive does not imply a complete model calibration/export interface for that method.
Reading the diagram: Alternative routes. Failed transfer may requantize ordinary GPTQ/AWQ or reject special formats. A custom converted runtime is not automatically a stock MLX-LM reloadable export.
Text version of the workflow
- Native MLX-LM quantization: Load in MLX-LM → Calibration tokens → GPTQ or supported AWQ → Save MLX-LM artifact.
- Packed checkpoint transfer: Saved quantized model → Validate source state → Transcode packed state → Construct MLX runtime.
The mlx optional dependency group in pyproject.toml declares:
| Dependency | Requirement |
|---|---|
| MLX | mlx>=0.32.4 |
| MLX-LM, Python < 3.11 | mlx_lm>=0.31.3 |
| MLX-LM, Python >= 3.11 | mlx_lm>=0.32.2 |
pip install 'gptqmodel[mlx]' --no-build-isolationThe native compute kernels use mx.fast.metal_kernel and target Apple Silicon/Metal. The EXL3 and ParoQuant processor availability probes explicitly require sys.platform == "darwin", platform.machine() == "arm64", and an importable mlx.core. Installing a package with an MLX name on an unrelated host does not satisfy those gates. The utility conversion module separately sets MLX_AVAILABLE=False if its MLX/MLX-LM runtime imports fail.
“Native MLX” does not mean every operation runs on the GPU: MLX's Cholesky and triangular inverse operations used here execute on the MLX CPU stream. They do not transfer the model into a Torch quantizer, but they do contribute to calibration/quantization time and memory use.
| Concept | Meaning in this backend |
|---|---|
| Native MLX-LM affine storage | Packed uint32 codes, scales, and additive biases. Reconstruction is code * scale + bias; this is not a GPTQ packed-zero tensor. |
| GPTQ/AWQ checkpoint conversion | Reads method-specific packed words, zero points, scale dtype, activation-order metadata, and layout. A same-bit conversion need not mean dequantize/requantize. |
| Pseudo-quantization | Returns floating values reconstructed from discrete codes. Those values are useful to calibration and error accounting but are not by themselves a packed inference checkpoint. |
| EXL3 weight orientation | Quantization pipeline uses [input_features, output_features], unlike most weight primitives' [output_features, input_features]. |
| Native/custom runtime | Some converted layers require project MLX modules that MLX-LM's ordinary QuantizedLinear loader cannot reconstruct. Runtime construction and portable checkpoint serialization are different promises. |
| FP8 and GGUF | Quantized byte payloads have their own scale and block conventions; they are not integer-affine aliases. |
See Packing Formats and Compatibility before interchanging files between GPT-QModel, Transformers, MLX-LM, and GGUF consumers.
The public wrapper in gptqmodel/models/auto.py is:
@staticmethod
def quantize_mlx(model_id_or_path: str, output_path: str, method: str = "gptq", **kwargs):
...It delegates to gptqmodel.quantization.mlx_native.quantize_mlx:
def quantize_mlx(
model_id_or_path: str,
output_path: str,
method: str = "gptq",
bits: int = 4,
group_size: int = 64,
num_samples: int = 128,
sequence_length: int = 512,
trust_remote_code: bool = False,
calibration_data=None,
seed: int = 123,
**kwargs,
):
...from gptqmodel import GPTQModel
output = GPTQModel.quantize_mlx(
"organization/dense-model",
"./model-mlx-gptq",
method="gptq",
bits=4,
group_size=64,
num_samples=128,
sequence_length=512,
batch_size=8,
damp_percent=0.01,
)| Option | Default and behavior |
|---|---|
method |
"gptq"; lowercased and restricted to gptq/awq
|
bits, group_size
|
4, 64; native GPTQ supports bits 2, 4, 8 and groups 32, 64, 128
|
num_samples, sequence_length
|
128, 512; passed to MLX-LM load_data when custom data is absent |
calibration_data |
None; otherwise supply a nonempty rank-2 token array, not text strings or a Python prompt list |
trust_remote_code |
False; when true, the installed MLX-LM load must explicitly expose this parameter or the wrapper raises ValueError
|
seed |
123; seeds mx.random
|
**kwargs |
Passed to gptq_quantize_model_mlx or MLX-LM awq_quantize; they are not a QuantizeConfig object or arbitrary loading options |
| Return | The output_path after mlx_lm.utils.save completes |
The loader uses lazy=True, return_config=True. For a Hub ID, saving resolves the cached config.json with hf_hub_download(..., local_files_only=True) to locate the source artifact directory. The source must therefore have been materialized in the local Hub cache by loading; this final lookup is not an independent network download.
def gptq_quantize_model_mlx(
model,
data,
bits: int = 4,
group_size: int = 64,
fallback_bits: int = 6,
fallback_group_size: int = 64,
batch_size: int = 8,
damp_percent: float = 0.01,
):
...
def inverse_hessian_mlx(hessian, damp_percent: float = 0.01):
...
def gptq_quantize_weight_mlx(
weight, inverse_hessian, bits: int = 4, group_size: int = 64
):
...The model path wraps exact nn.Linear and MLX-LM SwitchLinear leaf types with Hessian catchers. Only widths admitting a supported group size are caught. _effective_group_size tries 128, 64, 32, taking the largest size no greater than requested that divides the width. Every caught layer must receive calibration activations; otherwise quantization raises rather than silently using an empty Hessian.
Hessian accumulation flattens tokens and accumulates FP32 covariance. _hessian_partial_mlx and _accurate_matmul_mlx split values into BF16-high and FP32-residual components to correct reduced-precision GPU matmul input arithmetic. Damping uses the maximum of damp_percent * mean(diagonal), max(diagonal) * 1e-6, and 1e-12; damp_percent must be strictly between zero and one. The CPU-stream Cholesky/inverse/upper-Cholesky sequence returns the factor needed by the group update.
The weight primitive accepts rank-2 or rank-3 weights, flattens output prefixes, and performs a fused Metal dispatch per group. It returns (packed_weight, scales, biases) in native MLX affine layout. Error feedback updates the remaining columns. Quantized layer scale dtype is restored to the original layer dtype using set_dtype.
Remaining leaf modules with to_quantized use the fallback bits/group size, defaulting to 6-bit/64. Widths with no eligible group are marked False in the returned quantization config; per-layer group reductions are explicitly recorded. Thus a “4-bit” native GPTQ run may deliberately include 6-bit fallback modules and unquantized modules. This path is not equivalent to selecting every Torch GPTQ option such as activation ordering or static groups.
GPTQModel.quantize_mlx(
"organization/dense-model",
"./model-mlx-awq",
method="awq",
bits=4,
group_size=64,
)AWQ delegates to MLX-LM: it looks up AWQ_MODEL_CONFIGS[config["model_type"]], rejects unsupported model types, distributes calibration with dist_split(..., mx.distributed.init()), calls awq_quantize, and runs update_config. The supported model table and AWQ keyword defaults belong to the installed MLX-LM version, not GPT-QModel's Torch AWQ processor. See AWQ.
These APIs operate on MLX arrays and are opt-in building blocks. Unless noted, a weight uses [out_features, in_features]. Caller-owned calibration, activation ordering, smoothing, packer selection, and model serialization are not implicitly supplied.
def quantize_rtn_weight_mlx(weight, *, bits=4, group_size=128, sym=True):
...
def gptaq_correction_mlx(delta_covariance, inverse_hessian, *, alpha=0.25):
...
def gptaq_quantize_weight_mlx(
weight, inverse_hessian, correction, *, bits=4, group_size=128, sym=True,
):
...
def foem_quantize_weight_mlx(
weight,
inverse_hessian,
*,
bits: int = 4,
group_size: int = 128,
beta: float = 0.2,
sym: bool = True,
return_residuals: bool = False,
):
...
def qqq_quantize_weight_mlx(weight, inverse_hessian, *, group_size=128):
...
def qqq_quantize_weight_mlx_with_loss(
weight, inverse_hessian, *, group_size=128
):
...| Module/API | Supported contract | Outputs and important limits |
|---|---|---|
mlx_rtn.py / quantize_rtn_weight_mlx
|
Nonempty rank-2; bits 2..8; positive group size or -1 for one group per row; partial final group supported |
(dequantized_weight, scales, zeros, group_indices); scales/zeros FP32, indices int32. Input FP16/BF16/FP32 dtype is preserved in pseudo-weight output. No smoothing. |
mlx_gptaq.py / correction |
Square FP32 delta covariance and matching upper inverse-Hessian factor; finite alpha, default 0.25
|
Strict-upper activation correction via accurate FP32 matmuls. |
mlx_gptaq.py / quantization |
FP16/BF16/FP32 rank-2 weight; FP32 matching Hessian/correction; bits 2..8, group 16,32,64,128 dividing width; bool sym
|
FP32 (pseudo_weight, scales, zeros); dynamic groups, MSE search and activation order disabled, blocksize=group_size. |
mlx_foem.py |
Same bit/group/weight/factor contracts; finite inputs, positive factor diagonal; beta in [0,1]
|
Alpha-zero FOEM only. FP32 (pseudo_weight, scales, zeros); optional fourth per-weight residual tensor for loss accounting. No MSE search or activation-order permutation. |
mlx_qqq.py |
Four-bit QQQ; group_size=-1 or 128; group 128 must divide width; FP32 matching factor |
(pseudo_weight, scales, zeros, extra_scales); with_loss adds aggregate sum(error**2)/2. Inputs already preordered; not yet a QQQ inference layout. |
QQQ per-channel (-1) uses fixed row scales and signed codes [-7,7], returning no extra scale. Group-128 uses dynamic unsigned codes [0,15] with zero point 8 and returns original-weight int8 row scales (max_abs/127, with zero-row handling). Activation ordering and MSE search are caller responsibilities. See QQQ.
def quantize_fp8_weight_mlx(
weight,
*,
format="float8_e4m3fn",
weight_scale_method="row",
weight_block_size=None,
):
...mlx_fp8.py accepts finite nonempty rank-2 FP16/BF16/FP32 weights and returns (uint8_codes, inverse_scales). Supported format strings are float8_e4m3fn, float8_e5m2, float8_e4m3fnuz, and float8_e5m2fnuz. Scale modes are tensor, row, and block, normalized by the common config helpers. A block layout must divide both weight dimensions.
The stored scale is the multiplicative encoding/inverse-dequantization scale: values are encoded after multiplication by it. Decoding divides codebook values by it. This is not interchangeable with an affine integer group scale. The Metal encoder explicitly handles signed zero according to the FP8 format, saturation, ties-to-even rounding, float32 subnormal inputs, and scale overflow. See FP8 and NVFP4.
def quantize_4bit_weight_mlx(weight, *, quant_type="nf4", block_size=64, compress_statistics=False):
...
def quantize_int8_weight_mlx(weight):
...mlx_bitsandbytes.py accepts finite rank-2 FP16/BF16/FP32 weights. Four-bit types are nf4 and fp4; accepted block sizes are 32,64,128,256,512,1024,2048,4096. Two flattened codes are stored per byte, first code in the high nibble; an odd final value pads with the code for zero. This order differs from several integer-affine and GGUF nibble layouts.
Uncompressed output is (packed_uint8, FP32_block_absmax). With compress_statistics=True, the second value is a dict containing absmax (8-bit nested codes), nested_absmax, offset, and nested_code. Nested scale blocks use 256 values. The implementation includes a special all-subnormal FP32 packing/nested-scale path to avoid Metal flush-to-zero changing checkpoint bytes.
Int8 returns (int8_row_codes, FP32_row_absmax) using row normalization to 127 and ties-to-even. These are quantized weight payloads; the function does not itself install a full bitsandbytes module or model export. See BitsAndBytes.
def gguf_quantize_weight_mlx(weight, qtype: str):
...mlx_gguf.py uppercases the qtype and returns a uint8 matrix [rows, input_width / block_size * bytes_per_block]. Inputs must be rank-2 FP16/BF16/FP32 with nonzero rows and width divisible by the selected block size. This packs weight bytes, not a complete .gguf file with model metadata/tokenizer/container headers.
| Accepted qtype(s) | Values per block | Bytes per block |
|---|---|---|
Q1_0, Q1_0_G128
|
128 | 18 |
Q2_0 |
64 | 18 |
Q4_0 |
32 | 18 |
Q4_K, Q4_K_S, Q4_K_M
|
256 | 144 |
Q5_K, Q5_K_S, Q5_K_M
|
256 | 176 |
Q6_K |
256 | 210 |
TQ1_0 |
256 | 54 |
TQ2_0 |
256 | 66 |
MXFP4 |
32 | 17 |
NVFP4 |
64 | 36 |
Q8_0 |
32 | 34 |
At this weight-level entry point, _S/_M K aliases select the same block kernel; they do not independently implement a model-wide mixed-precision policy. Q1_0 stores mean-absolute FP16 scale and sign bits. Q2_0 uses ternary values inside a two-bit code layout. TQ1_0 packs base-3 symbols, while TQ2_0 stores two-bit ternary symbols. Ternary kernels choose a parallel launch at 65,536 blocks. FP4 scale encoding differs between MXFP4 and NVFP4. Do not substitute one format's bytes for another merely because nominal bits are similar. See GGUF.
The path is split into rotation, pseudo-quantization, checkpoint packing, and module-scope optimization. These functions are not aliases for the native MLX-LM AWQ algorithm.
def paroquant_rotate_mlx(
weight,
pairs,
theta,
*,
group_size: int = 128,
channel_scales=None,
inverse: bool = False,
):
...
def paroquant_quantize_weight_mlx(
weight,
scales,
*,
bits: int = 4,
group_size: int = 128,
sym: bool = True,
zero_point_float=None,
):
...
def paroquant_pack_weight_mlx(weight, scales, *, group_size: int):
...| Module | Contract |
|---|---|
mlx_paroquant_rotation.py |
Weight rank >=2, FP16/BF16/FP32; grouped Givens rotations; group 16,32,64,128 dividing input width. Integer pairs is [rotations,input_features], each group a complete permutation of local channel indices; theta is [rotations,input_features/2]. Optional positive finite channel scales multiply before forward rotation and divide after inverse rotation. Output always FP32. |
mlx_paroquant_quant.py |
Rank-2 weight; bits 2..8; positive even group dividing width, or -1 for full row. FP32 scales/optional asymmetric zero parameters shaped [out,groups] or [out*groups,1]. Symmetric mode forbids zero parameters; asymmetric requires them. Scales clamp to [1e-5,1e5]; ties-to-even; output preserves input weight dtype. |
mlx_paroquant.py |
Packs symmetric four-bit exported transformed weights into ParoQuant's AWQ-compatible qweight/qzeros/scales. Output width divisible by 32. Groups 16,32,64,128 or full input width, even and dividing width. Weight/scales share FP16/BF16/FP32 dtype. FP32 scales are stored as FP16, matching the Torch AWQ pack contract. Finite positive stored scales and codes in [0,15] are required. |
mlx_paroquant_optimize.py |
MLX-autodiff module reconstruction with rotation stage, then weight/quantizer fine-tuning, smooth-L1 loss, STE round/clamp, validation-best snapshots and cosine learning-rate decay to 1/20 of base LR. Supports adamw, adam, sgd; snapshot dtypes fp16,bf16,fp32. |
The optimizer has no defaults for its optimization controls. bias is a required positional argument but may be None:
def optimize_paroquant_linear_mlx(
weight,
bias,
inputs,
pairs,
theta_mask,
*,
bits: int,
group_size: int,
symmetric: bool,
train_rows: int,
val_rows: int,
batch_size: int,
rotation_epochs: int,
finetune_epochs: int,
rotation_lr: float,
weight_lr: float,
quantizer_lr: float,
optimizer_name: str,
optimizer_weight_decay: float,
optimizer_betas,
optimizer_eps: float,
optimizer_amsgrad: bool,
sgd_momentum: float,
sgd_dampening: float,
sgd_nesterov: bool,
best_state_dtype: str,
scale_clamp_min: float,
scale_clamp_max: float,
):
...The optimizer initializes zero theta and unit channel scales, subsamples evenly when calibration exceeds the requested train+validation rows, and masks selected angles to zero after updates. It returns eight values: pseudo-weight in the original domain, transformed pack weight, quantizer scales, quantizer zeros, theta, runtime reciprocal channel scales, train loss, validation loss. optimize_paroquant_linear_mlx_to_torch synchronizes MPS when necessary, bridges via DLPack, and returns the Torch processor's ParoQuantOptimizationResult with used_identity=False.
The packer serializes AWQ lane order [0,2,4,6,1,3,5,7] and zero point 8. Rotations and runtime channel scales remain essential metadata; the qweight alone is insufficient. See ParoQuant.
EXL3 has a complete calibrated weight-level pipeline, with a Torch processor bridge and individually exposed stages. Its quantization orientation is [input_features, output_features]; stage matrices generally require FP32.
def exl3_quantize_weight_mlx(
weight,
hessian,
input_signs,
output_signs,
*,
sample_count: int,
bits: int,
codebook: str = "3inst",
force_output_scales=None,
sigma_reg: float = 0.025,
workspace_bytes: int = 256 << 20,
):
...Both weight dimensions must be divisible by 128; Hessian is [input,input]; input/output signs are FP32 [input,1] and [1,output]. Bits are integers 1..8; pipeline codebooks are exactly 3inst, mcg, mul1. The default pipeline codebook differs from several low-level helpers' mcg default.
The pipeline returns (reconstructed_weight, proxy_error, packed_trellis, input_scales_fp16, output_scales_fp16, fallback, global_scale, apply_output_scales). Reconstruction is FP32; proxy error must be finite. Fallback proxy error is zero because calibrated error is not evaluated there, not because reconstruction is lossless.
| Source module | Key API and defaults | Algorithm/contract |
|---|---|---|
mlx_exl3_pipeline.py |
exl3_quantize_weight_mlx, exl3_quantize_weight_mlx_to_torch, exl3_mlx_quantization_available
|
Orchestrates all stages. Torch bridge generates deterministic signs with CPU Torch generators seeded seed and seed+1, DLPack-transfers inputs/results, and emits suh, svh, trellis, plus codebook markers for mcg/mul1. |
mlx_exl3_hessian.py |
exl3_finalize_hessian_mlx(..., sample_count, sigma_reg=0.025, block_size=16) |
Normalizes captured Hessian by sample count, adds sigma_reg * mean(diagonal), applies signs and two order-128 Hadamards, and returns fallback flag, transformed Hessian, LDLQ factor, regularized diagonal. Width divisible by 128. Fallback when count is zero or mean diagonal < 1e-20; nonfinite inputs/results raise. |
mlx_exl3_block_ldl.py |
exl3_block_ldl_mlx(hessian, *, block_size=16) |
CPU-stream Cholesky and diagonal-block triangular inverses, followed by fused Metal block-column normalization. Writes exact identity diagonal blocks. Finalization removes the scalar diagonal before LDLQ. Non-positive-definite input raises ValueError. |
mlx_exl3_rms.py |
exl3_block_rms_mlx(matrix, *, axis) |
FP32 RMS with 32-value reductions; axis 0 gives [1,columns], axis 1 gives [rows,1]. |
mlx_exl3_regularize.py |
exl3_output_regularize_mlx, exl3_input_regularize_mlx, exl3_regularize_transforms_mlx(..., force_output_scales=None, hessian_diagonal=None, fallback=False)
|
RMS normalization, sign/channel scales, fused left/right Hadamards. For calibrated input, sorts sqrt(hessian_diagonal) and divides the sum of the largest input_width//50 values by the full sum; absent an override, enables output scales when this ratio is below 0.15. Fallback uses the force setting. If mean output RMS is <=1e-30 and a force setting is supplied, output scales are enabled even when that setting was False. |
mlx_exl3_hadamard.py |
exl3_hadamard_128_mlx(matrix, *, axis) |
Normalized orthonormal/self-inverse order-128 transform along row/column blocks; finite FP32 rank-2 input; transformed dimension divisible by 128. |
mlx_exl3_gss.py |
exl3_sample_global_scale_tiles_mlx(..., width=3), exl3_global_scale_search_mlx(..., bits, codebook="mcg", width=3, workspace_bytes=256<<20)
|
Wrapped-diagonal tile sampling; golden-section search on [0.1,1.9] until interval <= 0.01, returning selected scale and mean of final two FP32 errors. |
mlx_exl3_tiles.py |
exl3_to_tensor_core_tiles_mlx, exl3_from_tensor_core_tiles_mlx
|
Exact 16x16 tensor-core lane permutation; matrix to [input/16,output/16,256] and back; both matrix dimensions divisible by 16. |
mlx_exl3.py |
exl3_quantize_tiles_mlx(..., bits, codebook="mcg", workspace_bytes=256<<20) |
Two-pass tail-biting Viterbi search on 256-value tiles; input FP32 is cast to FP16 for specified search arithmetic. Returns FP32 decoded values and int16 state bit patterns. |
mlx_exl3.py |
exl3_decode_states_mlx(..., codebook="mcg"), exl3_pack_trellis_mlx(..., bits)
|
Decode int16-as-uint16 codebook states; pack only low bits bits into checkpoint trellis, shape [input/16,output/16,16*bits] int16. Adjacent physical uint16 words are swapped to match EXL3's bitstream ABI. |
mlx_exl3_ldlq.py |
exl3_ldlq_quantize_mlx(..., bits, codebook="mcg", buffer_rows=128, workspace_bytes=256<<20) |
Descending buffered 16-row LDLQ error feedback with explicit FP32 product accumulation. Both dimensions must be divisible by buffer_rows, a positive multiple of 16; output width must also be divisible by 128. Pending compensation shrinks to unquantized prefix. |
mlx_exl3_fallback.py |
exl3_fallback_quantize_mlx(..., bits, codebook="mcg", workspace_bytes=256<<20) |
Quantizes regularized weight without Hessian compensation. Input width divisible by 16 and output width by 128. Returns reconstructed matrix and unpacked tile states. |
Viterbi workspace is bounded independently of tile count: edges = 1 << (16 - bits), with backtrace bytes per active slot edges * (256*2 + (0 if bits>=4 else 2*2)). Requested workspace must hold at least one slot. Active groups are capped by tile count, 256, and the budget. The budget bounds this search scratch, not total model weights, Hessians, output tensors, or all pipeline temporaries.
Regularization precedes global scaling; the selected scale multiplies regularized weights and divides input scales. Quantization uses either LDLQ or fallback, reconstructs with inverse Hadamards/channel scales, and finalizes the packed trellis. The low-level stage outputs are useful for independent oracle checks; treating unpacked int16 search states as a production inference cache would violate the packed-storage contract. See EXL3.
Several ordinary Torch quantization/packing paths can select MLX internally. This does not select the MLX inference backend, extend the high-level quantize_mlx(method=...) list, or mean the entire quantization loop runs in MLX. Their availability probes require Darwin ARM64 and importable mlx.core; unlike checkpoint-holder validation, these probes do not explicitly call mx.metal.is_available().
| Calling path | Minimum weight elements | Additional selection conditions in this revision |
|---|---|---|
quantization/rtn.py |
256 * 1024 |
Rank-2 CPU/MPS weight, no smoothing, integer bits 2..8, positive group size or -1. |
quantization/foem.py |
256 * 1024 |
CPU/MPS weight and same-device inverse factor; FOEM config with alpha=0, beta in [0,1]; no activation order, static groups, or MSE search; integer bits 2..8; group 16,32,64,128 dividing width and equal to blocksize. |
quantization/qqq.py |
4_194_304 |
CPU/MPS weight and same-device non-None inverse factor; debug disabled; blocksize=128; group -1 or width-dividing 128; no static groups or MSE search; symmetric quantizer with matching group metadata and maximum code 7 for whole-row or 15 for grouped operation. |
nn_modules/qlinear/gguf.py packing |
2 * 1024 * 1024 |
Counts the contiguous FP32 CPU matrix after optional smoothing and input padding; otherwise uses the existing non-MLX GGUF packer. |
nn_modules/qlinear/fp8.py packing |
60 * 1024 * 1024 |
Supported MLX FP8 format; counts the contiguous FP32 CPU matrix after optional smoothing. |
nn_modules/qlinear/bitsandbytes.py four-bit packing |
4 * 1024 * 1024 |
Four-bit branch; matrix has been converted to its configured compute dtype. |
nn_modules/qlinear/bitsandbytes.py INT8 packing |
8 * 1024 * 1024 |
INT8 branch; source matrix has been converted to FP16. |
looper/exllamav3_processor.py |
No element threshold in this branch | CPU/MPS target, captured Hessian, and positive captured sample count. Uses the native EXL3 bridge instead of the CUDA/HIP route. |
looper/paroquant_processor.py optimization |
No element threshold in this branch | Nonempty captured inputs, CPU/MPS weight, module optimization scope, opt_stage_impl="fast", and gradient checkpointing disabled. |
These are dispatch conditions, not a substitute for each primitive's shape/dtype validation. The weight-level GPTAQ helper is exposed separately; its existence does not establish automatic dispatch from the Torch GPTAQ loop.
from gptqmodel import GPTQModel
model = GPTQModel.load("organization/quantized-model", backend="mlx")
text = model.generate(prompt="Explain groupwise quantization.", max_tokens=128)This is the GPT-QModel checkpoint conversion route, not the direct native MLX-LM quantization route. To consume the latter's saved artifact, use MLX-LM:
from mlx_lm import load, generate
model, tokenizer = load("./model-mlx-gptq")
text = generate(model, tokenizer, prompt="Explain groupwise quantization.")gptqmodel/nn_modules/qlinear/mlx.py contains Torch checkpoint holders and conversion validators, not the final MLX nn.Module implementations. Their forward raises an error requiring conversion before inference. The common validator requires Darwin ARM64, importable MLX/MLX-LM, and mx.metal.is_available().
The holders declare MPS as their Torch-side device, FP16 checkpoint dtype, no training, no adapters, sharding support, and no automatic feature padding. These declarations do not mean the final MLX layer rejects BF16 activations: checkpoint validation and activation execution are separate stages.
Holder class in qlinear/mlx.py
|
Source format / method | Bits | Declared group sizes | Feature alignment (input, output)
|
|---|---|---|---|---|
MlxQuantLinear |
GPTQ / GPTQ_V2 / GPTQ_P | 2,3,4,5,6,7,8 |
-1,16,32,64,128,256,512,1024 |
(32,32) |
GPTQMarlinMlxQuantLinear |
GPTQ Marlin | 4,8 |
-1,32,64,128 |
(32,32) |
GPTQBitBLASMlxQuantLinear |
GPTQ BitBLAS | 2,4,8 |
-1,32,64,128 |
(32,32) |
AwqMlxQuantLinear |
AWQ GEMM | 2,3,4,5,6,7,8 |
-1,16,32,64,128 |
(8,8) |
AWQMarlinMlxQuantLinear |
AWQ Marlin | 4,8 |
-1,32,64,128 |
(8,8) |
AWQBitBLASMlxQuantLinear |
AWQ BitBLAS | 2,3,4,5,6,7,8 |
-1,32,64,128 |
(16,16) |
AwqGemvMlxQuantLinear |
AWQ GEMV | 4 |
-1,64,128 |
(32,64) |
AwqGemvFastMlxQuantLinear, LLMAwqMlxQuantLinear
|
AWQ GEMV_FAST / LLM_AWQ | 4 |
-1,16,32,64,128 |
(64,8) |
ParoMlxQuantLinear |
ParoQuant | 4 |
-1,16,32,64,128 |
(32,8) |
QQQMlxQuantLinear |
QQQ |
4 source bits |
-1,128 |
(64,64) |
GGUFMlxQuantLinear |
Supported GGUF types below | 1,2,4,5,6,8 |
Format/block dependent |
(32,8) plus full source blocks |
FP8MlxQuantLinear |
FP8 | 8 |
Scale-method dependent |
(1,1); native MXFP8 adds 32-input alignment |
BitsAndBytesMlxQuantLinear |
BNB NF4/FP4/INT8 | 4,8 |
Format dependent | (1,1) |
The table is a registry declaration, not the entire compatibility test. The repackers have additional constraints, notably native affine _pack_rows requires input width divisible by 32. Group-16 requires 32-input alignment even where a class declares a weaker generic alignment. Requested whole-row -1 must map to a supported effective group or an input width >=128 divisible by 128. Shapes, source class, zero layout, g_idx, scale dtype, and runtime metadata are validated too.
GPTQ holders require desc_act=False and monotonically grouped g_idx == arange(in_features)//group_size; nontrivial activation-order permutations are not silently discarded. GPTQ/GPTQ_V2 require normalized raw-zero v2 state (REQUIRES_FORMAT_V2=True); GPTQ_P is explicitly accepted with raw zero codes despite legacy metadata. The Marlin holder repairs legacy zero-minus-one storage before MLX packing. GPTQ Marlin is symmetric-only. AWQ GEMM also declares desc_act=False; several AWQ-specific source holders permit both metadata values. ParoQuant and QQQ require symmetric source layouts.
def convert_gptq_to_mlx_weights(
model_id_or_path: str,
model: Union[PreTrainedModel, BaseQModel],
gptq_config: dict,
lm_head_name: str,
):
...The conversion wrapper in utils/mlx.py runs conversion under mx.stream(mx.cpu) while restoring the prior stream afterward. _packed_mlx_weights validates each supported source layer, transcodes checkpoint weights, creates the MLX-LM architecture from config, quantizes matching leaf modules, installs custom runtime wrappers, and loads transferred arrays. If packed layer names do not match the architecture's to_quantized modules, it returns None rather than pretending that the mapping succeeded.
Supported conversion format values are GPTQ, GPTQ_V2, GPTQ_P, GEMM, GEMV, GEMV_FAST, LLM_AWQ, MARLIN, BITBLAS, PAROQUANT, QQQ, GGUF, FP8, BITSANDBYTES, and EXL3. This top-level list is broader than any one holder's bit/group contract. Nonempty dynamic configurations are also checked and currently accept dynamic bit values 2..8, not dynamic one-bit settings.
Every nonempty dynamic rule must explicitly contain bits: this conversion wrapper reads config["bits"] rather than inheriting the base value. A group-size-only rule therefore raises KeyError here even if it is valid for the ordinary QuantizeConfig path. Empty rule dictionaries skip this bit check. This is a conversion limitation, not a general dynamic-configuration requirement.
| Conversion case | Behavior |
|---|---|
| Transferable packed source | Preserves supported checkpoint semantics through a method-specific transcode; logs transfer without requantization. This is not a statement that every method retains the original bit width; grouped QQQ and seven-bit affine sources change storage width as described below. |
| Untransferable ordinary GPTQ/AWQ | Legacy fallback dequantizes source layers, casts to FP16, builds an MLX-LM model, and calls quantize_model; this is requantization, not a bit-exact transcode |
Legacy fallback group -1 or 16
|
Changes the config's group size to 64 before requantization |
| Special formats without transferable state | ParoQuant, QQQ, GGUF, FP8, BNB, EXL3, GEMV, GEMV_FAST, LLM_AWQ, Marlin, and BitBLAS raise instead of taking the generic requantization fallback |
| Custom-runtime marker |
_gptqmodel_group16_runtime or _gptqmodel_custom_mlx_runtime tells models/loader.py to retain the constructed runtime model directly |
| Ordinary native runtime | Loader uses a temporary MLX-LM save/config/tokenizer directory and reloads it with mlx_lm.load
|
Do not assume every returned runtime can be serialized and reloaded by stock MLX-LM. The custom markers exist precisely because the ordinary loader only reconstructs native quantized layers. The generic fallback also consumes/empties parts of the source model during conversion; it is not an API for retaining two independently usable model copies.
gptqmodel/utils/mlx_packing.py uses NumPy to move discrete codes into the target ABI. The key public-style helper signatures are:
def repack_gptq(qweight, qzeros, scales, in_features, out_features, bits, planar=None):
...
def repack_awq(qweight, qzeros, scales, in_features, out_features, bits=4, planar=False):
...
def repack_gptq_bitblas(qweight, qzeros, scales, in_features, out_features, bits, sym):
...
def repack_awq_bitblas(qweight, qzeros, scales, in_features, out_features, bits=4):
...| Source representation | Target mapping / caveat |
|---|---|
gptq_v2 continuous/planar |
Unpack input-axis codes and output-axis raw zeros; transpose scales to [out,groups]; additive bias -zero*scale; target uint32 row streams. Planar default is enabled for bits 5,6,7; caller may explicitly select source layout. |
| Native affine rows | Bits 2,4,8 fit integer codes per uint32; 3,5,6 cross uint32 boundaries in a continuous little-endian bitstream. |
| Seven-bit source | No native affine 7-bit target; preserves code values in packed 8-bit target words. Storage grows; it is not a lossy change of levels. |
| Large/full-row affine groups | Metadata repeats scales/biases into supported subgroups of at most 128 without changing codes. |
| Group 16 | Keeps one packed weight stream, separate even/odd 16-value scale/bias arrays; fallback executes two masked group-32 quantized products. |
| AWQ GEMM | Decodes output-axis continuous streams; four-bit AWQ reverses its lane permutation before row packing. Non-planar only. |
| GPTQ BitBLAS | Portable byte-packed rows only, bits 2,4,8; signed symmetric codes flip the sign bit for unsigned-affine mapping. Transformed private BitBLAS state is not assumed portable. |
| AWQ BitBLAS | Continuous little-endian byte streams, 2..8 bits; seven-bit target promoted to eight; source scales cast to FP16. |
| AWQ GEMV | Output-major INT4 plus padded output-major scale/zero rows; extracts only actual groups. |
| AWQ GEMV_FAST / LLM_AWQ | Undoes four-output interleave and the 32-input permutation; already-scaled zero offsets become additive biases. |
| ParoQuant | AWQ codes plus learned rotations/channel scales; cannot discard runtime transforms. |
| QQQ whole-row | Signed nibbles XOR sign bit, affine scale 16/bias -128; group-128 source uses packed 8-bit inference values after group-to-channel scaling. |
| FP8 | Native MXFP8 for a compatible subset, otherwise raw FP8 bytes + a 256-entry decode codebook and FP32 inverse-scale metadata. |
| BNB four-bit | Keeps compressed nibbles + NF4/FP4 codebook and decoded scale metadata; nested scale statistics may be decoded, not full dense weights. |
| BNB int8 | If width divides 128/64/32, exact unsigned affine mapping with row scale absmax/127, bias -128*scale; otherwise packed signed-int8 Metal path. |
| EXL3 | Checkpoint trellis remains packed; runtime reconstructs cyclic trailing 16-bit windows only inside its consuming kernel. |
Ordinary GPTQ/AWQ holders supply FP16 scales to their conversion packer; affine biases are computed in FP32. Unquantized Torch model parameters and many biases are also transferred as FP16. “Exact packed transfer” means preserving the discrete source checkpoint semantics, not a promise to preserve an arbitrary original BF16 dense model dtype end-to-end.
Current grouped-QQQ storage limitation: when the effective source group differs from the input width, QQQMlxQuantLinear.pack_source reconstructs the group's scaled integer values and stores the complete layer as packed 8-bit affine weights. This preserves the intended QQQ integer inference values, but expands the four-bit source into a retained eight-bit representation. Under the explicit INT4-to-INT8 prohibition in AGENTS.md, this route must not be described as passing the persistent-storage acceptance gate. The whole-row decision is module.group_size == module.in_features, which selects a distinct four-bit mapping; a group-128 layer with input width 128 therefore also takes that branch. The expanded grouped path is documented here as an implementation limitation, not endorsed as a compliant optimization.
The FP8 native MXFP8 subset is likewise conditional: the holder requires float8_e4m3fn, row or tensor inverse scaling, input width divisible by 32, finite positive inverse scales with finite reciprocals, and no E4M3 NaN codes. Other source-compatible FP8 layers use the packed-byte/codebook runtime; they are not silently converted to native MXFP8. Source compatibility also requires inverse-scale semantics and the exact weight matrix shape.
utils/mlx_gguf_packing.py separates affine and native FP4 modes:
def repack_gguf_affine(qweight: np.ndarray, qtype: str, in_features: int):
...
def repack_gguf_float4(qweight: np.ndarray, qtype: str, in_features: int):
...Both return (weight, scales, biases, quantization_params). GGUF affine transfer rejects padded/malformed blocks. Accepted source qtype names are case-sensitive here: Q1_0, Q1_0_g128, Q2_0, Q4_0, Q8_0, Q4_K, Q5_K, Q6_K, TQ1_0, TQ2_0, MXFP4, and NVFP4. Unlike the quantization entry point, the transfer helper does not uppercase arbitrary strings or accept _S/_M aliases directly.
| GGUF type | Target bits/group/mode |
|---|---|
| Q1_0 / Q1_0_g128 | 2-bit, group 128, affine; codes 0/1 with scale 2*d, bias -d
|
| Q2_0 | 2-bit, group 64, affine |
| Q4_0 | 4-bit, group 32, affine |
| Q8_0 | 8-bit, group 32, affine; signed byte plus 128 |
| Q4_K / Q5_K | 4-/5-bit, group 32, affine; per-subgroup scale and minimum |
| Q6_K | 6-bit packed rows, 16-value source scales; custom even/odd runtime despite group-32 config compatibility |
| TQ1_0 / TQ2_0 | 2-bit, group 128, affine; repeats block scale for the two subgroups |
| MXFP4 | 4-bit, group 32, mode="mxfp4"; uint8 encoded scales, no additive bias |
| NVFP4 | 4-bit, group 16, mode="nvfp4"; uint8 encoded scales, no additive bias |
Use repack_gguf_float4 for MXFP4/NVFP4; their appearance in the common supported-type set does not make them affine. The float4 helper reorders low/high half-block nibbles into native input order and retains each format's scale bytes.
The following are the actual MLX nn.Module classes, installed by utils/mlx.py rather than called as Torch checkpoint holders:
| Module | Runtime class(es) | Execution behavior |
|---|---|---|
mlx_gptq.py |
MlxGPTQLinear, MlxGPTQGroup16Linear
|
Ordinary affine wrapper casts output to activation dtype. Group-16 uses packed Metal single-row decode; selected BF16 prefill uses tiled Metal; otherwise two group-32 products. |
mlx_awq.py |
MlxAWQLinear, MlxAWQGroup16Linear
|
Same output-dtype contract; AWQ group-16 accepts target storage bits 2,3,4,5,6,8. |
mlx_group16.py |
MlxGroup16Linear |
Two masked mx.quantized_matmul calls using shared packed weights and independent even/odd scale metadata; adds bias and restores input dtype. |
mlx_gguf.py |
MlxGGUFLinear, MlxGGUFQ6KLinear
|
Ordinary quantized wrapper plus exact Q6_K group-16 fused packed decode/small-prefill kernel; two-product fallback for other prefill. |
mlx_paro.py |
MlxParoLinear |
Precomputes partner/cos/sin indexing metadata, applies channel scales and all Givens stages in FP32, then packed affine matmul and input-dtype output. Identity state skips rotation. |
mlx_qqq.py |
MlxQQQLinear |
Converts input to FP16, per-token max/127 activation quantization, affine product, input/channel scaling, FP16 rounding, bias, then original output dtype. |
mlx_fp8.py |
MlxFP8Linear, MlxFP8PackedLinear
|
Compatible native MXFP8 computes unscaled dot in FP32 then applies output scale; general packed Metal decodes raw FP8/codebook bytes with row/tensor/block scales. |
mlx_bitsandbytes.py |
MlxBitsAndBytesLinear |
Packed NF4/FP4 Metal matmul; aligned int8 affine path or signed-int8 Metal path; decoded four-bit and non-affine int8 weight values intentionally round through FP16 before FP32 accumulation. |
mlx_exl3.py |
MlxEXL3Linear |
Packed trellis decode, input/output Hadamard and channel scaling, single-row or 16-row tiled prefill; no dense weight fallback. Supports bits 1..8, codebooks 3inst,mcg,mul1, positive feature multiples of 128. |
| Kernel | Current source dispatch |
|---|---|
| GPTQ group-16 | Decode when flattened rows == 1; output width >=5120 selects multi-column warp decode, otherwise the original packed decode. Packed BF16 prefill requires input >=5120, output <=1024, and rows 2..8, 12, or 16. Otherwise generic group-16 path. |
| AWQ group-16 | Decode when rows == 1; output width >=5120 selects the same multi-column warp decode. Packed BF16 prefill for rows 2..16 and output <=2048. |
| GGUF Q6_K | Decode when rows == 1. Packed BF16 prefill for rows 2..16 and output <=2048; otherwise two-product fallback. |
| ParoQuant | Fuses all rotations when nonempty, group <=128, and BF16 input or more than one rotation. Nonempty FP16 otherwise uses per-stage Metal; other cases use array math. FP32 is retained between stages. |
| Packed FP8 | Tiled prefill when rows !=1, input <=8192, output >=8192, input divisible by64 and output divisible by32. Other cases use row-tiled packed reduction. |
| EXL3 | Single-row fuses output transform/scales/bias when output width > input width; other decode uses separate output transform. Multirow uses 16-row padded scratch and 16x16 bounded weight tiles. |
The GPTQ/AWQ wide single-row path reuses each activation load across four output columns per warp, with eight warps per threadgroup and guarded tail columns. It consumes packed weights directly, accumulates in FP32, and returns the input activation dtype, including FP16 and BF16. Selection is automatic; no configuration flag or checkpoint conversion is needed, and the prefill gates above are unchanged.
These conditions describe this source version, not stable application knobs. A benchmark of one row/shape/dtype does not measure every path. Zero-element inputs are explicitly handled in the custom packed kernels; test those alongside ordinary decode and prefill.
- Output dtype preservation does not mean all intermediates use that dtype. ParoQuant rotations, EXL3 transforms, Hessian stages, and many reductions deliberately use FP32.
- FP16 has a smaller exponent range than BF16. Native MXFP8 computes its unscaled dot in FP32 because it can overflow FP16 before the checkpoint output scale brings the final result back into range.
- QQQ converts activations to FP16 and applies FP16 output rounding even for BF16 callers, then casts the result back. Its per-token max/127 activation scale stays FP32; zero rows use scale
1.0, avoiding scale underflow/zero division. BF16 output is not a fully BF16 internal QQQ algorithm. - BNB's codebook/scale product rounds through FP16 in the packed Metal path. Its aligned int8 affine path is separate arithmetic and needs separate comparison.
- Group-16 may choose a special prefill kernel only for BF16; FP16 and BF16 cannot share one performance result by assumption.
- BF16 cannot represent every FP16 result (and vice versa). Compare the FP32 oracle and the oracle rounded to the requested output dtype separately; final-rounding drift is not the same as accumulation error.
- FP16 checkpoint-holder validation is distinct from support for BF16 activations. Do not change holder metadata to BF16 merely to request BF16 execution.
mlx_generate(model, tokenizer, **kwargs) requires prompt=; it does not consume Transformers tensor input IDs as a substitute. It delegates to mlx_lm.generate and returns that function's result.
| Accepted keyword | Default / handling |
|---|---|
prompt |
Required; missing or None raises ValueError
|
verbose |
False |
max_tokens |
256; use this, not an assumed max_new_tokens alias |
sampler |
Optional explicit sampler |
temp / temperature, top_p, min_p, min_tokens_to_keep
|
Creates a sampler if any is supplied and no explicit sampler exists; non-None temp takes precedence over temperature. Construction defaults temp=0.0, top_p=0.0, min_p=0.0, min_tokens_to_keep=1. |
logits_processors |
Optional existing list |
repetition_penalty, repetition_context_size
|
Non-None, non-1 penalty appends processors; context default 20
|
prefill_step_size |
512 |
kv_group_size, quantized_kv_start
|
64, 0
|
max_kv_size, prompt_cache, kv_bits, prompt_progress_callback
|
Forwarded only when present |
formatter |
Removed, not forwarded |
Only the explicitly collected sampling arguments reach MLX-LM. Unrecognized leftover keywords are not forwarded. A Transformers-style do_sample, max_new_tokens, attention-mask, or generation-config argument therefore must not be assumed to behave identically here. Apply any chat template deliberately with the tokenizer appropriate to the saved model. See Hugging Face Integration.
The repository includes target-hardware tests and synchronized benchmark programs. No throughput/error numbers are asserted by this page: obtain them on the actual Apple Silicon host, installed MLX/MLX-LM versions, checkpoint, shape, and dtype. A documentation/source audit is not a substitute for executing those tests.
| Validation area | Representative source files |
|---|---|
| Native model/weight GPTQ |
tests/test_mlx_native_quantization.py, tests/test_mlx_qwen38_native_gptq_audit.py
|
| Exact GPTQ/AWQ transfer |
tests/test_mlx_packing.py, tests/test_mlx_gptq_formats.py, tests/test_mlx_awq_formats.py, tests/test_mlx_awq_full_bits.py, tests/test_mlx_qwen38_packing_audit.py
|
| RTN/GPTAQ/FOEM/QQQ/FP8/GGUF/BNB quantization | Corresponding tests/test_mlx_*_quantization.py; QQQ processor also tests/test_mlx_qqq_processor.py
|
| ParoQuant stages |
tests/test_mlx_paroquant_rotation.py, tests/test_mlx_paroquant_quantization.py, tests/test_mlx_paroquant_packing.py, tests/test_mlx_paroquant_optimize.py
|
| EXL3 stages |
tests/test_mlx_exl3.py, tests/test_mlx_exl3_codebook.py, tests/test_mlx_exl3_viterbi.py, and tests/test_mlx_exl3_{block_ldl,gss,hadamard,hessian,ldlq,pipeline,regularize,rms,tiles,fallback}.py
|
| Output dtype |
tests/test_mlx_gptq_output_dtype.py, tests/test_mlx_bnb_output_dtype.py, tests/test_mlx_gguf_output_dtype.py, tests/test_mlx_exl3_output_dtype.py plus FP8/ParoQuant/QQQ runtime tests |
| Real projection matrix |
tests/qwen38_27b_shapes.py, tests/test_mlx_qwen38_27b_shapes.py; full attention Q/K/V/O, linear-attention fused QKV/O, MLP gate/up/down |
| Performance harnesses |
tests/benchmark_mlx_qwen38_*.py; individual format, dtype, decode/prefill, packer, and pipeline programs |
Example focused commands, to run on Apple Silicon with the project dependencies installed:
python -m pytest -q tests/test_mlx_native_quantization.py tests/test_mlx_packing.py
python -m pytest -q tests/test_mlx_gptq_output_dtype.py tests/test_mlx_gguf_output_dtype.py
python -m pytest -q tests/test_mlx_paroquant_rotation.py tests/test_mlx_exl3_pipeline.py
python tests/benchmark_mlx_qwen38_27b.py --rows 1 16 --repeats 5 --krot 1 --rotate
python tests/benchmark_mlx_qwen38_exl3_pipeline.py --samples 3 --dtype bf16The Qwen3.8-27B ParoQuant harness constructs each projection's packed source state, checks Torch-MPS/MLX output agreement at rtol=atol=0.002, warms up once, synchronizes every result, and reports median repeated milliseconds, speedup and max absolute error as JSON lines. This harness is a comparison of operations at those projection shapes, not a full-model token/s benchmark. Its FP16 input generation does not replace BF16 validation required by the current rules.
AGENTS.md, .agents/skills/gptqmodel-torch-oracle-numerics/SKILL.md, and docs/mlx-native-quantization-plan.md establish the following requirements:
| Gate | Required evidence |
|---|---|
| Independent oracle | Deterministic identical inputs/configuration; Torch reference performs its own arithmetic rather than calling the MLX kernel under test |
| Discrete outputs | Exact complete packed bytes/codes/indices, except individually proven rounding ties; no blanket tolerance on code mismatches |
| Quantization floating outputs |
rtol <=1e-6 and atol <=1e-6 away from ties; normalized matrix error <=1e-6 for scale-dependent stages; nonfinite checks |
| Boundary cases | Exact thresholds and adjacent representable values, zeros/signed zero where significant, saturation, clamp/scale/zero-point ties, supported dtype casts |
| Inference |
rtol and atol <=2e-3; exact selected token IDs when sampling/ties are fixed |
| Activation dtypes | FP16 and BF16 independently; assert output dtype; report raw FP32-oracle and same-output-dtype visible drift separately |
| Dispatch | Decode, tiled prefill, generic fallback, bias/no bias, applicable shapes and explicit unsupported-dtype rejection |
| Performance | Same operation against main, resident inputs, identical shape/dtype/warmup/synchronization; per-projection timings and stated aggregation |
| Artifact usability | Save and reload with the intended runtime; primitive-only byte production is not proof of complete model support |
| Persistent storage | Quantized weights and permitted scales/codebooks/index metadata only; no retained dense/dequantized/expanded caches across calls, including fallbacks/compiled paths |
For true near-ties, retain a higher-precision independent reference and classify the mismatch individually. Report total mismatches, proven tie mismatches, floating drift away from ties, reference precision, and output drift. Do not weaken all tolerances to excuse one boundary case.
Inspect state after first and repeated forwards, mutation/replacement, checkpoint loads, dtype/device moves and supported explicit-clear operations. Report unchanged checkpoint bytes, extra packed-layout bytes, and peak transient dequantization scratch separately. A full-layer expanded INT4-to-INT8 cache is not permitted just because it uses integers; moving it to CPU or a nonpersistent buffer does not fix its lifetime. Code inspection of one route does not establish compliance for all kernels.
The native-quantization plan is a design/acceptance document, not a current support matrix. In particular its historical statement that NVFP4 is dequantization-only is superseded by the current mlx_gguf.py NVFP4 packer and mlx_gguf_packing.py native FP4 transfer path described above.
- Choose the correct entry point: the high-level
quantize_mlxfactory accepts only GPTQ/AWQ, while native method primitives and Torch-processor bridges are separate interfaces. - Do not confuse custom runtime availability with a stock MLX-LM reloadable artifact. Exercise the intended consumer before publishing.
- Check actual per-layer config after native GPTQ: fallback six-bit layers and skipped modules may coexist with the requested four-bit layers.
- Preserve zero conventions and source packing metadata;
gptq(v1),gptq_v2,gptq_p, AWQ, and legacy Marlin are not interchangeable byte streams. - Nontrivial GPTQ activation order cannot be passed to the exact affine holder by ignoring
g_idx. - A seven-bit source maps to eight-bit packed target storage; group-16 may use two quantized matmuls. Account for both storage and work in benchmarks.
- GGUF
_S/_Maliases in a block packer do not choose model-wide mixed precision. A raw block array is not a complete GGUF file. - ParoQuant pairs, angles, and channel scales and EXL3 trellis/scales/codebook must travel together.
- BNB checkpoint conversion still requires its source library validation and, for nested scales, bitsandbytes scale decoding; the existence of an MLX weight packer alone does not remove those loader dependencies.
- Conversion may cast scale/nonquantized tensors to FP16. BF16 activation support does not mean that conversion preserves every source BF16 parameter.
-
max_tokensandpromptare MLX generation arguments here. Do not assume unused Transformers kwargs are forwarded. - Do not publish generic speedups without stating shape, row count, dtype, dispatch path, sync method, median/aggregation, errors, and weight/scratch storage.
| Area | Paths |
|---|---|
| Entry point, dependencies, runtime construction |
gptqmodel/models/auto.py, gptqmodel/models/loader.py, pyproject.toml
|
| Runtime conversion/generation | gptqmodel/utils/mlx.py |
| GPTQ/AWQ packing | gptqmodel/utils/mlx_packing.py |
| GGUF packing transfer | gptqmodel/utils/mlx_gguf_packing.py |
| Native model quantization | gptqmodel/quantization/mlx_native.py |
| Automatic Torch-workflow bridges |
gptqmodel/quantization/rtn.py, gptqmodel/quantization/foem.py, gptqmodel/quantization/qqq.py, gptqmodel/looper/exllamav3_processor.py, gptqmodel/looper/paroquant_processor.py, gptqmodel/nn_modules/qlinear/gguf.py, gptqmodel/nn_modules/qlinear/fp8.py, gptqmodel/nn_modules/qlinear/bitsandbytes.py
|
| RTN/GPTAQ/FOEM |
gptqmodel/quantization/mlx_rtn.py, gptqmodel/quantization/mlx_gptaq.py, gptqmodel/quantization/mlx_foem.py
|
| FP8/GGUF/QQQ/BNB |
gptqmodel/quantization/mlx_fp8.py, gptqmodel/quantization/mlx_gguf.py, gptqmodel/quantization/mlx_qqq.py, gptqmodel/quantization/mlx_bitsandbytes.py
|
| ParoQuant |
gptqmodel/quantization/mlx_paroquant.py, gptqmodel/quantization/mlx_paroquant_optimize.py, gptqmodel/quantization/mlx_paroquant_quant.py, gptqmodel/quantization/mlx_paroquant_rotation.py
|
| EXL3 codebooks, packing, fallback |
gptqmodel/quantization/mlx_exl3.py, gptqmodel/quantization/mlx_exl3_fallback.py
|
| EXL3 complete pipeline | gptqmodel/quantization/mlx_exl3_pipeline.py |
| EXL3 Hessian/factorization/LDLQ |
gptqmodel/quantization/mlx_exl3_hessian.py, gptqmodel/quantization/mlx_exl3_block_ldl.py, gptqmodel/quantization/mlx_exl3_ldlq.py
|
| EXL3 transforms/search |
gptqmodel/quantization/mlx_exl3_gss.py, gptqmodel/quantization/mlx_exl3_hadamard.py, gptqmodel/quantization/mlx_exl3_regularize.py, gptqmodel/quantization/mlx_exl3_rms.py, gptqmodel/quantization/mlx_exl3_tiles.py
|
| Checkpoint holders | gptqmodel/nn_modules/qlinear/mlx.py |
| GPTQ/AWQ/group-16 inference |
gptqmodel/nn_modules/qlinear/mlx_gptq.py, gptqmodel/nn_modules/qlinear/mlx_awq.py, gptqmodel/nn_modules/qlinear/mlx_group16.py
|
| MLX dispatch steering |
gptqmodel/quantization/mlx_splitk_dispatch.py — pads decode/prefill row counts toward better Metal kernels; mirrors the MLX 0.32.4 quantized.cpp dispatch and stops steering once the installed MLX is newer (unparsable versions count as unmirrored) |
| Method-specific inference |
gptqmodel/nn_modules/qlinear/mlx_bitsandbytes.py, gptqmodel/nn_modules/qlinear/mlx_exl3.py, gptqmodel/nn_modules/qlinear/mlx_fp8.py, gptqmodel/nn_modules/qlinear/mlx_gguf.py, gptqmodel/nn_modules/qlinear/mlx_paro.py, gptqmodel/nn_modules/qlinear/mlx_qqq.py
|
| Validation policy |
AGENTS.md, .agents/skills/gptqmodel-torch-oracle-numerics/SKILL.md, docs/mlx-native-quantization-plan.md
|
| Tests/benchmarks |
tests/test_mlx_*.py, tests/benchmark_mlx_qwen38_*.py, tests/qwen38_27b_shapes.py
|