Repository navigation
BitsAndBytes
Documentation home · Quickstart · Lifecycle
On this page
GPT-QModel supports BitsAndBytes weight-only checkpoints with four-bit FP4 or NF4 weights and eight-bit INT8 weights. BitsAndBytesConfig selects the quantization representation; BACKEND.BITSANDBYTES executes it through the BitsAndBytes package on supported CPU/CUDA installations. Apple Silicon has a separate packed MLX execution route.
This is a distinct quantization method, not a four-bit GPTQ packing variant. Its codebooks, block maxima, nested statistics, and state tensors must stay together. A nominal bit width alone does not identify a compatible checkpoint.
This page describes source revision c1976918. GPT-QModel's BitsAndBytesConfig is its own configuration class; it is not transformers.BitsAndBytesConfig and does not accept an interchangeable set of constructor arguments.
Reading the diagram: The packing lanes are alternatives and do not consume calibration data. block_size is not GPTQ group_size; INT8 does not use FP4/NF4 codebook state.
Text version of the workflow
- FP4 or NF4 packing: Choose codebook and block → Quantize weight blocks → Persist four-bit state.
- INT8 packing: Select rowwise INT8 → Scale and round rows → Persist INT8 state.
- Checkpoint loading: Validate format metadata → Restore packed payload → Select a valid runtime.
| Identifier | Value | Meaning |
|---|---|---|
METHOD.BITSANDBYTES |
"bitsandbytes" |
Weight-only quantization method |
FORMAT.BITSANDBYTES |
"bitsandbytes" |
Checkpoint family used for loader/kernel selection |
Config format
|
"fp4", "nf4", or "int8"
|
Concrete weight representation, constrained by bits
|
BACKEND.BITSANDBYTES |
"bitsandbytes" |
BitsAndBytes-based Torch runtime |
BACKEND.MLX |
"mlx" |
Packed Apple Silicon runtime after compatible state transfer |
format="bitsandbytes" is a family/default request; it normalizes to fp4 at four bits or int8 at eight bits. It is not a fourth numerical representation.
| Bits | Format | Quantized representation | Scale granularity |
|---|---|---|---|
| 4 | fp4 |
Two four-bit codebook indices per byte | Absolute maximum per flat block |
| 4 | nf4 |
Two indices per byte into the NF4 codebook | Absolute maximum per flat block |
| 8 | int8 |
Signed INT8 row codes with weight_scb
|
One absolute maximum per output row |
NF4 and FP4 have different nonlinear codebooks and are not interchangeable. INT8 is the direct vectorwise path with no outlier-column routing in this implementation; it is not a promise to reproduce every LLM.int8 mixed-precision configuration.
Supported block_size values are 32, 64, 128, 256, 512, 1024, 2048, 4096, with default 64. This controls four-bit block quantization, not GPTQ group_size. The INT8 packer uses row-wise scales instead of these four-bit blocks. compress_statistics controls nested four-bit scale quantization; it is not an INT8 weight-compression switch.
Four-bit reconstruction is conceptually codebook[code] * block_absmax. The exact codebook, state dtype, nested offset/scales, and implementation rounding matter. Eight-bit reconstruction in the Torch helper is:
weight.to(torch.float32) * (weight_scb.to(torch.float32) / 127.0).unsqueeze(1)The method-specific dataclass fields are copied below. The class also inherits general model/preprocessing options; this is not the complete generated constructor.
@dataclass
class BitsAndBytesConfig(PreProcessorConfig):
bits: int = field(default=4, metadata={"choices": [4, 8]})
method: METHOD = field(default=METHOD.BITSANDBYTES)
format: Optional[str] = field(default=None)
group_size: int = field(default=-1)
desc_act: Optional[bool] = field(default=False)
sym: bool = field(default=True)
block_size: int = field(default=64)
compress_statistics: bool = field(default=True)| Parameter | Default | Normalization and validation |
|---|---|---|
bits |
4 |
Only 4 or 8 |
method |
METHOD.BITSANDBYTES |
Any different method is rejected |
format |
None |
Four bits defaults to fp4; eight bits defaults to int8
|
block_size |
64 |
None becomes 64; coerced with int() and checked against the supported set |
compress_statistics |
True |
Coerced with Python bool()
|
group_size |
-1 |
Forced to -1 after base initialization; not used as a BNB block size |
desc_act |
False |
Forced to False; GPTQ activation ordering is not used |
sym |
True |
Forced to True; this config convention does not make FP4/NF4 uniform integer quantization |
dynamic |
Inherited | Per-module overrides validated for BitsAndBytes constraints |
| API/helper | Behavior |
|---|---|
_normalize_bitsandbytes_format(value: Optional[str], *, bits: Optional[int] = None) -> str |
Strips/lowercases format, replaces hyphens with underscores, and validates the representation against the bit width |
_normalize_bitsandbytes_block_size(value: Optional[int]) -> int |
Default/coercion/set-membership validation |
allowed_quant_methods() |
(METHOD.BITSANDBYTES,) |
supported_export_formats() |
(FORMAT.BITSANDBYTES,) |
uses_weight_only_lifecycle() |
True |
quant_linear_init_kwargs() |
Passes format, block_size, compress_statistics to the linear implementation |
_update_output_payload(...) |
Serializes normalized format, block size, and compression into checkpoint configuration |
Configuration serialization must preserve the concrete format, not just the bitsandbytes family name. A four-bit checkpoint cannot be interpreted as NF4 by changing only a filename or loader option.
from gptqmodel.quantization import BitsAndBytesConfig
default_four_bit = BitsAndBytesConfig()
assert default_four_bit.format == "fp4"
assert default_four_bit.block_size == 64
assert default_four_bit.compress_statistics is True
nf4 = BitsAndBytesConfig(
bits=4,
format=" NF4 ",
block_size=64,
compress_statistics=True,
)
assert nf4.format == "nf4"
int8 = BitsAndBytesConfig(bits=8)
assert int8.format == "int8"Four-bit format="int8", eight-bit format="nf4"/"fp4", unsupported widths, unsupported block sizes, and unknown format strings raise ValueError. An empty format or the family token requests the width's default. Use actual booleans: bool("false") is True, so string-valued booleans are not a safe configuration encoding.
| Legacy key/property | Canonical field | Normalization |
|---|---|---|
bnb_quant_type |
format |
Width-aware format normalizer |
bnb_block_size |
block_size |
Block-size normalizer |
bnb_compress_statistics |
compress_statistics |
Boolean coercion |
The config retains compatibility properties with these names and normalizes old serialized kwargs. Prefer canonical fields in new code. They do not imply support for every similarly named Transformers BNB option; in particular, do not substitute quant_type=... as an undocumented BitsAndBytesConfig constructor argument. The low-level packers call that concept quant_type; the public config calls it format.
config = BitsAndBytesConfig(
bits=4,
format="nf4",
dynamic={
r".*\.mlp\..*": {
"bits": 8,
"format": "int8",
},
r".*\.self_attn\..*": {
"bits": 4,
"format": "fp4",
"block_size": 128,
"compress_statistics": False,
},
},
)Negative rules are ordered before positive rules. A dynamic entry may use the legacy format/block/compression aliases, but new entries should use canonical names. Each layer's effective width and format must agree; specify both when changing between four-bit and INT8 instead of relying on an inherited subtype.
Dynamic validation differs from top-level coercion: a layer's group_size must be -1 or None, truthy desc_act is rejected, and explicit sym must be the actual boolean True. Bits must resolve to 4 or 8. Invalid overrides fail during config normalization rather than becoming arbitrary GPTQ-style parameters.
See QuantizeConfig Reference, Quantization Lifecycle, and Looper Processors for inherited controls.
| Path | Requirements and advertised contract |
|---|---|
| Project Python requirement | >=3.10 |
| Project Transformers dependency |
transformers>=5.20.2 in requirements.txt at the documented revision |
| Project extra |
pip install 'gptqmodel[bitsandbytes]'; current metadata declares bitsandbytes>=0.50.2
|
| Torch kernel import gate | Accepts BitsAndBytes >=0.49.0; the source error hint recommends a recent 0.49.x, for example >=0.49.3
|
| Torch runtime | CPU or CUDA device; FP16/BF16/FP32 compute declarations; no automatic padding, no training support, no adapters |
| CPU packed four-bit optimization | Explicit opt-in, available BNB private packing helpers, AVX512-BF16 support, BF16 compute, compatible row/weight/thread geometry, and accuracy gate |
| MLX runtime | Apple Silicon (Darwin, arm64), Metal, mlx, and mlx_lm; install gptqmodel[mlx]
|
| MLX extra versions |
mlx>=0.32.4; mlx_lm>=0.31.3 on Python <3.11 or >=0.32.2 on Python ≥3.11 |
The package-extra minimum and the kernel's internal compatibility floor are deliberately reported separately. A class declaring CPU/CUDA and all platforms does not guarantee that every upstream BitsAndBytes build implements every device/dtype/operation. GPT-QModel does not advertise a separate ROCm BitsAndBytes backend here; verify upstream wheel/build support rather than inferring it from the backend name.
The MLX checkpoint holder inherits BitsAndBytes state handling and quantization machinery. Installing MLX alone is not a guarantee that the BitsAndBytes dependency can be omitted from the checkpoint setup path.
from gptqmodel import GPTQModel
from gptqmodel.quantization import BitsAndBytesConfig
from gptqmodel.utils.backend import BACKEND
config = BitsAndBytesConfig(
bits=4,
format="nf4",
block_size=64,
compress_statistics=True,
)
model = GPTQModel.load("/models/source-hf", quantize_config=config)
model.quantize(backend=BACKEND.BITSANDBYTES)
model.save("/models/bnb-nf4")
loaded = GPTQModel.load(
"/models/bnb-nf4",
backend=BACKEND.BITSANDBYTES,
)The source must be an unquantized, supported HF model. This method uses the weight-only lifecycle, not GPTQ Hessian/error-feedback quantization. Call quantize() without calibration; a supplied calibration dataset is ignored. Configured preprocessing is planned without forward passes, and optional smoothing acts on weight blocks. This lifecycle rejects adapter/EoRA generation, GPTAQ, and FOEM rather than adding a calibration pass for them.
For rowwise INT8, replace the config with:
config = BitsAndBytesConfig(bits=8, format="int8")For an already packed, compatible checkpoint on Apple Silicon:
model = GPTQModel.load("/models/bnb-nf4", backend=BACKEND.MLX)MLX additionally validates model/module compatibility and Metal availability. It is not selected by setting a BNB compute dtype to mps.
GPTQModel.load(...) recognizes supported quantization metadata, including the bitsandbytes method, and dispatches to the quantized loader. The loader restricts FORMAT.BITSANDBYTES to AUTO, BITSANDBYTES, or MLX; AUTO becomes BITSANDBYTES.
Hub location does not define a storage schema. The reviewed GPT-QModel kernels consume the following module-local state:
| Representation | Required/potential state keys under a module prefix |
|---|---|
| Four-bit codes | weight |
| Four-bit state |
weight_absmax, weight_quant_map, weight_quant_state
|
| Compressed four-bit statistics |
weight_nested_absmax, weight_nested_quant_map, plus packed metadata for the offset/nested state |
| Eight-bit codes/scales |
weight, weight_scb
|
| Bias, where present | bias |
The helper _packed_state_key_to_buffer_name(...) maps BitsAndBytes' QuantState.as_dict(packed=True) keys into these buffers: quant_state.bitsandbytes__<format> becomes weight_quant_state, and the scale/map entries become their corresponding weight_* buffers. _quant_state_payload() reconstructs the dictionary for QuantState.from_dict(...) on the current device.
| Buffer/schema detail | Source behavior |
|---|---|
| Four-bit buffer shapes/dtypes |
_buffer_spec_4bit(...) quantizes a CPU zero template and uses the installed BNB QuantState.as_dict(packed=True) schema; do not hard-code the packed metadata length |
INT8 weight
|
(out_features, in_features), torch.int8
|
INT8 weight_scb
|
(out_features,), torch.float32
|
Optional bias
|
(out_features,), module compute dtype |
Four-bit schema allocation therefore has a temporary dense CPU template of the layer's weight shape. Packed checkpoint storage does not imply that layer construction has no dense temporary allocation.
_load_from_state_dict(...) resizes the packed quant-state buffer to the serialized tensor shape, restores canonical CPU packing state before loading, and resets the CPU-packing-disabled flag. post_init() force-rebuilds four-bit QuantState; normal inference also refreshes it when required. _save_to_state_dict(...) restores the canonical packed layout before serialization. Private CPU execution packing is not the checkpoint representation.
Do not assume arbitrary Transformers BNB checkpoints are directly interchangeable. Recognition of quant_method="bitsandbytes" is metadata recognition, not a generic translation of all Transformers load_in_4bit/load_in_8bit arguments, BNB config keys, or alternative tensor-key layouts. The verified GPT-QModel loader/state-dict path expects the schema above. Incompatible external checkpoints require a schema-aware conversion or loading/dequantization in their producing library first; changing metadata alone does not convert packed state.
gptqmodel/utils/model_dequant.py detects this schema through weight_quant_state or weight_scb. convert_bitsandbytes_shard(...) reconstructs four-bit QuantState and calls bnb.functional.dequantize_4bit, or calls int8_vectorwise_dequant for eight-bit weights. Auxiliary state tensors are dropped from the dense output.
The file-oriented conversion entry point is:
def dequantize_model(
model_path: Path | str,
output_path: Path | str,
*,
target_dtype: torch.dtype = torch.bfloat16,
device: Optional[str] = None,
resume: bool = False,
) -> None:
...Example using the actual file converter, not GPTQModel.export(format="hf"):
from gptqmodel.utils.model_dequant import dequantize_model
dequantize_model("/models/bnb-nf4", "/models/bnb-dense")The input must have config metadata and safetensors shards; an existing output directory is rejected unless resume=True. The converter needs BitsAndBytes, and its explicit GPU device path requires CUDA. The result reconstructs quantized weights at the target dtype; it does not recover the original pre-quantization weights. The generic GPTQModel.export(..., format="hf") branch uses a different, GPTQ TorchLinear-restricted dequantizer and is not a substitute for this BNB conversion path.
BitsAndBytesLinear extends WeightOnlyQuantLinear. The source packing signature is:
def pack_original(
self,
linear: nn.Module,
scales: torch.Tensor,
zeros: torch.Tensor,
g_idx: torch.Tensor = None,
smooth=None,
):
...GPTQ scales, zeros, and g_idx are ignored. The packer obtains a two-dimensional weight matrix (including Conv/Transformers Conv1D layout handling), copies it to CPU, applies an optional smoother, and emits the selected BNB state.
| Path | Main calls and behavior |
|---|---|
| Four-bit packing | bnb.functional.quantize_4bit(..., blocksize=..., compress_statistics=..., quant_type=..., quant_storage=torch.uint8) |
| Four-bit inference |
bnb.matmul_4bit(...) with reconstructed QuantState; input is cast to compute dtype and output restored to input dtype |
| Eight-bit packing | Cast source weights to FP16; int8_vectorwise_quant(..., threshold=0.0); reject nonempty outlier-column routing |
| Eight-bit inference | Cast activations to FP16 and quantize to INT8, call int8_linear_matmul, then int8_mm_dequant with bias; return input dtype |
dequantize_weight() |
BNB four-bit reconstruction or FP32 rowwise INT8 reconstruction for explicit consumers |
The explicit reconstruction method has the signature def dequantize_weight(self) -> torch.Tensor:. It returns contiguous (out_features, in_features) weights in the ordinary nn.Linear.weight orientation, unlike the transposed result of GGUFTorchLinear.dequantize_weight(...). It accepts no device or dtype override. Four-bit reconstruction uses the saved BNB state; the INT8 helper reconstructs in FP32 on the weight device.
The default INT8 inference route therefore quantizes activations too. A weight-only checkpoint/lifecycle does not imply that every inference backend uses floating-point activations throughout. Compare its arithmetic separately from a floating-activation dequantized-weight oracle.
| Control | Default and effect |
|---|---|
GPTQMODEL_BITSANDBYTES_CPU_PACKED |
Default off; a truthy value enables CPU packed four-bit attempts |
GPTQMODEL_BITSANDBYTES_CPU_INT8_DENSE |
Default on under conservative geometry/feature gates; a false value disables it, but enabling it does not bypass those gates |
Both controls use env_flag: truthy values are 1, true, yes, on, and y, ignoring case and surrounding whitespace. Other explicitly supplied values are false.
The four-bit optimization requires BF16 compute, CPU weights, compatible BitsAndBytes helpers, AVX512-BF16, output width divisible by 32, even input width divisible by the BNB block size, inference/no active input-or-bias gradient requirements, and row limits derived from matrix/thread geometry. Initial packed output is compared with canonical BNB output using relative L2 error norm(packed - canonical) / norm(canonical) at most 2e-3; a zero reference norm requires exact equality, and nonfinite values fail. A packing/execution failure or accuracy failure restores canonical layout and disables that optimization for the module. This accuracy gate is not a global guarantee for every BNB computation or dtype.
The current _cpu_packed_max_rows(...) limits use the PyTorch intra-op thread count. Rows mean x.numel() // in_features, including flattened leading batch/sequence dimensions.
| Weight/thread condition, evaluated in order | Maximum rows |
|---|---|
| At most one thread | 32 |
At most 512 * 512 weight elements and at least four threads |
512 |
At most 512 * 512 weight elements and two or three threads |
64 |
| Larger weights with more than one thread | 48 |
Exceeding these limits restores canonical BNB execution; setting the environment flag does not override them.
The CPU INT8 dense route is gated to 512×512 weights, 64–256 activation rows, BF16 compute, AVX512-BF16, and no active input gradient requirement. It reconstructs temporary 512-feature tiles, casts activation tiles to FP32, and accumulates FP32 Torch matmul under cpu_bf16x3_matmul. It does not retain a dense weight cache between calls. Its arithmetic uses floating activations rather than the standard INT8-activation BNB path, so performance and numerical comparisons must identify which route ran.
The low-level quantizers in quantization/mlx_bitsandbytes.py are:
def quantize_4bit_weight_mlx(weight, *, quant_type="nf4", block_size=64, compress_statistics=False):
...
def quantize_int8_weight_mlx(weight):
...The low-level Metal four-bit helper defaults to NF4 and uncompressed statistics, while public BitsAndBytesConfig() defaults to FP4 and compressed statistics. Higher-level packing passes the selected config format explicitly.
Unlike the public format normalizer, this low-level helper requires the exact lowercase strings "nf4" or "fp4"; it does not normalize case or whitespace.
| Quantizer | Validation/output |
|---|---|
| Four-bit | Nonempty finite rank-two FP16/BF16/FP32 matrix; NF4/FP4 and supported block size; returns (packed, absmax_or_state) with uint8 packed codes |
| Four-bit compressed statistics | Quantized block maxima with nested absolute maxima, offset, and a dynamic codebook |
| Four-bit uncompressed statistics | Float32 absolute maximum per block |
| INT8 | Nonempty finite rank-two supported floating matrix; returns (codes, absmax) with signed row codes and float32 row maxima |
The Metal implementation includes BitsAndBytes-derived NF4/FP4 codebooks and explicit rounding/tie/subnormal handling. It is not replaced by a uniform four-bit integer quantizer.
On Apple Silicon, BitsAndBytesLinear.pack_original(...) can use these Metal helpers for large weights: at least 4 * 1024 * 1024 elements for four-bit, and 8 * 1024 * 1024 for INT8. Smaller weights use BNB packing. The four-bit Torch bridge requests uncompressed Metal maxima, then constructs the BNB codebook/shape/dtype metadata and, if configured, compresses the scale statistics with BNB quantize_blockwise(..., blocksize=256). This is distinct from requesting the low-level Metal helper's own nested-statistics path directly. Results are installed into canonical Torch/BitsAndBytes checkpoint buffers; Metal packing does not change the checkpoint family.
BitsAndBytesMlxQuantLinear in qlinear/mlx.py holds compatible BNB checkpoint tensors for transfer. It advertises MPS/Darwin, bits 4/8, FP16 loading, no training/adapters, and no automatic padding. native_payload(...) extracts native codes, scales, optional codebook, shape, block size, and bias. Runtime execution is provided by MlxBitsAndBytesLinear in qlinear/mlx_bitsandbytes.py.
| Runtime path | Behavior |
|---|---|
| Four-bit | Metal reads packed nibbles, looks up FP4/NF4 values, multiplies by block scales, rounds reconstructed values through FP16, and accumulates with float arithmetic |
| INT8 affine transfer | For widths divisible by 128, 64, or 32, repack_int8_affine(...) shifts signed codes by 128 and repeats row scales/biases for exact affine mapping; executes through mx.quantized_matmul
|
| INT8 custom Metal fallback | Reads signed row codes and scales directly; reconstructs through FP16 and accumulates with float arithmetic |
| Output contract | Preserves input activation dtype; empty inputs return empty/zero-shaped output of that dtype; incorrect final width raises ValueError
|
The four-bit Metal kernel has distinct FP16/BF16/FP32 output specializations and row/thread geometry choices. Small-output operations can write the activation dtype directly; other paths use an FP32 intermediate and cast the result. The checkpoint holder's FP16 dtype declaration and the runtime's activation dtype behavior are different contracts; neither justifies silently treating BF16 as FP16 output.
The MLX INT8 routes multiply floating activations by the transferred quantized weights; they do not run BNB's INT8 activation quantizer. The custom Metal route also rounds reconstructed weights through FP16, while the affine route uses MLX quantized matmul. Exact code/scale transfer and preserving the activation output dtype do not imply identical arithmetic to either the Torch BNB A8W8 path or a full-FP32 weight-only oracle.
MLX packed inference is not a dense dequantized-weight cache. Quantized codes, codebooks, scale vectors, and packed affine layouts may persist; reconstructed weights are consumed inside the operation.
| Symptom/mistake | Check |
|---|---|
| NF4 expected, FP4 produced | Four-bit public default is FP4; set format="nf4" explicitly |
| Invalid format/width | NF4/FP4 require four bits; INT8 requires eight bits |
group_size used to tune BNB blocks |
Use block_size; top-level grouping/ordering/symmetry options are coerced to the BNB conventions |
| Loaded bytes but incorrect results | Preserve format codebook and all quantization-state tensors; do not reinterpret FP4 as NF4 |
| Arbitrary HF BNB checkpoint fails | Compare config and tensor names with the verified GPT-QModel schema; method metadata alone is insufficient |
| Packed CPU path not selected | Enable with a truthy value; verify BF16 compute, AVX512-BF16, helpers, row/thread gates, and prior accuracy failures |
| Different INT8 output after CPU-path change | Standard BNB quantizes activations; tiled dense fallback does not use the same arithmetic |
| MLX import/selection failure | Verify Apple Silicon/Metal and both checkpoint and MLX dependencies; a Torch MPS tensor is not the MLX runtime |
| String booleans behave unexpectedly | Python boolean coercion treats nonempty strings as true |
| Adapter/training request rejected | Both BNB and its MLX holder advertise neither training nor adapter support |
Benchmark with the same checkpoint, activation dtype, row count, shape, thread count, and selected runtime path. Quantizer parity, weight reconstruction error, activation-quantized inference error, and visible output rounding are separate measurements.
Paths below are relative to the repository and link to the documented revision.
| Source | Responsibility |
|---|---|
| gptqmodel/quantization/config.py | Enums, format/block normalizers, config, aliases, dynamic rules |
| gptqmodel/utils/backend.py | Backend enum values |
| gptqmodel/nn_modules/qlinear/bitsandbytes.py | Torch buffers, packing, state loading, CPU/CUDA inference |
| gptqmodel/nn_modules/qlinear/mlx_bitsandbytes.py | Packed Metal inference and INT8 affine repacking |
| gptqmodel/quantization/mlx_bitsandbytes.py | Metal NF4/FP4/INT8 quantizers and nested statistics |
| gptqmodel/nn_modules/qlinear/mlx.py | MLX checkpoint holder and native payload extraction |
| gptqmodel/utils/mlx.py | Model conversion/selection for MLX |
| gptqmodel/models/auto.py | Quantization metadata recognition and public dispatch |
| gptqmodel/models/loader.py | Checkpoint loading and allowed backends |
| gptqmodel/models/base.py | Quantization/save lifecycle entry points |
| gptqmodel/utils/model_dequant.py | Packed-state detection and dense safetensors conversion |
| gptqmodel/utils/env.py | Runtime flag parsing |
| gptqmodel/nn_modules/qlinear/torch.py | Generic HF export's TorchLinear-only dequantization restriction |
| pyproject.toml | Dependency extras and version floors |