Repository navigation
NF12
🏠 Home · 🧭 Choose a method · 🚀 Quickstart · 🧰 Help
On this page
NF12 (NyoomFloat12) stores BF16 weight bits losslessly using a 12-bit main payload plus an escape table. It is useful when you want to reduce weight storage without introducing another lossy weight-quantization step. This page is for readers exporting an NF12 checkpoint, reducing source storage during calibration, or integrating packed inference.
NF12 originates from Together AI's NyoomFloat12 work. GPT-QModel implements the tiled layout from dphnAI/nf12, not the paper's byte layout. Encoding and decoding BF16 weights is bit-exact; converting FP16/FP32 source weights to BF16 first can be lossy. Lossless weight storage does not promise bit-identical inference across kernels with different accumulation orders and output rounding.
This page describes source revision 4682e21d9. NF12 uses the calibration-free weight-only lifecycle, not GPTQ Hessian optimization.
| Choose this when… | Consider another path when… |
|---|---|
| Your source is BF16 and exact weight reconstruction matters | You need four/eight-bit lossy compression: compare GPTQ, BitsAndBytes, or FP8 |
| You want an NF12 model checkpoint | Use NF12Config(); automatic source storage alone does not change your output format |
| You want less source-weight storage while quantizing to another format | Use AUTO_NF12; the final target remains independent |
| You are writing NVIDIA CUDA inference code against packed weights | Use the lightweight API; it does not support AMD ROCm |
| You need native AMD ROCm or Apple Silicon execution | Use the model-lifecycle HIP or MLX paths within their explicit device/shape limits |
Lossless does not mean fixed-size or identical inference. Escaped groups add storage beyond the main 12 bits/value. Converting an FP16/FP32 source to BF16 may lose information before encoding. Different GEMM reduction orders may also change output bits even when decoded weights are exact.
Before you start: install GPT-QModel, use a supported model, and check each selected matrix: output width must divide by 16, input width by 64. There is no auto-padding. The portable Torch path is available where its operations are supported; native NVIDIA CUDA requires SM80+ and its toolchain, native AMD HIP is limited to the listed RDNA4 targets, and MLX needs a compatible Apple Silicon model. See dependency details.
Read in three passes: export or use source storage → understand the representation → integrate advanced APIs. For a broader comparison, see Dtypes and Formats.
from gptqmodel import GPTQModel
from gptqmodel.quantization import NF12Config
from gptqmodel.utils.backend import BACKEND
config = NF12Config() # bits=12; no calibration dataset
model = GPTQModel.load("/path/to/bf16-model", quantize_config=config)
model.quantize()
model.save("/path/to/nf12-model")
reloaded = GPTQModel.load("/path/to/nf12-model", backend=BACKEND.NF12)The normal model definition and module-plan rules still decide which modules are quantized. Every encoded matrix must have out_features % 16 == 0 and in_features % 64 == 0; this backend does not automatically pad incompatible dimensions. NF12Quantizer accepts nn.Linear, Transformers Conv1D, and ConvNd weights after conversion to an output-by-input matrix.
For lazily loaded source weights without a PyTorch version counter, NF12 packing makes a temporary copy before taking matrix views. This avoids Inference tensors do not track version counter errors during direct packing. See the NF12 weight preparation code.
The factory alternative is QuantizeConfig(method="nf12", bits=12). The public symbols are NF12Config, METHOD.NF12, FORMAT.NF12, and BACKEND.NF12; method, format, and backend all serialize as "nf12". The method-aware legacy BACKEND.TORCH selector also resolves to BACKEND.NF12.
For supported quantization continuation runs, packed-module bundles record the data-dependent escape-table row count and recreate that shape before loading tensors. Nonempty escape tables no longer cause a resume schema mismatch. This metadata belongs to execution checkpoints; the NF12 inference format is unchanged.
Older AD-suffixed Python symbols, module paths, and the backend string "nf12_ad" have been removed, not retained as aliases. Update imports and explicit backend selections when upgrading. The method/format string "nf12" and packed checkpoint layout are unchanged; this API rename does not require requantizing existing NF12 weights.
BaseQuantizeConfig.auto_dtype_cast defaults to AutoDtypeCast.AUTO_NF12 ("auto_nf12"). This is a source-storage policy for quantization forwards, not a request to export an NF12 checkpoint. The requested quantization method, bits, and output format remain separate.
from gptqmodel import AutoDtypeCast, QuantizeConfig
config = QuantizeConfig(bits=4, auto_dtype_cast=AutoDtypeCast.AUTO_NF12)
dense_source_config = QuantizeConfig(bits=4, auto_dtype_cast=AutoDtypeCast.OFF)The enum is exported from gptqmodel and gptqmodel.quantization; the config also accepts the exact strings "auto_nf12" and "off". The field is serialized in quantize_config.json. OFF retains dense source storage where the policy would otherwise pack it.
Eligibility is checked per materialized source module:
| Requirement | Contract |
|---|---|
| Device | NVIDIA compute capability 8.x or newer, or AMD RDNA4 gfx1200/gfx1201/gfx1250 through the separate HIP runtime. Other devices stay dense. |
| Module | Exact nn.Linear, calibration HookedLinear, or nn.Embedding with max_norm=None, when selected by the lifecycle. Arbitrary subclasses and other module types are not automatically eligible. |
| Weight | Materialized two-dimensional matrix [N, K], with N % 16 == 0 and K % 64 == 0. |
| Dtype | BF16, or FP16 only if every FP16 bit pattern survives FP16 → BF16 → FP16, including signed zero. Non-exact FP16 and other dense dtypes stay dense. |
The policy does not repack METHOD.NF12, a selected native floatx forward plan, or an already quantized module used for post-quantization replay. Floatx sources decoded to eligible BF16 matrices can use it; the native floatx path takes precedence. Norm vectors and biases are not NF12-encoded matrices.
EXL3 also honors this default policy for eligible source modules: its preprocessor installs packed source storage, and the quant_source boundary restores the exact dense weight bits before the EXL3 task consumes them. auto_dtype_cast="off" keeps eligible sources dense instead. The final trellis format is unchanged; see EXL3 source restoration.
Dynamic exclusions and entirely skipped decoder layers stay dense during output replay. AUTO_NF12 conversion is not applied merely to refresh such a layer. Before excluded-layer replay, leaked packed AUTO_NF12 buffers raise RuntimeError rather than silently changing the dense reference path. Packing is transactional: failed conversion leaves/restores the original dense weight and does not record a successful event. See excluded-layer replay.
During eligible calibration forwards, linear weights remain canonical NF12; embedding tables use the row-major layout. Immediately before the quantizer consumes a source, the original dtype and weight bits are restored and rebound to its task. Packed source buffers are then released; GPTQ + EoRA can retain them until the FP32 w - wq difference is computed, using fused decode/subtract when available. EoRA's subsequent capture uses the quantized layer, not a newly packed dense source. Hooked replacements preserve calibration hooks, removable-hook handles, rotation, and W4A replay behavior.
model.auto_dtype_cast_events records applied conversions with module name, device, kind, source dtype, and dense/packed byte counts; device telemetry emits auto_dtype_cast_nf12. Smaller source storage does not guarantee a lower whole-run memory peak or faster calibration. Lossless reconstruction also does not make packed and dense GEMM reduction orders identical. Packing/restoration errors are not reported as successful compression. See the source policy and measured comparisons and implementation.
This is an encoding, not a fitted quantization algorithm. No Hessian, calibration objective, or learned codebook is needed to preserve a BF16 bit pattern.
BF16 has one sign bit
Zeros/subnormals and Inf/NaN have special rules; the codec preserves their original bits rather than using this normal-value formula.
NF12 takes advantage of exponent locality. It selects a matrix-wide base
A group of eight escapes if any value lies outside the window, or if its metadata would collide with the zero sentinel. The main payload then points to all eight original BF16 values in the escape table. This is how the representation remains lossless for values that do not compress well. Automatic base selection minimizes escaped groups, breaking ties toward the smallest base.
For
The 12-bit main payload alone is 25% smaller than BF16. The actual saving depends on escapes; enough escapes can eliminate that saving. See the tiled layout specification and Torch codec.
| Option | Default / contract |
|---|---|
bits |
12; the only supported NF12 width |
method / format
|
METHOD.NF12 / FORMAT.NF12, both "nf12"
|
group_size |
Forced to -1; not a grouping knob |
desc_act / sym
|
Forced to False / True; not tunable quantization policies |
exp_base |
None: choose each matrix's exponent base to minimize escaped groups. A global integer in [0, 120] forces the same base policy for all modules. |
dynamic |
Module exclusion and compatible overrides follow the shared config machinery; a per-layer exp_base is rejected. Dynamic bits must remain 12 and group size must remain -1/None. |
NF12Config inherits PreProcessorConfig, but direct NF12 packing explicitly rejects a non-None smooth argument. Do not combine smoothing with a lossless-storage claim. A forced exp_base uses the Torch codec because the native encoders select their own base.
For a module with weight matrix [N, K], the saved state contains:
| Buffer | Dtype | Shape |
|---|---|---|
nf12_packed |
torch.uint8 |
[N, K * 3 // 2] |
nf12_escapes |
torch.int16 |
[E, 8] |
nf12_exp_base |
torch.uint8 |
[1] |
bias, when present |
Source bias dtype at pack time | [N] |
The core packed matrix takes 12 bits per weight, but the escape table and exponent add overhead: matrix payload bytes are N*K*3//2 + E*16 + 1, excluding bias, config, and container headers. Do not assume every model is exactly 25% smaller than BF16. Escapes preserve out-of-window BF16 groups, including special values and their payload bits.
Packed bytes and the nf12_packed / nf12_escapes / nf12_exp_base keys are shared by the CUDA, HIP, and Torch codec paths and are unchanged by the API rename. The Hugging Face/Accelerate checkpoint loader sizes variable-length nf12_escapes buffers from checkpoint metadata before assigning tensors; keep these buffers in exports rather than replacing them with empty placeholders.
Inference retains only packed weights, escapes, exponent metadata, and bias. A fallback may reconstruct a matrix for the current forward, but does not cache that dense weight between calls. Exponent-buffer replacement and ordinary versioned mutation invalidate the scalar exponent cache. After manually changing an inference-mode exponent tensor in place, call the layer's clear_weight_cache().
GPTQModel.load(path, backend=BACKEND.MLX) can transfer a compatible NF12 checkpoint into MlxNF12Linear on an MLX-LM-supported Apple Silicon model. Canonical 12-bit bytes, int16 escape groups, and exponent base are preserved; the holder supports FP16/BF16 and no adapters/training. K remains divisible by 64 and N by 16. For 1–15 flattened activation rows, execution fuses packed decode and dot on Metal, sharing activation loads across adjacent output blocks. At 16 rows, output widths below 65536 use the dot path and widths at least 65536 use a packed matrix tile. Larger calls use an exact temporary BF16 decode, never a retained dense weight. All paths return the input activation dtype. This path does not require the NVIDIA CUDA or RDNA4 HIP NF12 extension. See native MLX inference.
The short-row Metal dot planner chooses an adjacent-output-block count that divides N // 16, reducing it to two or one block when needed. Thus every valid 16-column output tail is written, including widths such as 1040 and 2064; no extra 32/64-column alignment is required. The packed format and FP16/BF16 output-dtype contract are unchanged.
The low-level MlxNF12Embedding holder can gather exact BF16 rows directly from canonical packed tables, with arbitrary leading ID dimensions, duplicate IDs, and positive-zero output for invalid IDs. It requires embedding width divisible by 64 and vocabulary size divisible by 16. It is not automatically selected by GPTQModel.load(..., backend=BACKEND.MLX): the current loader does not emit NF12 embedding modules. MLX variants are still absent for NF12 encoding, MoE, row-major embedding, BF16-twin operations, and FP32 subtraction; the kernel manifest records these gaps explicitly.
NF12Linear declares portable device support and FP16/BF16/FP32 activation dtypes; optional native execution requires BF16 activations, matching activation/packed-weight devices, and BF16 bias when present. A mixed-dtype bias bypasses native execution rather than narrowing the checkpoint bias. CPU fallback, FP16 activations on any device, an autocast dtype that cannot preserve BF16 weights, or a mixed-dtype bias select temporary FP32 decoding and matmul/bias addition with autocast disabled, followed by one cast to the activation dtype. This preserves BF16 weight range, including values outside FP16 range and below its subnormal range. Other non-native calls use decode plus the shared FP32-reduced matmul helper. Meta inputs return shape/dtype metadata without decoding; meta and empty inputs apply an adapter at most once. The module does not cache a dense weight.
When autograd is enabled and the activation requires gradients, NF12Linear uses decode plus differentiable matmul, including for BF16 inputs on NVIDIA CUDA and AMD RDNA4. Eligible calls under torch.no_grad() or torch.inference_mode() use the fused native path. This preserves activation gradients through frozen NF12 projections.
Checkpoint buffers assigned under inference mode are normalized once during post_init(). The CPU decode-and-matmul forward avoids querying Tensor.is_inference(), which previously broke Dynamo graphs even though the CPU backend declares Inductor support. See the NF12 implementation.
| Target | Execution path |
|---|---|
| NVIDIA CUDA | Vendored fused decode/GEMM kernels when loadable; JIT build requires SM80+, CUDA toolkit 13.0+, CMake 3.26+, Ninja, and a C++ compiler. The build checks all visible CUDA architectures. |
| AMD RDNA4 | HIP encoder/decoder and fused small-M / WMMA paths for gfx1200, gfx1201, gfx1250; larger or ineligible calls use sliced decode plus rocBLAS. Requires a working ROCm development toolchain. This is not a declaration of native NF12 support for every AMD family. |
| Other devices, non-BF16 activations, or unavailable native extension | Torch codec plus matmul where the target's Torch operations are supported; no native speed guarantee. |
On Windows the portable Torch path remains importable without POSIX fcntl. The vendored native CUDA JIT is not supported on Windows.
The HIP linear wrapper normalizes BF16 activations to a contiguous [M, K] matrix, where M is the product of all leading dimensions, and restores x.shape[:-1] + (N,) on every return. Sliced/padded-row and higher-rank activations can therefore use the same routes without reading storage gaps or flattening the returned shape. Eligible calls use a token-count-specialized fused kernel for M <= 15; eligible calls with 16 <= M <= 256 use WMMA on gfx1200/gfx1201, and other calls use row-sliced decode/GEMM. gfx1250 lacks the required WMMA instruction and does not use that matrix-core route.
Split-K scratch comes from PyTorch's allocator and follows the packed tensor's device/current stream, including non-current GPU ordinals. Split partials remain unbiased FP32 values; the reduction adds bias once before BF16 output conversion. Launch tuning is shape-specific; see the HIP implementation and measurements, not a universal speedup claim.
The recorded validated environment is ROCm SDK 10.1.0 / HIP 7.16.26385, PyTorch 2.16.0.dev20261025+rocm10.1, free-threaded Python 3.14, and Radeon AI PRO R9700 (gfx1201). This is a tested combination, not a newly imposed minimum. The HIP component version (torch.version.hip or hipconfig --version) is distinct from the ROCm SDK release.
RDNA4 buffering and split-K choices are derived from the launch shape. At M=128, the WMMA planner can split M for long-K or narrow-N cases to improve occupancy, rather than applying the same choice to every shape. The refreshed upstream format primitives and HIP kernels preserve the existing packed-byte layout; no checkpoint migration or new NF12Config flag is required. The registered nf12_hip Torch API now includes canonical and row-major embedding lookups, the row-major transform, decode_fp32, and sub_fp32, alongside encode, decode, linear, selftest, and arch_info. The JIT source set also includes MoE helper kernels and a BF16-twin packer; compiled source availability is not a public fused-MoE execution API or automatic MoE dispatch.
For native C-ABI callers, dense encode/decode and fused linear reject invalid 16-by-64 geometry rather than truncating it; encoders also check workspace capacity before writing. MoE helpers check slice bounds, BF16 pointer/row-stride alignment, metadata alignment, and signed-index byte spans; malformed arguments return hipErrorInvalidValue. Routing endpoints use a clamped running maximum within the assignment count, so negative, decreasing, or oversized offsets cannot select guard rows. Zero-assignment alignment publishes zero offsets without requiring nonempty scratch.
WMMA launch bounds use X up to 2147483647 and Y up to 65535, not the earlier blanket 32767-block cap. Routed WMMA distributes excess assignment tiles across Y/Z; grids beyond the supported limits return hipErrorNotSupported before launch. The native wmma_available() helper probes ordinals outside its bounded 64-device cache instead of declaring them unsupported solely because they are uncached. Encoder scheduling uses four super-atoms per analysis warp and two-super-atom count/write units while preserving flat escape order and packed bytes. These are native implementation contracts, not new user configuration flags; see the HIP device/workspace reference.
The CUDA compatibility loader delegates to gptqmodel_nf12. It reuses an already registered complete operator set (encode, decode, linear, embedding, rowmajor_pack, rowmajor_embedding, encode_moe, moe_linear, _fused_moe), but no longer tries to import a standalone nf12 package. Otherwise it loads packaged native libraries when present or JIT-builds the vendored sources; no separate nf12 installation is required. JIT artifacts live under $XDG_CACHE_HOME/gptqmodel/nf12/<cache-key> (or ~/.cache/gptqmodel/nf12/<cache-key>). The key includes sources, visible architectures, Torch/CUDA identity, Python executable, and build-tool environment. Use XDG_CACHE_HOME to relocate the CUDA cache. HIP remains separate, uses GPTQMODEL_NF12_HIP_BUILD_ROOT, and stages sources in its writable cache rather than modifying an installed package. The older AD-suffixed cache paths and HIP build-root variable are no longer used. CUDA_HOME/CUDACXX and ROCM_PATH/ROCM_HOME help locate the respective toolchains.
The GPT-QModel wheel also includes gptqmodel_nf12, which imports without initializing the quantization framework. This API is for NVIDIA CUDA, not the portable NF12Linear wrapper: its native loader explicitly rejects ROCm. Import alone does not load or build the native libraries; the first operation that needs them does.
The FP32-output projection helpers import Triton on first use. Their dependency is separate from the CUDA/CMake toolchain used to build the native BF16 codec and inference operators.
import torch
import gptqmodel_nf12 as nf12
weight = torch.randn(128, 256, device="cuda:0", dtype=torch.bfloat16)
packed = nf12.encode(weight)
x = torch.randn(2, 256, device=weight.device, dtype=torch.bfloat16)
y_bf16 = nf12.linear(x, packed)
y_fp32 = nf12.linear_fp32(x.float(), packed)
rows = torch.tensor([0, 17, 63], device=weight.device, dtype=torch.int64)
shortlist = nf12.linear_fp32(x.float(), packed, rows=rows)
y_wide = nf12.linear_bf16_fp32(x, packed)
ids = torch.tensor([0, 17], device=weight.device, dtype=torch.int64)
embeddings = nf12.embedding(ids, packed)
hc_embeddings = nf12.embedding_repeat4(ids, packed) # [2, 4, 256], BF16encode() requires a two-dimensional BF16 CUDA weight with positive N divisible by 16 and K divisible by 64. NF12Weight holds packed bytes, escapes, and an integer exponent base; decode() returns the original BF16 matrix. linear() takes BF16 activations and returns BF16; linear_fp32() takes FP32 activations and returns FP32, optionally selecting output rows with a one-dimensional CUDA int32/int64 rows tensor. FP32 projection bias, when supplied, must be FP32 [N] on the activation device. linear_bf16_fp32() takes BF16 activations but returns FP32. Its eligible short-K multirow path uses packed Tensor Core tiles; other shapes use the general FP32 projection path.
embedding() accepts int32/int64 CUDA IDs and returns BF16 rows with shape ids.shape + (K,); out-of-range IDs produce zero rows. NF12Linear and NF12Embedding are module wrappers, and encode_moe() / fused_moe() expose packed expert inference. gptqmodel_nf12.NF12Linear accepts BF16 or FP32 activations with matching output dtype; it is distinct from gptqmodel.nn_modules.qlinear.nf12.NF12Linear in the model lifecycle, which also declares FP16 fallback support. Module-wrapper state uses weight.nf12_packed, weight.nf12_escapes, and weight.nf12_exp_base, whereas quantized model linears use the unprefixed buffer names below. Norm vectors and other one-dimensional checkpoint tensors are not NF12-encoded matrices.
FP32 small-row projections read packed weights directly. Larger full-output batches may decode a temporary dense matrix for the current GEMM, but no dense weight persists between calls. BF16 linear also selects fused or sliced decode/GEMM by architecture and shape; the SM80 N=384–512, K=5120, M=128–1023 prefill band uses call-local decode/GEMM. These are dispatch policies, not configuration knobs. Different reduction orders can affect close scores: qualify token selections and routing decisions in the serving application rather than assuming bitwise dense-GEMM parity. See the API source and packed FP32 projections.
decode_fp32(weight) (also weight.decode_fp32()) decodes directly to FP32 without an intermediate BF16 matrix when the native widening operator is present; its result is bit-identical to decode(weight).float(). sub_fp32(weight, wq) fuses that widening decode with subtraction of an FP32 reconstructed weight matrix of the same shape/device, avoiding a separate FP32 source matrix for EoRA. Already-loaded runtimes without these optional operators use decode/cast or decode/subtract fallbacks. These helpers return current-call tensors, not retained dense weight caches. NF12Linear.dequantize_weight(dtype=torch.float32) in the model lifecycle uses the same native widening path on CUDA or supported HIP when available.
embedding_repeat4(ids, weight) accepts the same int32/int64 CUDA IDs and canonical NF12Weight as embedding(), and returns BF16 with shape ids.shape + (4, K). Each copy preserves the exact BF16 bits of the ordinary lookup; negative or out-of-range IDs produce four positive-zero rows. Empty IDs preserve the output shape, and the native binding makes noncontiguous IDs contiguous when needed.
With the current native operator library, the kernel decodes each packed row block once and writes all four activation copies in one launch. This avoids the intermediate single-copy activation and separate repeat operation for four-way hyper-connections. If an older operator library is already loaded and lacks embedding_repeat4, the helper uses its single-copy lookup followed by an activation repeat, preserving the same shape and BF16 bits. The persistent weight remains the original packed bytes and escapes. Call this helper explicitly when four copies are needed; embedding() and NF12Embedding.forward() continue to return one copy.
On SM80 at K=5120, int64 IDs with 1536 <= num_ids <= 2048 use a specialized one-ID CTA with streaming output stores. Other shapes follow the shared embedding dispatch. The four-copy embedding reference records exact-output checks and paired measurements against dense lookup plus repeat. Those measurements cover this activation stage, not whole-service performance.
Speculative verification can submit several physical rows for one logical batch-one request. The lightweight API exposes explicit helpers that retain the ordinary one-row kernel configuration and reduction tree:
| Helper | Input and behavior |
|---|---|
linear_staged_b1(x, weight, bias=None) |
BF16 CUDA matrix [M, K], 1 <= M <= 8; returns BF16. Preserves ordinary single-row arithmetic: the GEMV route launches once per row; otherwise small-M shares a packed traversal with the M=1 configuration. Raises on unsupported one-row routes or staged configurations rather than switching to a different reduction. |
linear_fp32_staged_b1(x, weight, bias=None) |
FP32 CUDA activations containing 1–8 flattened rows; returns FP32. Full output only, with no rows selector. Rejects the short-K full-head route (K <= 512 and N >= 1024). Six long-K rows share one packed traversal with independent one-row reductions. |
prepare_fp32_gate(weight) |
Load-time preparation for CUDA weights shaped [128, 5120] or [384, 5120]; other shapes/devices raise ValueError. Returns None. Subsequent linear_fp32() M=1 and linear_fp32_staged_b1() M=6 calls use the prepared view automatically. |
linear_fp32_gate_rows(x, weight, bias=None, *, rows_n=2, rows_m=6, unroll=False) |
Explicit multi-row projection for a prepared gate. Takes FP32 CUDA [..., 5120] activations and optional FP32 bias of shape [out_features] on the weight device. Returns FP32 [..., out_features]. Requires Triton Gluon. |
prepare_fp32_gate() temporarily decodes BF16 and builds a second lossless, compressed K-contiguous view. That view is weakly owned by the NF12Weight; the original tiled NF12 representation remains available for other calls. The extra packed payload is 983,040 bytes for 128 rows or 2,949,120 bytes for 384 rows, plus its escape table. No dense BF16 weight remains resident. Prepare eligible weights before inference or graph capture, allowing for this additional compressed storage.
The prepared view tracks packed/escape tensor replacement, ordinary versioned in-place writes, and exp_base edits. A detected change discards the stale view. The automatic routes can fall back, and linear_fp32_gate_rows() requires preparation again. Every explicit prepare_fp32_gate(weight) rebuilds it. Call it again after unversioned inference-tensor or .data writes. If rebuilding fails, the previous view stays discarded; a source change detected during preparation raises instead of publishing a mismatched view.
linear_fp32_gate_rows() computes all nonempty activation rows in one launch and uses the prepared batch-one gate's FP32 reduction order independently for each row. rows_n can be 1, 2, 4, or 8 and must divide the output width. rows_m can be any integer from 1 through 6. The activation row count is a runtime argument: for a fixed kernel configuration, Gluon compiles full-tile and masked-tail variants rather than a variant for every row count. See the public API and gate implementation.
Ordinary linear_fp32() also has an automatic prefill path for full [384, 5120] router weights with at least 384 activation rows on NVIDIA compute capability 8.x or newer. It decodes a BF16 weight only for the current call and runs TF32x3 Tensor Core GEMM, preserving FP32 activations and output. Other large full-output shapes use temporary FP32 decode/GEMM. These policies do not change NF12Config or promise bitwise agreement with every dense GEMM. See the staged projection and router reference for dispatch, numerical checks, and measured workloads.
The FP32 prefill gate passes the activation row count to Triton at runtime. Different eligible prefill lengths no longer require distinct compiled variants just because the row count changed. The tile shape and accumulation order are unchanged. See the prefill gate implementation.
The general FP32 packed-decode kernel and the short-K BF16-input/FP32-output Tensor Core kernel also take the row count at runtime. Different row counts can reuse a compiled variant when the tile and other specialization inputs match. This reduces recompilation across batch widths without changing accumulation order.
gptqmodel_nf12.abi pins the upstream dphnAI/nf12 1.1 surface at commit 97c4101541a8d92d2b3d89c910c452cfa8efc9c9: 25 C entry points, both weight structs, status/version values, 14 upstream Torch schemas, the exported-symbol map, and 21 Python names. GPT-QModel adds operations without changing those upstream signatures. This upstream pin is separate from the GPT-QModel source revision linked on this page.
gptqmodel_nf12.kernels records the registered CUDA and ROCm schemas, their signatures, and any MLX variants. CUDA uses torch.ops.nf12; ROCm uses torch.ops.nf12_hip, so callers must not substitute namespaces or assume every operator/signature exists on both. In particular, CUDA linear exposes static_weights, while the HIP schema does not. The lightweight Python package remains NVIDIA-only.
After assembling the lightweight Torch module wrappers from partial or sharded loads, call gptqmodel_nf12.verify_weights(model). It raises ValueError naming weights still on meta or whose escaped-group count disagrees with their escape rows; a non-Torch-module argument raises TypeError. This consistency check is not a native-kernel availability or numerical-performance test. library_path(name=None) resolves the codec library by default; an explicit "libnf12_torch.so" requests the Torch library. See the upstream ABI contract and verification implementation.
gptqmodel_nf12.linear(x, weight, bias=None, *, static_weights=False) automatically selects kernels by device and shape. SM103 adds a single-row GEMV family and direct/asynchronous small-M families for selected 2–8-row shapes. These paths require no workspace; shapes outside their heuristic bands continue to the existing narrow, small-M, large-M, or call-local decode/GEMM routes. Other SM80+ devices retain those existing routes. The HIP backend has separate dispatch.
SM103 short-K prefill (K <= 128, 512 <= N <= 8192, M >= 64) can use external decode/GEMM. This is automatic dispatch, not an NF12Config option or a throughput guarantee. Packed weights and escapes remain the persistent representation; any decoded slice belongs only to its consuming call.
The optional static_weights=True argument permits early packed-weight and escape reads before a preceding programmatically dependent kernel finishes. Both buffers must already be ready when that kernel releases dependents and remain unchanged until this linear operation completes. Activations, bias, and output still wait. The caller owns this readiness promise; ordinary stream ordering alone does not establish it, and the option does not replace cross-stream events. The default is False, including module forwards; paths without early staging may ignore it. See the CUDA dispatch and weight-readiness reference for the native flow and shape heuristics.
Short-M scheduling also includes an SM120/121 specialization for M=16, K <= 2048, and 8192 <= N < 65536, plus shape-dependent RDNA4 split-K tuning. M counts flattened activation rows, not requests: input IDs [B, S] normally produce M=B*S. These choices are automatic and workload-specific; see the short-M dispatch and measured sweep.
NF12 CUDA MoE forwards no longer copy expert escape offsets to the host and synchronize the stream on every call. NF12MoEWeight validates offsets at load time; native forwards retain structural checks and bound each expert's local escape lookup on device, allowing prepared MoE weights to participate in CUDA graph capture. Direct low-level callers still own value validation and must not mutate validated offsets or escape metadata into an invalid state.
Staged CUDA decode paths, including SM90 large-M, carry the expert-local escape count and check each escape index against that slice rather than trusting a staged index. These changes preserve packed checkpoint bytes and do not introduce a new NF12Config flag. See the MoE capture and escape contract.
embedding() keeps its existing ID/dtype/output contract. On SM80 with K=5120 and 256 <= num_ids < 4096, it automatically selects a two-dimensional prefill grid: eight IDs per CTA through 512 IDs, sixteen through 2048 IDs, and one ID per CTA above 2048. The sixteen-ID path bypasses L1 for output stores so nearby lookups can reuse packed source lines; the single-copy one-ID path uses streaming output stores to preserve packed source lines in cache. No new cache, checkpoint layout, or user flag is introduced; outputs remain exact BF16 rows, including zero rows for invalid IDs.
The recorded SM80 measurements show improvements for repeated code IDs at M=1536 with the prefill grid and at M=3072 with streaming stores, while single-copy NF12 embedding remains slower than dense BF16 on the observed prefill workloads. Treat these as workload-specific results. The embedding dispatch reference and output-store measurements include shape boundaries, exact-output checks, and paired measurements.
gptqmodel_nf12.rowmajor_pack(weight) converts canonical CUDA NF12Weight into an embedding-only layout using the normal native extension. Production AutoNF12Embedding uses this lookup-oriented layout on NVIDIA SM80+ and supported RDNA4 HIP devices. The lightweight package itself remains NVIDIA-only; HIP has separate helpers in gptqmodel.utils.nf12_hip.
import gptqmodel_nf12 as nf12
# packed is a canonical NF12Weight on a supported NVIDIA CUDA device.
row_packed, row_escapes, original_escapes, collisions = nf12.rowmajor_pack(packed)
rows = nf12.rowmajor_embedding(ids, row_packed, row_escapes, packed.exp_base)The native lookup accepts int32/int64 IDs of any shape on the packed tensor's device, makes strided IDs contiguous, and returns exact BF16 rows with shape ids.shape + (K,). Invalid IDs produce positive-zero rows; empty IDs return an empty result without a kernel launch. HIP exposes rowmajor_pack(packed, escapes, exp_base) and rowmajor_embedding(ids, row_packed, row_escapes, exp_base) with the same layout and output contract. Its canonical embedding(ids, packed, escapes, exp_base) is also available and decodes only requested rows.
Operator presence is probed individually rather than assuming an entire preloaded NF12 namespace is complete. Missing row-major pack/lookup operators fall back to canonical packed lookup; if native helpers are unavailable, operation-local Torch decoding uses the packed tensor's device. Canonical fallback also masks negative/out-of-range IDs to positive-zero rows rather than indexing invalid memory. No dense table is retained between calls.
The row-major payload contains uint8 bytes [N, K*3//2] and int16 escapes [E_row, 4], where E_row = 2 * original_escapes + collisions. Each selected-half metadata collision adds an eight-byte escape record. Conversion allocates an independent packed copy; retaining both layouts costs both allocations and does not track later source mutations. AutoNF12Embedding retains only its row-major buffers and exponent, not a dense table or the transient canonical copy. Canonical linears and checkpoint loaders cannot reinterpret row-major bytes; the ordinary NF12 checkpoint contract below remains canonical. See the row-major storage, backend contract, and measurements and HIP canonical lookup dispatch.
The old gptqmodel_nf12.experimental_rowmajor module, NF12RowMajorWeight wrapper, experimental geometry/store-policy arguments, and experimental row-major repeat-four helper have been removed. Use the production helpers above. The separate canonical embedding_repeat4() API is unchanged; it does not consume row-major bytes.
| Symptom | Check |
|---|---|
| Checkpoint is larger than “12 bits” suggests | Count escaped groups and include bias, uncompressed tensors and container overhead |
| FP32 source changed after packing | Check the conversion to BF16 before encoding; NF12 only promises exact reconstruction of the encoded BF16 |
| Native route is unavailable | Verify architecture, BF16 activations/bias, matching devices, and the matching CUDA or HIP development toolchain |
| Lightweight import works but first operation fails | Native libraries load/build lazily; import success does not prove the runtime is ready |
| Outputs differ from dense GEMM | Compare decoded weight bits separately from accumulation/output-rounding error |
| Geometry is rejected | Check each encoded matrix's 16-by-64 divisibility; metadata edits cannot pad packed bytes |
| AUTO_NF12 did not change the final checkpoint format | That policy changes temporary source storage; choose NF12Config for NF12 export |
Run these optional diagnostics from the source repository:
python scripts/validate_nf12.py --device cpu --backend auto --quick --accuracy-only
python scripts/validate_nf12.py --device cuda:0 --backend cuda --quick
# On a supported AMD RDNA4 host (PyTorch's HIP namespace is also cuda):
python scripts/validate_nf12.py --device cuda:0 --backend hip --quick--backend accepts auto (default), cuda, or hip; --device defaults to cuda:0. Other flags are --accuracy-only, --perf-only, and --json-out PATH. Explicit native requests fail if the target/device or extension is unavailable; they do not silently count Torch codec results as native validation. Automatic CPU mode checks the codec and skips native benchmarks. GPU timers synchronize the selected device, native throughput uses that runtime's encoder, and accuracy cases use stable seeds. Performance-only exceptions produce a failing exit status.
Module repr() is safe before buffer allocation and while disk-offloaded on meta; an unavailable exponent is displayed without trying to materialize it. Native availability problems can be inspected through gptqmodel.utils.nf12.last_error() or the HIP counterpart. A successful codec check is not evidence that a native kernel ran.
Byte-exact BF16 encode/decode is separate from bitwise linear-output parity across different reduction schedules. The HIP WMMA source now dispatches the BF16-twin reference through the same (tiles, splits, m_blocks) plan as NF12, including split-K scratch and reduction. Comparisons no longer silently contrast dynamic NF12 launches with a fixed eight-tile/single-split twin. This is a native reference-path fix, not a checkpoint-format change or new quantization flag.
Family-wide HIP builds guard the packed BF16 dot instruction on gfx1250, which lacks dot12-insts; that target uses an FP32 FMA fallback with FP64 recomputation when non-finite intermediates would break finite-input cancellation. This does not add the missing WMMA route to gfx1250 or qualify arbitrary AMD architectures.
- Lightweight inference API and exports, module wrappers, and native loader/cache
- Config and public enum values
- Weight encoding and native encoder selection
- Inference layer and checkpoint buffers
- Torch codec, CUDA loader, HIP loader
- AlpinDale layout specification and attribution
- Validation script
Next: measure task quality with Evaluation, then inspect Loading and Inference for your chosen runtime.
Documentation home · Quickstart · Troubleshooting · Repository · PyPI
Public source d4712ee13 declares package version 7.12.4. Individual pages identify their source revisions. Choose a method · Dtypes and formats. Maintained by ModelCloud.ai.
🧭 1 · Choose and understand
🚀 2 · Use it
- Installation and Quickstart
- Loading and Inference
- Evaluation
- Serving
- Troubleshooting and FAQ
- Memory, Offload, Multi-GPU
- MLX / Apple Silicon
- Tenstorrent / Block Floats
- Hugging Face Integration
- Multimodal and MoE Models
🎛️ Methods and representations
Each guide: choose → examples → math / advanced
- GPTQ / GPTAQ / RTN
- AWQ
- QQQ
- MACaQ / MXFP4
- EXL3
- CPTC
- VPTQ
- Paro
- FP8 and NVFP4
- NF12: lossless BF16 storage
- GGUF families
- BitsAndBytes: NF4 / FP4 / INT8
🧰 Refinement and execution
🧠 3 · Theory and advanced reference