Skip to content

NF12

Jump to bottom
Qubitium edited this page Nov 9, 2026 · 3 revisions

NF12: Lossless BF16 Weight Compression

Documentation home · Quickstart · Lifecycle

On this page

Overview

NF12 (NyoomFloat12) is Together AI's 12-bit BF16 weight encoding. GPT-QModel implements the AlpinDale modified tiled layout from dphnAI/nf12, not the paper's byte layout. Encoding and decoding BF16 weights is bit-exact; converting FP16/FP32 source weights to BF16 first can be lossy. Lossless weight storage does not promise bit-identical inference across kernels with different accumulation orders and output rounding.

This page describes source revision 2b5e28eb6. NF12 uses the calibration-free weight-only lifecycle, not GPTQ Hessian optimization.

Quantize, save, and reload

from gptqmodel import GPTQModel
from gptqmodel.quantization import NF12ADConfig
from gptqmodel.utils.backend import BACKEND

config = NF12ADConfig()  # bits=12; no calibration dataset
model = GPTQModel.load("/path/to/bf16-model", quantize_config=config)
model.quantize()
model.save("/path/to/nf12-model")

reloaded = GPTQModel.load("/path/to/nf12-model", backend=BACKEND.NF12_AD)

The normal model definition and module-plan rules still decide which modules are quantized. Every encoded matrix must have out_features % 16 == 0 and in_features % 64 == 0; this backend does not automatically pad incompatible dimensions. NF12ADQuantizer accepts nn.Linear, Transformers Conv1D, and ConvNd weights after conversion to an output-by-input matrix.

The factory alternative is QuantizeConfig(method="nf12", bits=12). Use the public method/format string "nf12", not "nf12_ad". Python symbols retain their AD suffix: NF12ADConfig, METHOD.NF12_AD, FORMAT.NF12_AD, and BACKEND.NF12_AD. Only the backend enum serializes as "nf12_ad"; method and format both serialize as "nf12". The method-aware legacy BACKEND.TORCH selector also resolves to BACKEND.NF12_AD.

Configuration

Option Default / contract
bits 12; the only supported NF12 width
method / format METHOD.NF12_AD / FORMAT.NF12_AD, both "nf12"
group_size Forced to -1; not a grouping knob
desc_act / sym Forced to False / True; not tunable quantization policies
exp_base None: choose each matrix's exponent base to minimize escaped groups. A global integer in [0, 120] forces the same base policy for all modules.
dynamic Module exclusion and compatible overrides follow the shared config machinery; a per-layer exp_base is rejected. Dynamic bits must remain 12 and group size must remain -1/None.

NF12ADConfig inherits PreProcessorConfig, but direct NF12 packing explicitly rejects a non-None smooth argument. Do not combine smoothing with a lossless-storage claim. A forced exp_base uses the Torch codec because the native encoders select their own base.

Checkpoint layout and memory

For a module with weight matrix [N, K], the saved state contains:

Buffer Dtype Shape
nf12_packed torch.uint8 [N, K * 3 // 2]
nf12_escapes torch.int16 [E, 8]
nf12_exp_base torch.uint8 [1]
bias, when present Source bias dtype at pack time [N]

The core packed matrix takes 12 bits per weight, but the escape table and exponent add overhead: matrix payload bytes are N*K*3//2 + E*16 + 1, excluding bias, config, and container headers. Do not assume every model is exactly 25% smaller than BF16. Escapes preserve out-of-window BF16 groups, including special values and their payload bits.

Use nf12_packed, not the earlier development key nf12_ad_packed. Packed bytes are shared by the CUDA, HIP, and Torch codec paths. The Hugging Face/Accelerate checkpoint loader sizes variable-length nf12_escapes buffers from checkpoint metadata before assigning tensors; keep these buffers in exports rather than replacing them with empty placeholders.

Inference retains only packed weights, escapes, exponent metadata, and bias. A fallback may reconstruct a matrix for the current forward, but does not cache that dense weight between calls. Exponent-buffer replacement and ordinary versioned mutation invalidate the scalar exponent cache. After manually changing an inference-mode exponent tensor in place, call the layer's clear_weight_cache().

Inference and native dependencies

NF12ADLinear declares portable device support and FP16/BF16/FP32 activation dtypes; optional native execution requires BF16 activations and matching activation/packed-weight devices. Non-native calls use decode plus FP32-reduced matmul and return the activation dtype. Training is not supported by this backend.

Target Execution path
NVIDIA CUDA Vendored fused decode/GEMM kernels when loadable; JIT build requires SM80+, CUDA toolkit 13.0+, CMake 3.26+, Ninja, and a C++ compiler. The build checks all visible CUDA architectures.
AMD RDNA4 HIP encoder/decoder and fused small-M / WMMA paths for gfx1200, gfx1201, gfx1250; larger or ineligible calls use sliced decode plus rocBLAS. Requires a working ROCm development toolchain. This is not a declaration of native NF12 support for every AMD family.
Other devices, non-BF16 activations, or unavailable native extension Torch codec plus matmul where the target's Torch operations are supported; no native speed guarantee.

The HIP linear wrapper normalizes BF16 activations to a contiguous [M, K] matrix, where M is the product of all leading dimensions, and restores x.shape[:-1] + (N,) on every return. Sliced/padded-row and higher-rank activations can therefore use the same routes without reading storage gaps or flattening the returned shape. Eligible calls use a token-count-specialized fused kernel for M <= 15; eligible calls with 16 <= M <= 256 use WMMA on gfx1200/gfx1201, and other calls use row-sliced decode/GEMM. gfx1250 lacks the required WMMA instruction and does not use that matrix-core route.

Split-K scratch comes from PyTorch's allocator and follows the packed tensor's device/current stream, including non-current GPU ordinals. Split partials remain unbiased FP32 values; the reduction adds bias once before BF16 output conversion. Launch tuning is shape-specific; see the HIP implementation and measurements, not a universal speedup claim.

The recorded validated environment is ROCm SDK 10.1.0 / HIP 7.16.26385, PyTorch 2.16.0.dev20261025+rocm10.1, free-threaded Python 3.14, and Radeon AI PRO R9700 (gfx1201). This is a tested combination, not a newly imposed minimum. The HIP component version (torch.version.hip or hipconfig --version) is distinct from the ROCm SDK release.

RDNA4 buffering and split-K choices are derived from the launch shape. At M=128, the WMMA planner can split M for long-K or narrow-N cases to improve occupancy, rather than applying the same choice to every shape. The refreshed upstream format primitives and HIP kernels preserve the existing packed-byte layout; no checkpoint migration or new NF12ADConfig flag is required. The JIT source set also includes MoE helper kernels and a BF16-twin packer, but the registered nf12_ad_hip Torch API still exposes only encode, decode, linear, selftest, and arch_info. Compiled source availability is not a public fused-MoE execution API or automatic MoE dispatch.

For native C-ABI callers, dense encode/decode and fused linear reject invalid 16-by-64 geometry rather than truncating it; encoders also check workspace capacity before writing. MoE helpers check slice bounds, BF16 pointer/row-stride alignment, metadata alignment, and signed-index byte spans; malformed arguments return hipErrorInvalidValue. Routing endpoints use a clamped running maximum within the assignment count, so negative, decreasing, or oversized offsets cannot select guard rows. Zero-assignment alignment publishes zero offsets without requiring nonempty scratch.

WMMA launch bounds use X up to 2147483647 and Y up to 65535, not the earlier blanket 32767-block cap. Routed WMMA distributes excess assignment tiles across Y/Z; grids beyond the supported limits return hipErrorNotSupported before launch. The native wmma_available() helper probes ordinals outside its bounded 64-device cache instead of declaring them unsupported solely because they are uncached. Encoder scheduling uses four super-atoms per analysis warp and two-super-atom count/write units while preserving flat escape order and packed bytes. These are native implementation contracts, not new user configuration flags; see the HIP device/workspace reference.

The CUDA loader first tries an installed nf12 package and requires callable torch.ops.nf12.encode, decode, and linear; a namespace alone is insufficient. Missing/stale operators trigger the vendored JIT build. No separate nf12 Python package is required. The default CUDA build root is $XDG_CACHE_HOME/gptqmodel/nf12_ad (or ~/.cache/gptqmodel/nf12_ad); GPTQMODEL_NF12_AD_BUILD_ROOT overrides it. HIP uses GPTQMODEL_NF12_AD_HIP_BUILD_ROOT and stages sources in its writable cache rather than modifying an installed package. CUDA_HOME/CUDACXX and ROCM_PATH/ROCM_HOME help locate the respective toolchains.

Validation and troubleshooting

Run from the source repository:

python scripts/validate_nf12_ad.py --device cpu --backend auto --quick --accuracy-only
python scripts/validate_nf12_ad.py --device cuda:0 --backend cuda --quick
# On a supported AMD RDNA4 host (PyTorch's HIP namespace is also cuda):
python scripts/validate_nf12_ad.py --device cuda:0 --backend hip --quick

--backend accepts auto (default), cuda, or hip; --device defaults to cuda:0. Other flags are --accuracy-only, --perf-only, and --json-out PATH. Explicit native requests fail if the target/device or extension is unavailable; they do not silently count Torch codec results as native validation. Automatic CPU mode checks the codec and skips native benchmarks. GPU timers synchronize the selected device, native throughput uses that runtime's encoder, and accuracy cases use stable seeds. Performance-only exceptions produce a failing exit status.

Module repr() is safe before buffer allocation and while disk-offloaded on meta; an unavailable exponent is displayed without trying to materialize it. Native availability problems can be inspected through gptqmodel.utils.nf12_ad.last_error() or the HIP counterpart. A successful codec check is not evidence that a native kernel ran.

Byte-exact BF16 encode/decode is separate from bitwise linear-output parity across different reduction schedules. In this revision's HIP WMMA source, the BF16-twin branch still launches a fixed eight-tile, single-K-split kernel, despite sharing the planner query with NF12. Do not assume the twin follows NF12's dynamically selected tile/split launch or use that assumption as proof of bitwise linear equivalence.

Source files

See also

Documentation home · Quickstart · Troubleshooting · Repository · PyPI

Source baseline: GPT-QModel 7.9.0 / 40948a6. Maintained by ModelCloud.ai.

Home

Getting started

Quantization methods

Complementary configs

Using a model

Platforms

Internals

Clone this wiki locally