Tenstorrent

Tenstorrent and Block-Float Checkpoints

Documentation home · Quickstart · Lifecycle

On this page

• Choose a checkpoint and execution path
• Install the Tenstorrent runtime
• Load packed GPTQ on the cards
• Create a native block-float checkpoint
• Storage and portable loading
• CUDA, ROCm, and standalone CPU execution
• Source files
• See also

GPT-QModel supports packed GPTQ execution on Tenstorrent through Torch/XLA or native TTNN, plus a separate native block-float checkpoint family. This page describes source revision 7e63a4184. The repository's hardware qualification uses Blackhole p300c cards; a declared device family is not qualification of every Tenstorrent architecture or model.

Choose a checkpoint and execution path

Backend	Input checkpoint	Execution and storage
GPTQ_TT_XLA (gptq_tt_xla)	Symmetric 4-bit GPTQ / GPTQ_V2	XLALinear: Torch/XLA → StableHLO → Tenstorrent PJRT → TT-MLIR. Persistent packed byte planes; transient BF16 decode and matmul. Highest-priority GPTQ candidate on DEVICE.XLA.
GPTQ_TTNN (gptq_ttnn)	Symmetric 4-bit GPTQ / GPTQ_V2	TTNNLinear: native ttnn.MeshDevice, packed integer decode and BF16 matmul. Request explicitly to select the native runtime.
GPTQ_TT_BF8 (gptq_tt_bf8)	Symmetric 4-bit GPTQ / GPTQ_V2	Experimental TTNNBlockFloat8Linear: converts decoded GPTQ weights to TT bf8_b runtime storage. Adds a block-float rounding step and roughly doubles weight-code storage relative to INT4. This is a runtime conversion kernel, not the native block-float checkpoint loader.
TT_BLOCKFLOAT (tt_blockfloat)	quant_method="tt-blockfloat", format tt_blockfloat	TTBlockfloatLinear and packed embedding/tied-head modules. Portable Torch execution, with automatic eligible HIP/CUDA emulation. This class declares CPU, CUDA, ROCm, XPU, NPU, and MPS; it does not declare XLA.

DEVICE.XLA is the Tenstorrent selector family for both GPTQ runtimes. Native TTNN keeps the Hugging Face wrapper and its activations on CPU and performs its own transfers; Torch/XLA uses actual xla:N tensors. The ordinary GPTQ_TORCH implementation explicitly rejects XLA because Tenstorrent's PJRT integer lowering cannot safely decode its packed 32-bit words. Other portable Torch families also enumerate their supported devices without admitting XLA implicitly.

Install the Tenstorrent runtime

Use an isolated CPython 3.12 environment and the repository's dedicated manifest and installer:

# Run from the GPT-QModel source checkout, with the intended environment active.
PYTHON_BIN=python3.12 requirements/install-tenstorrent-py312.sh "$PWD"

The installer uses Tenstorrent's package index and the CPU Torch index, installs the pinned manifest, and installs the checkout with --no-deps. The manifest pins CPU Torch 2.11.0+cpu, Torch/XLA 2.9.0+gited8a445, pjrt-plugin-tt==1.5.0.dev20260831000501, Transformers 5.22.2, and NumPy 1.26.4. This special stack predates the normal Torch requirements in Installation and Quickstart; do not replace the base package's pins globally or re-resolve this environment against them.

Native TTNN additionally needs an importable vendor ttnn runtime and free devices. A PJRT installation alone does not establish native TTNN availability. The installer verifies PJRT in a separate subprocess. The repository records sfpi, tenstorrent-dkms, and tenstorrent-tools system versions separately; the Python installer does not install those system packages. See the pinned runtime manifest and installation/compatibility notes, including the known maintenance-tool dependency conflicts.

Load packed GPTQ on the cards

Both XLALinear and TTNNLinear require the following contract:

Property	Supported values
Method / format	GPTQ; GPTQ or GPTQ_V2, non-planar
Bits / packing	4; torch.int32 words
Symmetry / activation order	sym=True; desc_act=True or False
Group size	-1, 16, 32, 64, 128, 256, 512, 1024
Activations / platform	BF16; Linux
Shape	Input and output widths divisible by 8; no automatic padding
Training / adapters	Unsupported

In a fresh process, select native TTNN explicitly:

import torch
from gptqmodel import BACKEND, GPTQModel

model = GPTQModel.load(
    "/models/symmetric-gptq-4bit",
    device="xla",
    backend=BACKEND.GPTQ_TTNN,
    dtype=torch.bfloat16,
)
# Native TTNN accepts CPU BF16 activations and owns its mesh itself.
# Call after the final use of this native model, before another model load.
model.release_device()

For Torch/XLA, use BACKEND.GPTQ_TT_XLA and an xla:N target in a separate process. AUTO on XLA normally selects XLALinear (format priority 20); the two native GPTQ conversion classes have lower priority 15. AUTO is priority/validation based, not a speed benchmark. Synchronize with torch_xla.sync(wait=True) before host readback of XLA results. generate() uses torch.no_grad() on XLA to preserve the view/version-counter semantics needed by that runtime.

TT-Metal permits one runtime owner per chip. Querying PJRT's addressable devices initializes it and claims the chips; switching to native TTNN afterward fails with TTNNUnavailable. Use separate processes when changing runtimes. Native GPTQ_TTNN device resolution avoids PJRT probes. The experimental GPTQ_TT_BF8 kernel should not be assumed to share that complete model-loader path.

The native mesh helper defaults to one device. Multi-card TTNN shards output features and gathers them over the fabric; out_features must divide evenly across the selected device count. Closing a native mesh through release_device() requests idle clocks. It returns False when there is no native mesh; it is not a PJRT shutdown API. Runtime thermal pacing defaults to a 5-second cooldown, an 80°C GDDR soft limit, and a 900-second maximum thermal wait. Unavailable telemetry skips the temperature guard. See the runtime and hardware notes for mesh ownership and pacing details.

Create a native block-float checkpoint

Tenstorrent bf4_b and bf8_b are vendor-specific shared-exponent formats: one exponent covers 16 values. They are distinct from OCP FP8/MX and NVIDIA NVFP4. The public checkpoint config permits 4- or 8-bit codes; the experimental bf2_b grid and standalone CPU library do not establish 2-bit model-checkpoint or TTNN support.

QuantizeConfig(method="tt-blockfloat") resolves to TTBlockfloatConfig, which describes storage for loading/export. Ordinary model.quantize() rejects this method. Produce the checkpoint using the dedicated grid-aware GPTQ script, starting with the dense source model and calibration data:

python scripts/tt_quantize_blockfloat.py /models/dense-source \
  --out /models/tt-blockfloat \
  --bits 4 --embed-bits 8 \
  --calibration-dataset /data/calibration.parquet

Script option	Default / requirement
Positional model	Required model ID or local path
--out	Required output directory
--bits	4; choices 4, 8 for decoder linears
--embed-bits	8; choices 4, 8 for the embedding/output-head payload
--bits-map	None; comma-separated per-layer patterns such as q_proj=8,k_proj=8, with unmatched layers keeping --bits
--nsamples	8
--dtype	auto; choices auto, float32, bfloat16, float16
--calibration / --calibration-dataset	Exactly one required: calibration text, or a supported local dataset file/Hub dataset ID
--calibration-rows	0 (all records); restricts dataset records when positive

The default source dtype follows checkpoint metadata; a BF16 source stays BF16 while Hessians accumulate in FP32. Loading a converted checkpoint likewise preserves its declared precision unless dtype explicitly requests another. Quantizing the source directly onto this grid is different from converting an existing INT4 GPTQ checkpoint to bf4_b; the latter introduces additional quantization error.

Storage and portable loading

The saved model uses safetensors shards and model.safetensors.index.json, with config.json recording quantization_config.quant_method="tt-blockfloat". Each quantized module owns codes (packed uint8), exponents (int8), and code_shape (int64). Code bytes round up for partial bytes; exponent rows have shape [out_features, in_features/16] for the default block size 16. Mixed widths are recorded in bits_per_layer. Tied embeddings and output heads share packed storage. Non-quantized tensors remain in the checkpoint's safetensors payloads, and dense linear biases retain their source dtype rather than being narrowed to FP16.

The generated checkpoint modeling file subclasses the source model's Transformers causal-LM architecture, rather than a fixed Llama class. A wrapper/subclass can resolve through its declared model_type; export raises if no known Transformers causal-LM base can be identified. This preserves non-Llama architecture reconstruction on reload.

import torch
from gptqmodel import GPTQModel

model = GPTQModel.load("/models/tt-blockfloat")
# The special loader builds on CPU. Move the underlying Torch model explicitly
# when selecting CUDA or ROCm emulation (both use Torch's cuda namespace).
if torch.cuda.is_available():
    model.model.to("cuda:0")

The block-float loader detects the method from config.json and rebuilds modules using the safetensors index. It takes this dedicated route before generic qlinear dispatch; setting backend or a device map does not turn it into a native TTNN/XLA load. Portable modules dequantize only for the current operation and retain no expanded weight cache.

CUDA, ROCm, and standalone CPU execution

Path	Selection and limits
HIP block-float kernels	On ROCm, TTBlockfloatLinear, embedding, and tied head use eligible packed bf4_b/bf8_b kernels. FP16/BF16/FP32 operands are supported; block size is 16. GPU extensions build from bundled sources. Unsupported calls use Torch.
CUDA block-float kernels	Automatically eligible on NVIDIA SM80+ with FP16/BF16/FP32 inputs, 4/8-bit storage, and block size 16. Fused packed GEMM and embedding gather avoid a persistent dense weight. An unavailable/failed extension build logs and uses the Torch reference path.
Portable Torch	CPU and other declared device families use current-call dequantization and Torch operators. The main package does not automatically dispatch CPU layers to ttbfp.
Standalone gptqmodel_ext/ttbfp	Separate C++17/OpenMP CPU library for bf8_b, bf4_b, bf2_b: scalar, AVX2+FMA, and AVX-512 runtime dispatch; packed GEMM, quantize/dequantize, and TT tile helpers. Its Python interface uses NumPy and optional Torch helpers. It is neither built by setup.py nor imported by gptqmodel.

CUDA eligibility is checked on the actual activation or token-ID tensor's device, not whichever GPU is current. The extension build result is shared, but the SM80+ capability check runs for each target: probing an older GPU does not disable native execution on a supported GPU, and a successful build does not enable native execution on an unsupported GPU. Ineligible devices keep the operation-local Torch fallback.

Both GPU routes require block size 16: wider-block linears and embeddings remain on the operation-local Torch path even when their device, storage bits, and dtype otherwise qualify. This also covers tied embedding/output-head storage; passing a wider block to a 16-value exponent kernel would decode it incorrectly.

Build the standalone library from the source checkout:

make -C gptqmodel_ext/ttbfp
make -C gptqmodel_ext/ttbfp test

The standalone CPU library's upstream-LLK-compatible quantizer and the Python grid-aware model converter are separate APIs. Consult their definitions before assuming identical encoding or rounding behavior. bf2_b has no TTNN dtype and no HIP/CUDA model-kernel route in this revision.

Source files

• Backend enum, config definitions, loader.
• XLA kernel, TTNN kernel, GPTQ-to-bf8 runtime kernel.
• Block-float modules, checkpoint loader, quantization script.
• HIP emulation, CUDA emulation, standalone CPU library.

See also

Inference Backends · Kernel Backends Reference · Packing Formats · QuantizeConfig Reference