Skip to content
Qubitium edited this page Nov 18, 2026 · 34 revisions

🏠 GPT-QModel Documentation

On this page

Make a model smaller, check what changed, and run it on a compatible runtime. GPT-QModel brings quantization, calibration, evaluation, conversion, and inference into one workflow.

This wiki is for builders compressing their first model, operators loading existing checkpoints, and contributors extending model or kernel support. Start with your task; the implementation details are here when you need them.

Find your starting point

I want to… Start here Then
Understand quantization without getting lost in jargon What is Quantization? Choose a method and dtype
Pick a recipe for my hardware and data Choosing a Method and Dtype Dtypes and Formats · Backends
Quantize my first model Installation and Quickstart Lifecycle · Evaluation
Run a checkpoint I already have Loading and Inference Packing compatibility · Serving
Fit a larger model or resume work Memory, Offload, and Multi-GPU Lifecycle and recovery
Fix unexpected output or a loading error Troubleshooting and FAQ Evaluation · Supported Models
Add a model, method, or backend Developer Guide Module plans · Processors

🧭 Read at your depth

  1. Choose and understand: plain-English value, audience, decision tables, prerequisites, and honest limits.
  2. Use it: practical load → quantize/convert → save → reload examples, with hardware and calibration requirements.
  3. Go deeper: intuition, defined math/number representations, advanced configuration, internals, and source/research citations.

Method pages follow this progression while retaining their detailed references. Short calibration strings are API demos, never accuracy recipes. For real decisions, use representative calibration and disjoint evaluation data.

Keep the layers separate: an algorithm chooses weights; a number representation defines their values; a checkpoint layout/container stores them; a backend executes them. GSQ refinement, EoRA adapters, and fusion are additional choices—not replacement dtypes.

Workflow at a glance

From source model to deployment. Choose a preparation path, then save, reload, evaluate, and deploy.

Reading the diagram: The first two lanes are alternatives; both continue into the final lane. Existing quantized models start at Reload. Resume checkpoints are separate from inference exports.

Text version of the workflow
  • Prepare a quantized model — calibration-based methods: Load source + config → Prepare calibration → Quantize.
  • Prepare a quantized model — calibration-free methods: Load source + config → Read source weights → Convert and pack.
  • After either preparation path: Save → Reload → Evaluate → Generate or serve.

Open full-size SVG

Getting started

Quantization methods

Your direction Full guides: practical examples → intuition/math → advanced controls
Scalar integer quantization GPTQ / GPTAQ / RTN · AWQ
Low-precision activations QQQ: INT4 + INT8 · MACaQ: MXFP4 · FP8 and NVFP4 streams
Alternative low-bit representations EXL3 · VPTQ · Paro
Calibration-free weight packing FP8 · BitsAndBytes NF4 / FP4 / INT8 · GGUF block families · RTN
Exact BF16 weight compression NF12 — variable escape overhead; not guaranteed fixed-size lossy 12-bit quantization
Shared-exponent block floats / card execution Tenstorrent — distinguish model checkpoints, native card routes, and standalone CPU bf8_b / bf4_b / bf2_b kernels

GGUF is not one dtype. FP4, NF4, MXFP4 and NVFP4 are not interchangeable. See Dtypes and Formats before comparing bit-width labels.

Complementary configs

Using a quantized model

Platform support

NVIDIA CUDA and AMD ROCm are distinct support families. A Triton implementation or Torch cuda device name is not proof of every vendor/architecture's support. Check the selected backend's actual declarations and architecture/dtype/shape gates. A support claim is not benchmark or numerical qualification.

Internals

For contributors and readers debugging the pipeline; you do not need these pages to run your first model.

Documentation scope

The current main branch reports GPT-QModel 7.12.0, at source revision e16a259dd. This is a source-version marker, not a claim that a release or package is published. Individual pages and historical measurements retain their stated source scope; check the source matching your installed version for exact behavior.

For release history, see Releases, the 7.11.1 highlights in the pinned README, and release notes.

Workflow SVGs have text alternatives. Their lanes describe alternatives, independent concerns, or nested loops; they complement rather than replace API validation rules.

Clone this wiki locally