Quantization Theory

Quantization Theory

Reference for the quantization concepts BitForge explores and implements.

Why quantize?

Foundation models are trained in FP32 or BF16, but running at full precision locally is often impractical. Quantization maps weights (and sometimes activations) to narrower integer types—INT8, INT4, INT2—to reduce memory footprint and increase inference throughput.

The trade-off is accuracy loss. How gracefully a model degrades depends on the method, architecture, calibration data, and target bit width. BitForge exists to measure those trade-offs openly.

Taxonomy

Family	When it runs	Typical use
PTQ (Post-Training Quantization)	After training, no gradient updates	Fast compression of existing checkpoints
QAT (Quantization-Aware Training)	During or after fine-tuning with simulated quant	Best quality at low bit widths, higher cost
Hybrid	PTQ + lightweight fine-tune (e.g. QLoRA on 4-bit base)	Balance of speed and recoverable accuracy

BitForge focuses on PTQ and hybrid workflows common in local inference.

Methods in BitForge

GPTQ (Generalized Post-Training Quantization)

Optimizes weight rounding layer-by-layer using a second-order Taylor approximation of quantization error. A Hessian (or its inverse) captures how errors in one weight affect the layer output, allowing compensated rounding:

error compensation uses H⁻¹ to adjust subsequent weights

Implemented in bitforge.core.gptq.GPTQQuantizer. Best suited for weight-only INT4/INT3 on linear layers when you have a GPU and calibration set.

AWQ (Activation-aware Weight Quantization)

Identifies salient weight channels by observing activation magnitudes on calibration data, scales those channels before quantization, then applies group-wise weight quantization:

W' = W · s^α,   X' = X / s^α   →   quantize(W')

Implemented in bitforge.core.awq.AWQQuantizer. Often competitive with GPTQ on 4-bit LLMs with faster calibration.

bitsandbytes (NF4 / FP4)

Runtime quantization loaded via Hugging Face + bitsandbytes. Common for QLoRA fine-tuning where the base stays 4-bit in memory. Less ideal as a portable artifact format but excellent for training workflows.

GGUF / GGML

File format used by llama.cpp, Ollama, and Super-Ollama. Supports many preset quant types (Q4_K_M, Q8_0, Q2_K, etc.) with block-wise scales and optional importance matrices.

Implemented via bitforge.core.gguf.GGUFConverter for Hugging Face → GGUF export.

Granularity schemes

Scheme	Scope	Trade-off
Per-tensor	One scale for entire weight matrix	Smallest metadata, worst accuracy
Per-channel	One scale per output channel	Good default for linear layers
Per-group	Scale every group_size elements (e.g. 128)	Standard for INT4 LLMs
Block-wise (GGUF)	Fixed-size blocks with independent scales	Optimized for CPU/GPU kernels in llama.cpp

BitForge's QuantizationConfig.group_size defaults to 128, matching common LLM practice.

Weight vs. activation quantization

Type	What is quantized	Notes
Weight-only	Linear layer weights	Most local LLM formats (GPTQ, AWQ, GGUF)
Weight + activation	Weights and intermediate tensors	Needed for true INT8 inference; harder due to dynamic ranges

Activation spikes (outliers) in a small number of channels can dominate dynamic range and destroy INT8 accuracy. BitForge's detect_outliers metric and Experiment 04 explore suppression strategies:

• SmoothQuant — joint scaling of weights and activations
• LLM.int8() — mixed-precision decomposition for outlier dimensions

Calibration datasets

Quantizers observe real inputs to estimate scales (AWQ) or Hessians (GPTQ). BitForge ships loaders for:

Dataset	Characteristics
Wikitext-2	Small, clean prose; fast default
C4	Large web crawl subset; more generic distribution
Custom	Domain-specific (code, medical, legal) for OOD analysis

Open question: calibration distribution may matter more for smaller models or narrower bit widths. BitForge experiments are designed to test this systematically.

Bit-width guidance (rules of thumb)

Bits	Typical quality	Typical use
FP16 / BF16	Baseline	Reference, fine-tuning
INT8	Near-lossless for many models	Server inference, AVX/Tensor Core paths
INT4	Small measurable perplexity increase	Default for local 7B–70B inference
INT2 / ternary	Often significant degradation	Experimental; niche hardware

Exact breakpoints are model-dependent. Use BitForge benchmarks rather than assuming a universal threshold.

Key formulas

Asymmetric quantization:

scale = (max - min) / (2^b - 1)
zero_point = round(-min / scale)
W_q = clamp(round(W / scale) + zero_point, 0, 2^b - 1)
W_dequant = (W_q - zero_point) × scale

Perplexity:

PPL = exp(cross_entropy_loss)

See the Interactive Lab for visualizations and the simulator UI.