Repository navigation
Projection Fusion
🏠 Home · 🧭 Choose a method · 🚀 Quickstart · 🧰 Help
On this page
Projection fusion is an opt-in inference optimization: run compatible Q/K/V or gate/up projections together while keeping the quantized checkpoint and the parent's attention/MLP math intact.
For: users who already have a working quantized model and want to test an execution optimization—not choose a new quantization method or dtype.
GPT-QModel can combine compatible Q/K/V or gate/up projections into one wider quantized linear operation for inference. Fusion is disabled by default and requires an explicitly qualified model definition. It does not change the quantization method, checkpoint keys, or selected backend.
This page describes source revision
162e34541,
with projection fusion originally introduced in PR #3567.
Use an installation containing the documented fixes; the release number alone
does not establish that these APIs and behaviors are available.
| Choose fusion when… | Consider another path when… |
|---|---|
| A qualified definition declares compatible sibling projections for your already-selected native backend. | You only need basic model support: Supported Models is not the fusion qualification matrix. |
| You can measure an inference workload in eval mode with stable packed weights. | You need training, compiled fused execution, projection hooks, or adapters: those paths are excluded or fall back. |
| You want to reduce separate linear launches and can verify actual fused calls. | You need a guaranteed speedup: eligibility is not a latency result. |
| You want the same checkpoint keys and quantized storage. | You need a different compression method or runtime: start with Choosing a Method and Dtype. |
Prerequisites for the first example: a locally loadable, qualified quantized model; compatible dtype/device and native layout; no adapter, online projection rotation, projection hooks, or parent offload hook. The code below assumes a single-device causal-LM checkpoint with its tokenizer. Multi-device deployments must place inputs for their actual entry module.
Read in three passes: enable and verify → inspect decisions and overrides → qualification, storage, and extension reference.
Fusion leaves AUTO's native kernel choice intact. A rejected backend is not silently replaced by Torch just to make fusion appear supported. Keep NVIDIA CUDA and AMD ROCm architecture requirements separate.
Start from a quantized model returned by the GPT-QModel wrapper. This is an API/numerical smoke check, not a latency benchmark or an accuracy recipe:
import torch
from gptqmodel import BACKEND, GPTQModel
model = GPTQModel.load("path/to/quantized-llama", backend=BACKEND.AUTO)
model.eval()
inputs = model.tokenizer(
"Explain why the sky appears blue.", return_tensors="pt"
).to(model.model.device)
# Baseline first, then enable and run the same workload.
with torch.inference_mode():
baseline = model.model(**inputs).logits.float()
plan = model.fuse()
plan.print()
with torch.inference_mode():
fused = model.model(**inputs).logits.float()
print("maximum absolute logit drift:", (fused - baseline).abs().max().item())
print("actual fused calls:", plan.to_dict()["fused_calls"])
decisions = plan.to_dict()
model.fuse(False)
Success means: the plan contains eligible groups and fused_calls grows after the forward. Inspect the measured logit drift using tolerances appropriate to your dtype/backend; do not assume bitwise identity. Evaluate held-out quality separately. A plan with zero eligible groups or zero calls is not evidence of an optimization.
model.fuse() applies fusion and returns a FusionPlanner, not the model.
Each enabled call rebuilds the plan; retain its return value for inspection
without mutation. model.fuse(False) restores separate execution and
independent contiguous packed operands, releases the executors, and returns
None. A retained old plan remains inspectable after disabling.
| Entry point | Default | Behavior |
|---|---|---|
GPTQModel.load(..., fuse=False) |
False |
Opt in with True; a quantized wrapper applies fusion during initialization. A source wrapper records the request for quantization finalization. |
model.quantize(..., fuse=None) |
None |
Inherit the wrapper's current fusion setting. True enables fusion after successful quantization; False disables it. The return value remains quantization diagnostics, not the fusion plan. |
model.fuse(enabled=True, *, overrides=None) |
True, no overrides |
Apply/rebuild a plan, or disable with False. Overrides map family names or exact parent paths to booleans. |
The model-definition from_pretrained() and from_quantized() loaders also
accept fuse=False; the public factory methods forward it through their
keyword arguments. This is a runtime/quantization-call option, not a field
of QuantizeConfig. Enabled values must be actual booleans, not 0, 1,
or strings; only quantize() additionally accepts None.
For a source-model quantization run:
from gptqmodel import CalibrationConfig, GPTQModel, QuantizeConfig
model = GPTQModel.load(
"/path/to/source-llama",
quantize_config=QuantizeConfig(bits=4, group_size=128, sym=True, desc_act=False),
)
# calibration_dataset is a representative dataset prepared by the caller.
# A few short strings would be an API demo only, not an accuracy recipe.
logs = model.quantize(CalibrationConfig(dataset=calibration_dataset), fuse=True)
plan = model.fuse()
plan.print()
Calling fuse() directly on an unquantized wrapper raises RuntimeError.
A definition without a matching, complete FusableConfig raises on opt-in
rather than silently enabling nothing; quantization validates the requested
definition before calibration. Disabling remains valid for unqualified models.
A qualified model can still return a plan with no eligible groups.
plan.print(file=...) writes the terminal summary; plan.render() returns
the same text. plan.to_dict() includes:
| Key | Meaning |
|---|---|
fusable |
Serialized definition profiles, model types, policy, origins, and composition diagnostics. |
groups |
Decisions with family, full parent path, layer-relative paths, enabled, reason, profile_origins, and winning policy. |
fused_calls |
Count of fused executions performed by this plan; inspect after inference to distinguish eligibility from actual use. |
An enabled decision means that a group passed planning checks, not that every subsequent forward uses it. Runtime input, training, hooks, and compiler state can cause separate execution. Excluded MoE parents are not enumerated as fusion candidates in the first place.
Built-in role profiles are qkv (q, k, v) and gate_up (gate, up).
Paths resolve from the compiled module plan, not a name
suffix or an HF class-path allowlist. Every role must identify exactly one
distinct quantizable sibling, with compatible live operands and parent code.
# Each call replaces the previous plan and its per-call overrides.
plan = model.fuse(overrides={"qkv": False})
plan = model.fuse(overrides={"model.layers.0.mlp": False})
An exact full parent path wins over a layer-relative parent, which wins over the family within the same policy scope. Runtime overrides are checked before definition policy; definition layer-scoped policy precedes its family default. These are exact selectors, not regular expressions or dynamic quantization rules. Use paths shown by the plan for the actual model.
Effective quantization options are compared after per-module dynamic
overrides. Different bits, group size, symmetry, activation ordering, format,
or other serialized options can reject the whole sibling group. A disabled
dynamic target also rejects it. Equal global settings alone do not prove
compatibility.
The native parent forward must make each declared projection call exactly once on the same input, without extra call arguments. Conditional/helper calls, rewritten inputs, extra projection-weight references, decorated or instance-overridden forwards, and unavailable source fail closed. The parent retains its native RoPE, cache updates, attention dispatch, activation, multiplication, and output projection. A preamble runs once even on fallback.
For the Q/K/V example, let X ∈ R[T, d_in] be the shared input and W_q, W_k, W_v have output widths d_q, d_k, d_v. Conceptually,
Splitting its last dimension at widths d_q, d_k, and d_v gives the three projection outputs. The widths need not be mathematically equal; each native backend still imposes its own tile/shape limits.
This is a shape explanation, not a dense implementation: the executor combines compatible packed quantized operands, not a persistent dequantized W_cat. Q/K/V still flow through the native parent's RoPE, KV-cache, and attention code. Gate/up follows the same concatenation/splitting idea; the activation, product, and down projection remain in their native order.
Fusion is not quantization-aware training or a new reconstruction objective. Its numerical checks compare combined execution against separate native calls, and its performance must be measured for the actual workload.
| What you observe | What to check next |
|---|---|
Fusion requires a post-quantized model |
Quantize first, or reload a quantized checkpoint. |
| Opt-in rejects the model definition | Consult the exact model qualification, not just ordinary model support. A model name or matching shapes cannot substitute for a declaration. |
effective quantization configurations differ or exclude a projection |
Check per-module dynamic overrides for every sibling, including exclusions, not only global settings. |
native forward cannot be safely rewritten from declared projection calls |
The live parent/source shape is unsupported; do not bypass the fail-closed inspection. |
Eligible groups, but fused_calls does not grow |
Keep the same plan before/after inference. Check eval/inference mode, input dtype/device, hooks, compile capture, and runtime operands. Repeated fuse() calls rebuild the counters. |
| Calls grow, but latency does not improve | Compare warmed-up runs with the same prompts, batch, dtype, and prefill/decode lengths. Separate launch overhead from end-to-end generation work. |
| Saved checkpoint does not auto-enable fusion | Expected: opt in again on load. Fusion is runtime state, not a new storage format. |
Start with a numerical baseline, then benchmark only after actual fused use is confirmed. Keep results scoped to the measured model/backend/device. The published CPU, NVIDIA CUDA, and AMD ROCm reports below are evidence for those environments, not tests performed by this wiki edit.
The checked-in inventory covers 205 registered model types, of which 102 have qualified profiles. The remaining 103 record explicit gaps. These are tiny native-model topology and projection checks, not full-model quality/performance results for every type. The exact, version-pinned model matrix and fixture list profiles and explicit qualification gaps.
granite_swa and granitemoe_swa are registered for ordinary quantization but explicitly set fusable=None. The current fusion executor does not model their per-layer full/sliding-window attention dispatch and post-attention sink scaling; inheritance from a Llama-like definition does not qualify them for fusion.
| Example | Qualified scope |
|---|---|
| Llama, Qwen2 | QKV and gate/up, subject to runtime checks. |
olmo_hybrid |
Full-attention QKV and MLP gate/up; the DeltaNet linear-attention parent is excluded. |
jamba |
Attention QKV only; both routed experts and dense feed-forward gate/up are excluded. |
| MoE models with attention-only contracts | Compatible non-expert attention only; MoE paths, including expert projections, are excluded. |
| Gemma4 / Gemma4 text | Gate/up only; shared-KV attention is not admitted. |
gemma3n_text |
Gate/up; this does not qualify the optional-dependency multimodal wrapper. |
| Ricursive Ripple | Not qualified: value-residual preprocessing and helper-based chunked MLP calls are outside the direct-forward executor. |
| DeepSeek V3/V4 | Not qualified: no separate non-expert QKV or gate/up role group in the declared tree. |
Ordinary model support does not imply fusion support. Multimodal profiles cover only the declared decoder text stack.
Backend selection happens before fusion. AUTO keeps the selected native class; the planner does not switch an unsupported backend to a portable one just to admit a group. Actual architecture, dependency, dtype, and native shape requirements still apply.
| Native layout | Important limits |
|---|---|
| Marlin GPTQ/AWQ on NVIDIA | Symmetric INT4, requested group size 128, desc_act=False, finalized unpadded operands, equal input widths, complete N64 output tiles, no activation reordering. |
| Portable Torch GPTQ/AWQ | Compatible int32-packed 2/4/8-bit storage and unpadded shapes; GPTQ group indices must match. Portable AWQ GQA QKV with unequal widths is rejected; native Marlin does not share that exclusion. |
| CPU ATen/AVX512/legacy INT4 | Finalized native int4pack storage and complete N32 tiles. AVX512 is BF16-only; legacy INT4 additionally requires symmetry. The AVX512 executor consumes the concatenated packed operands, without restoring a single source projection's snapshot over them. |
| AWQ GEMM/GEMV and eligible Triton variants | Exact native packing and settings must match. AWQ Triton and fast GEMV paths are FP16-only; fast packing uses int16 N4/K64 tiles. |
| ExLlamaV2 GPTQ/AWQ | Finalized descriptors and complete tiles; canonical input order only, not desc_act=True. |
| AllSpark / CPU INT8 | Native eight-bit layouts with their per-channel metadata; not persistent expansion of lower-bit weights. |
| GPTQ W4AFP8 | Ordinary tensor inputs only; encoded W4A activation streams are rejected. |
The repository's device report records validation on Zen 5 CPU, RTX 4090 NVIDIA CUDA, and R9700 AMD ROCm. Its observed AUTO choices are CPU ATen/AVX512, NVIDIA Marlin, and eligible AMD Torch/Triton paths; these are measured environments, not a universal priority override. Refer to the complete kernel inventory for exact classes, tested dtypes, and architecture/dependency exclusions.
Standalone GGUF, EXL3, FP8/FP4, BitsAndBytes, Paro, QQQ, VPTQ, MLX, BitBLAS, Machete, Swordfish, and W4A-NVFP4 inference support does not grant projection-fusion eligibility in this revision. Adapters, online projection rotation, projection hooks, and parent offload hooks are also excluded.
Original projection modules remain checkpoint owners. Their packed tensors
become disjoint views into one combined packed allocation. CPU INT4 sources
also capture non-persistent checkpoint state before finalization replaces
their checkpoint operands with native int4pack storage, even when their own
forward() has never run. Eligible ATen/fused sources replace the duplicate
qweight clone with a reconstruction descriptor only after verifying a
bit-exact inverse; unsupported layouts retain the packed snapshot. Affine
metadata is preserved as needed, and AVX512 also preserves bias widened to
FP32. See CPU checkpoint reconstruction.
Fusion retains no dense, dequantized,
or expanded weight cache. Native metadata and permitted workspace follow the
backend's own contract.
Checkpoint keys and projection tensor values are preserved; fusion execution
is runtime state, so opt in again when reloading a saved checkpoint.
Saving through get_state_dict_for_save() restores transformed CPU sources to
checkpoint layout before collecting tensors. Disabling fusion with
model.fuse(False) also restores that layout and gives each source independent
contiguous operands. If a save already restored the checkpoint while fusion
was installed, disabling fusion does not reinterpret those tensors as native
int4pack operands. The next eligible inference call rebuilds the native layout.
See Packing Formats and Compatibility
for checkpoint export and state-dict loading.
CPU cache clearing preserves the live packed operands and refreshes derived metadata. Partial checkpoint/runtime state loads are classified from the supplied operands; changed runtime values cannot be exported by restoring an older snapshot. Reload checkpoint-layout weights before such an export.
ExLlamaV2 GPTQ/AWQ descriptors borrow packed tensors that native initialization
shuffles in place. Cache clears and device/dtype moves rebuild descriptors
without shuffling finalized words a second time. Fusion keeps source and
combined descriptors live while rebinding packed views, and removal restores
independent source operands before releasing combined storage. After rebinding,
fusion clears ExLlamaV2 _q_fast operand snapshots on the sources and fused
executor so they cannot retain the replaced standalone packed allocations.
Descriptors and permutation metadata stay live; a direct source call rebuilds
its snapshot. Raw checkpoint
loads detach fused views before copying. GPTQ runtime state_dict() layout
metadata and AWQ runtime shapes distinguish finalized words from checkpoints;
after AWQ finalization, raw checkpoint qweight and qzeros must be loaded
together. See the ExLlamaV2 descriptor lifecycle.
Use inference/evaluation mode with compatible input device/dtype. Training,
gradient-requiring inputs, torch.compile capture, incompatible runtime
operands, or global forward hooks take the separate execution path. Fusion
does not make a backend trainable or qualify compiled fused execution.
Mutation guards invalidate/rebuild packed execution when source bindings,
versions, device/dtype, shape, native metadata, or workspace change. For manual
writes to tensors created under torch.inference_mode() without version
counters, call clear_weight_cache() on a source projection before reuse.
Unversioned writes through raw .data or storage aliases also require explicit
clearing when storage identity and metadata stay unchanged.
Do not assume an earlier eligible decision still proves runtime use; inspect
fused_calls after the workload.
Performance depends on method, kernel, dtype, device, and workload. Use the repository's pinned Llama 3.2 1B latency and accuracy tables rather than extrapolating a speedup to every qualified architecture.
BaseQModel.fusable defaults to None. FusableConfig, FusableProfile, and
LlamaFusableConfig are public gptqmodel exports. Composition is immutable
and target-first; it neither imports source model identities nor enables
execution:
from gptqmodel import FusableConfig, LlamaFusableConfig
fusable = FusableConfig("qwen2").merge(LlamaFusableConfig())
attention_only = FusableConfig("qwen2_moe").merge(
LlamaFusableConfig(), profiles=("qkv",)
)
policy = fusable.override("qkv", enabled=False, layers=(0, 1))
Definitions bind the completed result as their class's fusable contract.
Identical contributions deduplicate. Inherited profile conflicts require
replace_profile(); inherited policy conflicts require override().
Incomplete or conflicting contracts cannot execute. Adding a profile is not
evidence that the model or a new packed backend is numerically qualified.
See the pinned
composition reference
and the required real-model qualification tests in the
model-plan skill.
-
gptqmodel/models/base.py—fuse()and quantization finalization. -
gptqmodel/models/auto.pyandloader.py— loader opt-in and defaults. -
gptqmodel/models/fusable.py— definition contracts, composition, and scoped policy. -
gptqmodel/models/fusion.py— decisions, native-forward compilation, packed ownership, and fallback. -
gptqmodel/models/fusion_kernels.py— exact native layout contracts. -
gptqmodel/nn_modules/qlinear/_fused_train_state.pyandgptqmodel/utils/model.py— packed checkpoint snapshots and export restoration.
Documentation home · Quickstart · Troubleshooting · Repository · PyPI
Version scope: public source 54888582e (package version 7.12.0); older pinned evidence is identified on individual pages. Choose a method · Dtypes and formats. Maintained by ModelCloud.ai.
🧭 1 · Choose and understand
🚀 2 · Use it
- Installation and Quickstart
- Loading and Inference
- Evaluation
- Serving
- Troubleshooting and FAQ
- Memory, Offload, Multi-GPU
- MLX / Apple Silicon
- Tenstorrent / Block Floats
- Hugging Face Integration
- Multimodal and MoE Models
🎛️ Methods and representations
Each guide: choose → examples → math / advanced
- GPTQ / GPTAQ / RTN
- AWQ
- QQQ
- MACaQ / MXFP4
- EXL3
- VPTQ
- Paro
- FP8 and NVFP4
- NF12: lossless BF16 storage
- GGUF families
- BitsAndBytes: NF4 / FP4 / INT8
🧰 Refinement and execution
🧠 3 · Theory and advanced reference