Skip to content

Projection Fusion

Qubitium edited this page Nov 17, 2026 · 10 revisions

⚡ Projection fusion: fewer sibling linear calls

🏠 Home · 🧭 Choose a method · 🚀 Quickstart · 🧰 Help

On this page

Projection fusion is an opt-in inference optimization: run compatible Q/K/V or gate/up projections together while keeping the quantized checkpoint and the parent's attention/MLP math intact.

For: users who already have a working quantized model and want to test an execution optimization—not choose a new quantization method or dtype.

GPT-QModel can combine compatible Q/K/V or gate/up projections into one wider quantized linear operation for inference. Fusion is disabled by default and requires an explicitly qualified model definition. It does not change the quantization method, checkpoint keys, or selected backend.

This page describes source revision 162e34541, with projection fusion originally introduced in PR #3567. Use an installation containing the documented fixes; the release number alone does not establish that these APIs and behaviors are available.

🧭 Choose the optimization, not a new format

Choose fusion when… Consider another path when…
A qualified definition declares compatible sibling projections for your already-selected native backend. You only need basic model support: Supported Models is not the fusion qualification matrix.
You can measure an inference workload in eval mode with stable packed weights. You need training, compiled fused execution, projection hooks, or adapters: those paths are excluded or fall back.
You want to reduce separate linear launches and can verify actual fused calls. You need a guaranteed speedup: eligibility is not a latency result.
You want the same checkpoint keys and quantized storage. You need a different compression method or runtime: start with Choosing a Method and Dtype.

Prerequisites for the first example: a locally loadable, qualified quantized model; compatible dtype/device and native layout; no adapter, online projection rotation, projection hooks, or parent offload hook. The code below assumes a single-device causal-LM checkpoint with its tokenizer. Multi-device deployments must place inputs for their actual entry module.

Read in three passes: enable and verify → inspect decisions and overrides → qualification, storage, and extension reference.

Fusion leaves AUTO's native kernel choice intact. A rejected backend is not silently replaced by Torch just to make fusion appear supported. Keep NVIDIA CUDA and AMD ROCm architecture requirements separate.

🚀 Use it: enable, verify, and control

Enable, inspect, and disable

Start from a quantized model returned by the GPT-QModel wrapper. This is an API/numerical smoke check, not a latency benchmark or an accuracy recipe:

import torch
from gptqmodel import BACKEND, GPTQModel

model = GPTQModel.load("path/to/quantized-llama", backend=BACKEND.AUTO)
model.eval()
inputs = model.tokenizer(
    "Explain why the sky appears blue.", return_tensors="pt"
).to(model.model.device)

# Baseline first, then enable and run the same workload.
with torch.inference_mode():
    baseline = model.model(**inputs).logits.float()

plan = model.fuse()
plan.print()

with torch.inference_mode():
    fused = model.model(**inputs).logits.float()

print("maximum absolute logit drift:", (fused - baseline).abs().max().item())
print("actual fused calls:", plan.to_dict()["fused_calls"])
decisions = plan.to_dict()

model.fuse(False)

Success means: the plan contains eligible groups and fused_calls grows after the forward. Inspect the measured logit drift using tolerances appropriate to your dtype/backend; do not assume bitwise identity. Evaluate held-out quality separately. A plan with zero eligible groups or zero calls is not evidence of an optimization.

model.fuse() applies fusion and returns a FusionPlanner, not the model. Each enabled call rebuilds the plan; retain its return value for inspection without mutation. model.fuse(False) restores separate execution and independent contiguous packed operands, releases the executors, and returns None. A retained old plan remains inspectable after disabling.

Entry point Default Behavior
GPTQModel.load(..., fuse=False) False Opt in with True; a quantized wrapper applies fusion during initialization. A source wrapper records the request for quantization finalization.
model.quantize(..., fuse=None) None Inherit the wrapper's current fusion setting. True enables fusion after successful quantization; False disables it. The return value remains quantization diagnostics, not the fusion plan.
model.fuse(enabled=True, *, overrides=None) True, no overrides Apply/rebuild a plan, or disable with False. Overrides map family names or exact parent paths to booleans.

The model-definition from_pretrained() and from_quantized() loaders also accept fuse=False; the public factory methods forward it through their keyword arguments. This is a runtime/quantization-call option, not a field of QuantizeConfig. Enabled values must be actual booleans, not 0, 1, or strings; only quantize() additionally accepts None.

For a source-model quantization run:

from gptqmodel import CalibrationConfig, GPTQModel, QuantizeConfig

model = GPTQModel.load(
    "/path/to/source-llama",
    quantize_config=QuantizeConfig(bits=4, group_size=128, sym=True, desc_act=False),
)
# calibration_dataset is a representative dataset prepared by the caller.
# A few short strings would be an API demo only, not an accuracy recipe.
logs = model.quantize(CalibrationConfig(dataset=calibration_dataset), fuse=True)
plan = model.fuse()
plan.print()

Calling fuse() directly on an unquantized wrapper raises RuntimeError. A definition without a matching, complete FusableConfig raises on opt-in rather than silently enabling nothing; quantization validates the requested definition before calibration. Disabling remains valid for unqualified models. A qualified model can still return a plan with no eligible groups.

What the plan reports

plan.print(file=...) writes the terminal summary; plan.render() returns the same text. plan.to_dict() includes:

Key Meaning
fusable Serialized definition profiles, model types, policy, origins, and composition diagnostics.
groups Decisions with family, full parent path, layer-relative paths, enabled, reason, profile_origins, and winning policy.
fused_calls Count of fused executions performed by this plan; inspect after inference to distinguish eligibility from actual use.

An enabled decision means that a group passed planning checks, not that every subsequent forward uses it. Runtime input, training, hooks, and compiler state can cause separate execution. Excluded MoE parents are not enumerated as fusion candidates in the first place.

Overrides and selection

Built-in role profiles are qkv (q, k, v) and gate_up (gate, up). Paths resolve from the compiled module plan, not a name suffix or an HF class-path allowlist. Every role must identify exactly one distinct quantizable sibling, with compatible live operands and parent code.

# Each call replaces the previous plan and its per-call overrides.
plan = model.fuse(overrides={"qkv": False})
plan = model.fuse(overrides={"model.layers.0.mlp": False})

An exact full parent path wins over a layer-relative parent, which wins over the family within the same policy scope. Runtime overrides are checked before definition policy; definition layer-scoped policy precedes its family default. These are exact selectors, not regular expressions or dynamic quantization rules. Use paths shown by the plan for the actual model.

Effective quantization options are compared after per-module dynamic overrides. Different bits, group size, symmetry, activation ordering, format, or other serialized options can reject the whole sibling group. A disabled dynamic target also rejects it. Equal global settings alone do not prove compatibility.

The native parent forward must make each declared projection call exactly once on the same input, without extra call arguments. Conditional/helper calls, rewritten inputs, extra projection-weight references, decorated or instance-overridden forwards, and unavailable source fail closed. The parent retains its native RoPE, cache updates, attention dispatch, activation, multiplication, and output projection. A preamble runs once even on fallback.

🧠 Go deeper: mechanism and reference

🔎 What is being combined?

For the Q/K/V example, let X ∈ R[T, d_in] be the shared input and W_q, W_k, W_v have output widths d_q, d_k, d_v. Conceptually,

$$W_{\mathrm{cat}}= \begin{bmatrix}W_q\\W_k\\W_v\end{bmatrix},\qquad Y_{\mathrm{cat}}=XW_{\mathrm{cat}}^T+b_{\mathrm{cat}}.$$

Splitting its last dimension at widths d_q, d_k, and d_v gives the three projection outputs. The widths need not be mathematically equal; each native backend still imposes its own tile/shape limits.

This is a shape explanation, not a dense implementation: the executor combines compatible packed quantized operands, not a persistent dequantized W_cat. Q/K/V still flow through the native parent's RoPE, KV-cache, and attention code. Gate/up follows the same concatenation/splitting idea; the activation, product, and down projection remain in their native order.

Fusion is not quantization-aware training or a new reconstruction objective. Its numerical checks compare combined execution against separate native calls, and its performance must be measured for the actual workload.

🛠️ Troubleshooting and measurement

What you observe What to check next
Fusion requires a post-quantized model Quantize first, or reload a quantized checkpoint.
Opt-in rejects the model definition Consult the exact model qualification, not just ordinary model support. A model name or matching shapes cannot substitute for a declaration.
effective quantization configurations differ or exclude a projection Check per-module dynamic overrides for every sibling, including exclusions, not only global settings.
native forward cannot be safely rewritten from declared projection calls The live parent/source shape is unsupported; do not bypass the fail-closed inspection.
Eligible groups, but fused_calls does not grow Keep the same plan before/after inference. Check eval/inference mode, input dtype/device, hooks, compile capture, and runtime operands. Repeated fuse() calls rebuild the counters.
Calls grow, but latency does not improve Compare warmed-up runs with the same prompts, batch, dtype, and prefill/decode lengths. Separate launch overhead from end-to-end generation work.
Saved checkpoint does not auto-enable fusion Expected: opt in again on load. Fusion is runtime state, not a new storage format.

Start with a numerical baseline, then benchmark only after actual fused use is confirmed. Keep results scoped to the measured model/backend/device. The published CPU, NVIDIA CUDA, and AMD ROCm reports below are evidence for those environments, not tests performed by this wiki edit.

Model qualification

The checked-in inventory covers 205 registered model types, of which 102 have qualified profiles. The remaining 103 record explicit gaps. These are tiny native-model topology and projection checks, not full-model quality/performance results for every type. The exact, version-pinned model matrix and fixture list profiles and explicit qualification gaps.

granite_swa and granitemoe_swa are registered for ordinary quantization but explicitly set fusable=None. The current fusion executor does not model their per-layer full/sliding-window attention dispatch and post-attention sink scaling; inheritance from a Llama-like definition does not qualify them for fusion.

Example Qualified scope
Llama, Qwen2 QKV and gate/up, subject to runtime checks.
olmo_hybrid Full-attention QKV and MLP gate/up; the DeltaNet linear-attention parent is excluded.
jamba Attention QKV only; both routed experts and dense feed-forward gate/up are excluded.
MoE models with attention-only contracts Compatible non-expert attention only; MoE paths, including expert projections, are excluded.
Gemma4 / Gemma4 text Gate/up only; shared-KV attention is not admitted.
gemma3n_text Gate/up; this does not qualify the optional-dependency multimodal wrapper.
Ricursive Ripple Not qualified: value-residual preprocessing and helper-based chunked MLP calls are outside the direct-forward executor.
DeepSeek V3/V4 Not qualified: no separate non-expert QKV or gate/up role group in the declared tree.

Ordinary model support does not imply fusion support. Multimodal profiles cover only the declared decoder text stack.

Native backends and limits

Backend selection happens before fusion. AUTO keeps the selected native class; the planner does not switch an unsupported backend to a portable one just to admit a group. Actual architecture, dependency, dtype, and native shape requirements still apply.

Native layout Important limits
Marlin GPTQ/AWQ on NVIDIA Symmetric INT4, requested group size 128, desc_act=False, finalized unpadded operands, equal input widths, complete N64 output tiles, no activation reordering.
Portable Torch GPTQ/AWQ Compatible int32-packed 2/4/8-bit storage and unpadded shapes; GPTQ group indices must match. Portable AWQ GQA QKV with unequal widths is rejected; native Marlin does not share that exclusion.
CPU ATen/AVX512/legacy INT4 Finalized native int4pack storage and complete N32 tiles. AVX512 is BF16-only; legacy INT4 additionally requires symmetry. The AVX512 executor consumes the concatenated packed operands, without restoring a single source projection's snapshot over them.
AWQ GEMM/GEMV and eligible Triton variants Exact native packing and settings must match. AWQ Triton and fast GEMV paths are FP16-only; fast packing uses int16 N4/K64 tiles.
ExLlamaV2 GPTQ/AWQ Finalized descriptors and complete tiles; canonical input order only, not desc_act=True.
AllSpark / CPU INT8 Native eight-bit layouts with their per-channel metadata; not persistent expansion of lower-bit weights.
GPTQ W4AFP8 Ordinary tensor inputs only; encoded W4A activation streams are rejected.

The repository's device report records validation on Zen 5 CPU, RTX 4090 NVIDIA CUDA, and R9700 AMD ROCm. Its observed AUTO choices are CPU ATen/AVX512, NVIDIA Marlin, and eligible AMD Torch/Triton paths; these are measured environments, not a universal priority override. Refer to the complete kernel inventory for exact classes, tested dtypes, and architecture/dependency exclusions.

Standalone GGUF, EXL3, FP8/FP4, BitsAndBytes, Paro, QQQ, VPTQ, MLX, BitBLAS, Machete, Swordfish, and W4A-NVFP4 inference support does not grant projection-fusion eligibility in this revision. Adapters, online projection rotation, projection hooks, and parent offload hooks are also excluded.

Storage, fallback, and saving

Original projection modules remain checkpoint owners. Their packed tensors become disjoint views into one combined packed allocation. CPU INT4 sources also capture non-persistent checkpoint state before finalization replaces their checkpoint operands with native int4pack storage, even when their own forward() has never run. Eligible ATen/fused sources replace the duplicate qweight clone with a reconstruction descriptor only after verifying a bit-exact inverse; unsupported layouts retain the packed snapshot. Affine metadata is preserved as needed, and AVX512 also preserves bias widened to FP32. See CPU checkpoint reconstruction. Fusion retains no dense, dequantized, or expanded weight cache. Native metadata and permitted workspace follow the backend's own contract. Checkpoint keys and projection tensor values are preserved; fusion execution is runtime state, so opt in again when reloading a saved checkpoint.

Saving through get_state_dict_for_save() restores transformed CPU sources to checkpoint layout before collecting tensors. Disabling fusion with model.fuse(False) also restores that layout and gives each source independent contiguous operands. If a save already restored the checkpoint while fusion was installed, disabling fusion does not reinterpret those tensors as native int4pack operands. The next eligible inference call rebuilds the native layout. See Packing Formats and Compatibility for checkpoint export and state-dict loading.

CPU cache clearing preserves the live packed operands and refreshes derived metadata. Partial checkpoint/runtime state loads are classified from the supplied operands; changed runtime values cannot be exported by restoring an older snapshot. Reload checkpoint-layout weights before such an export.

ExLlamaV2 GPTQ/AWQ descriptors borrow packed tensors that native initialization shuffles in place. Cache clears and device/dtype moves rebuild descriptors without shuffling finalized words a second time. Fusion keeps source and combined descriptors live while rebinding packed views, and removal restores independent source operands before releasing combined storage. After rebinding, fusion clears ExLlamaV2 _q_fast operand snapshots on the sources and fused executor so they cannot retain the replaced standalone packed allocations. Descriptors and permutation metadata stay live; a direct source call rebuilds its snapshot. Raw checkpoint loads detach fused views before copying. GPTQ runtime state_dict() layout metadata and AWQ runtime shapes distinguish finalized words from checkpoints; after AWQ finalization, raw checkpoint qweight and qzeros must be loaded together. See the ExLlamaV2 descriptor lifecycle.

Use inference/evaluation mode with compatible input device/dtype. Training, gradient-requiring inputs, torch.compile capture, incompatible runtime operands, or global forward hooks take the separate execution path. Fusion does not make a backend trainable or qualify compiled fused execution.

Mutation guards invalidate/rebuild packed execution when source bindings, versions, device/dtype, shape, native metadata, or workspace change. For manual writes to tensors created under torch.inference_mode() without version counters, call clear_weight_cache() on a source projection before reuse. Unversioned writes through raw .data or storage aliases also require explicit clearing when storage identity and metadata stay unchanged. Do not assume an earlier eligible decision still proves runtime use; inspect fused_calls after the workload.

Performance depends on method, kernel, dtype, device, and workload. Use the repository's pinned Llama 3.2 1B latency and accuracy tables rather than extrapolating a speedup to every qualified architecture.

Extending a definition

BaseQModel.fusable defaults to None. FusableConfig, FusableProfile, and LlamaFusableConfig are public gptqmodel exports. Composition is immutable and target-first; it neither imports source model identities nor enables execution:

from gptqmodel import FusableConfig, LlamaFusableConfig

fusable = FusableConfig("qwen2").merge(LlamaFusableConfig())
attention_only = FusableConfig("qwen2_moe").merge(
    LlamaFusableConfig(), profiles=("qkv",)
)
policy = fusable.override("qkv", enabled=False, layers=(0, 1))

Definitions bind the completed result as their class's fusable contract. Identical contributions deduplicate. Inherited profile conflicts require replace_profile(); inherited policy conflicts require override(). Incomplete or conflicting contracts cannot execute. Adding a profile is not evidence that the model or a new packed backend is numerically qualified. See the pinned composition reference and the required real-model qualification tests in the model-plan skill.

Source files

See also

Clone this wiki locally