Skip to content

Projection Fusion

Jump to bottom
Qubitium edited this page Nov 11, 2026 · 3 revisions

Post-quantization Projection Fusion

Documentation home · Quickstart · Lifecycle

On this page

GPT-QModel can combine compatible Q/K/V or gate/up projections into one wider quantized linear operation for inference. Fusion is disabled by default and requires an explicitly qualified model definition. It does not change the quantization method, checkpoint keys, or selected backend.

This page describes source revision 128367187, with projection fusion originally introduced in PR #3567. Use an installation containing the documented fixes; the release number alone does not establish that these APIs and behaviors are available.

Enable, inspect, and disable

Start from a quantized model returned by the GPT-QModel wrapper:

from gptqmodel import GPTQModel
from gptqmodel.utils.backend import BACKEND

model = GPTQModel.load("/path/to/quantized-llama", backend=BACKEND.AUTO)
model.eval()
plan = model.fuse()
plan.print()
text = plan.render()
decisions = plan.to_dict()

model.fuse(False)

model.fuse() applies fusion and returns a FusionPlanner, not the model. Each enabled call rebuilds the plan; retain its return value for inspection without mutation. model.fuse(False) restores separate execution and independent contiguous packed operands, releases the executors, and returns None. A retained old plan remains inspectable after disabling.

Entry point Default Behavior
GPTQModel.load(..., fuse=False) False Opt in with True; a quantized wrapper applies fusion during initialization. A source wrapper records the request for quantization finalization.
model.quantize(..., fuse=None) None Inherit the wrapper's current fusion setting. True enables fusion after successful quantization; False disables it. The return value remains quantization diagnostics, not the fusion plan.
model.fuse(enabled=True, *, overrides=None) True, no overrides Apply/rebuild a plan, or disable with False. Overrides map family names or exact parent paths to booleans.

The model-definition from_pretrained() and from_quantized() loaders also accept fuse=False; the public factory methods forward it through their keyword arguments. This is a runtime/quantization-call option, not a field of QuantizeConfig. Enabled values must be actual booleans, not 0, 1, or strings; only quantize() additionally accepts None.

For a source-model quantization run:

from gptqmodel import GPTQModel, QuantizeConfig

model = GPTQModel.load(
    "/path/to/source-llama",
    quantize_config=QuantizeConfig(bits=4, group_size=128, sym=True, desc_act=False),
)
# calibration_dataset is a representative dataset prepared by the caller.
logs = model.quantize(calibration_dataset, fuse=True)
plan = model.fuse()
plan.print()

Calling fuse() directly on an unquantized wrapper raises RuntimeError. A definition without a matching, complete FusableConfig raises on opt-in rather than silently enabling nothing; quantization validates the requested definition before calibration. Disabling remains valid for unqualified models. A qualified model can still return a plan with no eligible groups.

What the plan reports

plan.print(file=...) writes the terminal summary; plan.render() returns the same text. plan.to_dict() includes:

Key Meaning
fusable Serialized definition profiles, model types, policy, origins, and composition diagnostics.
groups Decisions with family, full parent path, layer-relative paths, enabled, reason, profile_origins, and winning policy.
fused_calls Count of fused executions performed by this plan; inspect after inference to distinguish eligibility from actual use.

An enabled decision means that a group passed planning checks, not that every subsequent forward uses it. Runtime input, training, hooks, and compiler state can cause separate execution. Excluded MoE parents are not enumerated as fusion candidates in the first place.

Overrides and selection

Built-in role profiles are qkv (q, k, v) and gate_up (gate, up). Paths resolve from the compiled module plan, not a name suffix or an HF class-path allowlist. Every role must identify exactly one distinct quantizable sibling, with compatible live operands and parent code.

# Each call replaces the previous plan and its per-call overrides.
plan = model.fuse(overrides={"qkv": False})
plan = model.fuse(overrides={"model.layers.0.mlp": False})

An exact full parent path wins over a layer-relative parent, which wins over the family within the same policy scope. Runtime overrides are checked before definition policy; definition layer-scoped policy precedes its family default. These are exact selectors, not regular expressions or dynamic quantization rules. Use paths shown by the plan for the actual model.

Effective quantization options are compared after per-module dynamic overrides. Different bits, group size, symmetry, activation ordering, format, or other serialized options can reject the whole sibling group. A disabled dynamic target also rejects it. Equal global settings alone do not prove compatibility.

The native parent forward must make each declared projection call exactly once on the same input, without extra call arguments. Conditional/helper calls, rewritten inputs, extra projection-weight references, decorated or instance-overridden forwards, and unavailable source fail closed. The parent retains its native RoPE, cache updates, attention dispatch, activation, multiplication, and output projection. A preamble runs once even on fallback.

Model qualification

The checked-in inventory covers 198 registered model types, of which 101 have qualified profiles. These are tiny native-model topology and projection checks, not full-model quality/performance results for every type. The exact, version-pinned model matrix and fixture list profiles and explicit qualification gaps.

Example Qualified scope
Llama, Qwen2 QKV and gate/up, subject to runtime checks.
MoE models with attention-only contracts Compatible non-expert attention only; MoE paths, including expert projections, are excluded.
Gemma4 / Gemma4 text Gate/up only; shared-KV attention is not admitted.
gemma3n_text Gate/up; this does not qualify the optional-dependency multimodal wrapper.
Ricursive Ripple Not qualified: value-residual preprocessing and helper-based chunked MLP calls are outside the direct-forward executor.
DeepSeek V3/V4 Not qualified: no separate non-expert QKV or gate/up role group in the declared tree.

Ordinary model support does not imply fusion support. Multimodal profiles cover only the declared decoder text stack.

Native backends and limits

Backend selection happens before fusion. AUTO keeps the selected native class; the planner does not switch an unsupported backend to a portable one just to admit a group. Actual architecture, dependency, dtype, and native shape requirements still apply.

Native layout Important limits
Marlin GPTQ/AWQ on NVIDIA Symmetric INT4, requested group size 128, desc_act=False, finalized unpadded operands, equal input widths, complete N64 output tiles, no activation reordering.
Portable Torch GPTQ/AWQ Compatible int32-packed 2/4/8-bit storage and unpadded shapes; GPTQ group indices must match. Portable AWQ GQA QKV with unequal widths is rejected; native Marlin does not share that exclusion.
CPU ATen/AVX512/legacy INT4 Finalized native int4pack storage and complete N32 tiles. AVX512 is BF16-only; legacy INT4 additionally requires symmetry. The AVX512 executor consumes the concatenated packed operands, without restoring a single source projection's snapshot over them.
AWQ GEMM/GEMV and eligible Triton variants Exact native packing and settings must match. AWQ Triton and fast GEMV paths are FP16-only; fast packing uses int16 N4/K64 tiles.
ExLlamaV2 GPTQ/AWQ Finalized descriptors and complete tiles; canonical input order only, not desc_act=True.
AllSpark / CPU INT8 Native eight-bit layouts with their per-channel metadata; not persistent expansion of lower-bit weights.
GPTQ W4AFP8 Ordinary tensor inputs only; encoded W4A activation streams are rejected.

The repository's device report records validation on Zen 5 CPU, RTX 4090 NVIDIA CUDA, and R9700 AMD ROCm. Its observed AUTO choices are CPU ATen/AVX512, NVIDIA Marlin, and eligible AMD Torch/Triton paths; these are measured environments, not a universal priority override. Refer to the complete kernel inventory for exact classes, tested dtypes, and architecture/dependency exclusions.

Standalone GGUF, EXL3, FP8/FP4, BitsAndBytes, ParoQuant, QQQ, VPTQ, MLX, BitBLAS, Machete, Swordfish, and W4A-NVFP4 inference support does not grant projection-fusion eligibility in this revision. Adapters, online projection rotation, projection hooks, and parent offload hooks are also excluded.

Storage, fallback, and saving

Original projection modules remain checkpoint owners. Their packed tensors become disjoint views into one combined packed allocation. CPU INT4 sources also retain a non-persistent checkpoint snapshot because finalization replaces their checkpoint operands with native int4pack storage. The snapshot contains packed weights and affine metadata; the AVX512 classes also preserve the bias they widen to FP32. It is captured before fusion transforms the sources, even when their own forward() has never run. Fusion retains no dense, dequantized, or expanded weight cache. Native metadata and permitted workspace follow the backend's own contract. Checkpoint keys and projection tensor values are preserved; fusion execution is runtime state, so opt in again when reloading a saved checkpoint.

Saving through get_state_dict_for_save() restores transformed CPU sources to checkpoint layout before collecting tensors. Disabling fusion with model.fuse(False) also restores that layout and gives each source independent contiguous operands. If a save already restored the checkpoint while fusion was installed, disabling fusion does not reinterpret those tensors as native int4pack operands. The next eligible inference call rebuilds the native layout. See Packing Formats and Compatibility for checkpoint export and state-dict loading.

CPU cache clearing preserves the live packed operands and refreshes derived metadata. Partial checkpoint/runtime state loads are classified from the supplied operands; changed runtime values cannot be exported by restoring an older snapshot. Reload checkpoint-layout weights before such an export.

ExLlamaV2 GPTQ/AWQ descriptors borrow packed tensors that native initialization shuffles in place. Cache clears and device/dtype moves rebuild descriptors without shuffling finalized words a second time. Fusion keeps source and combined descriptors live while rebinding packed views, and removal restores independent source operands before releasing combined storage. Raw checkpoint loads detach fused views before copying. GPTQ runtime state_dict() layout metadata and AWQ runtime shapes distinguish finalized words from checkpoints; after AWQ finalization, raw checkpoint qweight and qzeros must be loaded together. See the ExLlamaV2 descriptor lifecycle.

Use inference/evaluation mode with compatible input device/dtype. Training, gradient-requiring inputs, torch.compile capture, incompatible runtime operands, or global forward hooks take the separate execution path. Fusion does not make a backend trainable or qualify compiled fused execution.

Mutation guards invalidate/rebuild packed execution when source bindings, versions, device/dtype, shape, native metadata, or workspace change. For manual writes to tensors created under torch.inference_mode() without version counters, call clear_weight_cache() on a source projection before reuse. Unversioned writes through raw .data or storage aliases also require explicit clearing when storage identity and metadata stay unchanged. Do not assume an earlier eligible decision still proves runtime use; inspect fused_calls after the workload.

Performance depends on method, kernel, dtype, device, and workload. Use the repository's pinned Llama 3.2 1B latency and accuracy tables rather than extrapolating a speedup to every qualified architecture.

Extending a definition

BaseQModel.fusable defaults to None. FusableConfig, FusableProfile, and LlamaFusableConfig are public gptqmodel exports. Composition is immutable and target-first; it neither imports source model identities nor enables execution:

from gptqmodel import FusableConfig, LlamaFusableConfig

fusable = FusableConfig("qwen2").merge(LlamaFusableConfig())
attention_only = FusableConfig("qwen2_moe").merge(
    LlamaFusableConfig(), profiles=("qkv",)
)
policy = fusable.override("qkv", enabled=False, layers=(0, 1))

Definitions bind the completed result as their class's fusable contract. Identical contributions deduplicate. Inherited profile conflicts require replace_profile(); inherited policy conflicts require override(). Incomplete or conflicting contracts cannot execute. Adding a profile is not evidence that the model or a new packed backend is numerically qualified. See the pinned composition reference and the required real-model qualification tests in the model-plan skill.

Source files

See also

Documentation home · Quickstart · Troubleshooting · Repository · PyPI

Source baseline: GPT-QModel 7.9.0 / 40948a6. Maintained by ModelCloud.ai.

Home

Getting started

Quantization methods

Complementary configs

Using a model

Platforms

Internals

Clone this wiki locally