Repository navigation
HuggingFace Integration
🏠 Home · 🧭 Choose a method · 🚀 Quickstart · 🧰 Help
On this page
Use GPT-QModel with Hugging Face model repositories, Transformers, and PEFT while keeping checkpoint and loader responsibilities clear. This page is for people loading, saving, or adapting a quantized model, and for integrators wiring those steps into another library.
A Hugging Face repository is a distribution location; it does not tell you which quantization algorithm, packed layout, or runtime the weights require. Begin with Choosing a Method and Dtype if those decisions are still open.
| Entry point | Choose this when | Consider another path when |
|---|---|---|
GPTQModel.load() |
You want GPT-QModel’s model support, backend selection, or quantization workflow. | You need an application that owns the model lifecycle through Transformers. |
Transformers from_pretrained()
|
The artifact’s quantization metadata and your installed Transformers/GPT-QModel bridge support that path. | You need a GPT-QModel-specific algorithm or loader feature absent from the bridge. |
Standalone hf_ packing helpers |
You own individual layers and need explicit feasibility, packing, or repacking. | You expect them to discover the full model architecture or run calibration. |
| PEFT / LoRA | You want an adapter around a compatible packed base model. | You expect adapter training to redefine the base checkpoint’s storage format. |
Before you begin: check the model architecture, quantization metadata, required optional backend, and any remote-code requirement. NVIDIA CUDA and AMD ROCm remain separate hardware contracts even when an underlying Torch runtime spells both device namespaces cuda.
| Reading level | Start here |
|---|---|
| Choose / understand | Who owns each integration layer and native load hooks |
| Use it | Load, save, or train/interchange adapters |
| Go deeper | Compatibility normalization, GGUF discovery, and layer packing |
For a standalone .gguf file, follow GGUF artifact discovery. GGUF encompasses multiple tensor qtypes; a file extension is not a dtype or a promise of Transformers compatibility.
Guide source: hf.py, hf_load.py, auto.py.
GPT-QModel integrates with the Hugging Face ecosystem at several distinct levels: model/configuration/tokenizer loading, Transformers and Optimum quantized-layer selection and packing, checkpoint serialization, PEFT adapter interoperability, and local replacements for optional Hub kernels. These layers should not be confused: a checkpoint being hosted on the Hub does not imply that every Transformers backend can execute its packing format.
This page describes source revision 2e7992a18, including compatibility code for Transformers 5.x. Dependency minimums in this tree are transformers>=5.20.2 on Python <3.11 or >=5.23 on Python >=3.11 (requirements.txt) and optimum>=2.3.0 (the hf extra in pyproject.toml). Install the integration extra and PEFT separately when fine-tuning:
pip install 'gptqmodel[hf]' peftBackend availability still depends on hardware and optional dependencies. Selecting a CUDA kernel in Transformers does not make it usable on CPU, ROCm, or Apple silicon. See Loading and Inference, Inference Backends, and Packing Formats and Compatibility.
For a first load, choose one loader from the examples below. Use an existing quantized checkpoint whose method, format and architecture are supported by that loader. BACKEND.AUTO applies declared compatibility and priority rules; it does not benchmark your model. A portable method-specific backend is a useful diagnostic alternative when an optimized backend rejects the artifact.
After loading, keep the model’s tokenizer or multimodal processor with it. Compare prompts and token IDs against the source model before interpreting poor generation as quantization error.
For a supported quantized checkpoint, GPT-QModel owns backend selection:
from gptqmodel import GPTQModel
model = GPTQModel.load("organization/quantized-model")Transformers can instead load supported quantized checkpoints through its quantizer integration:
import gptqmodel # Install GPTQModel's process-local compatibility patches.
from transformers import AutoConfig, AutoModelForCausalLM
model_id = "organization/quantized-model"
config = AutoConfig.from_pretrained(model_id)
config.quantization_config["backend"] = "gptq_torch" # GPTQ example.
model = AutoModelForCausalLM.from_pretrained(
model_id,
config=config,
device_map="auto",
)The explicit backend above is a GPTQ example, not a backend valid for all methods. For AWQ LoRA training, use torch_awq, the Transformers-facing alias for AWQ_TORCH. The available method/format/backend combinations are described in Kernel Backends Reference.
# `model` is a quantized GPTQModel wrapper.
model.save("./quantized-output")save() and save_pretrained() choose a writer based on model state. On an unquantized wrapper, both delegate to the underlying Transformers model's save_pretrained(); save the tokenizer or processor separately if you need a complete dense artifact. For an ordinary quantized model, both use save_quantized(). For a weight-only module-replacement checkpoint, both use save_quantized_embeddings() so the packed endpoints survive. The replacement writer also saves the live generation_config.json when available for a generative model.
save_quantized() requires a quantized model and writes packed safetensors with optional sharding/index, model config containing quantization_config, standalone quantize_config.json, and generation config. It preserves first-match ordering of dynamic rules after Transformers config serialization.
The writer also saves tokenizer files, then the multimodal processor so a processor-aware chat template is not overwritten by a text-only tokenizer template. For trusted remote-code checkpoints it copies Python source files. It restores a source-compatible tokenizer class name instead of leaving an incompatible modern backend name in saved metadata. EoRA adapter output may be in a separate directory selected by adapter path or eora_path; include it intentionally when publishing.
There is no dedicated GPTQModel.push_to_hub implementation or upload-folder pipeline in this tree. BaseQModel.__getattr__ forwards unknown attributes to the underlying HF model, so an apparent model.push_to_hub(...) may be the Transformers method. It must not be assumed to run GPT-QModel's quantized writer, preserve all auxiliary artifacts, or serialize the intended checkpoint ABI.
For reliable publication, save with GPT-QModel first, inspect/reload that directory with the intended runtime, and upload the directory using the supported Hugging Face Hub client/CLI. utils/hub.py exposes create_repo for repository creation, but does not expose upload_folder or upload_file. Those are Hub-library operations, not extra model.save parameters. Upload the quantization configs, safetensors and index, tokenizer/processor files, required trusted code, and intended adapter artifacts together. Avoid uploading unrelated local calibration data or credentials.
The same separation applies to an MLX-native output: save the MLX-LM checkpoint with GPTQModel.quantize_mlx, then publish that artifact as MLX-LM output rather than advertising it as a GPTQ safetensors checkpoint.
Next: validate a fresh local reload before distributing the directory. See Packing Formats and Compatibility for the difference between the serialization container and the packed tensor ABI.
The quantized base stays frozen; train only the LoRA parameters. Inference-only kernels do not provide the gradient propagation needed through the quantized layers.
| Checkpoint method | Transformers training backend |
|---|---|
| GPTQ | gptq_torch |
| AWQ | torch_awq |
For FP8 loaded through GPT-QModel's FP8_TORCH backend, PEFT can wrap TorchFP8Linear with LoRA. The layer exposes qweight as an alias of the existing packed weight for PEFT's GPTQ dispatcher; FP8 checkpoint tensor names do not change. This does not add training support to FP8 inference-only backends. See the Torch FP8 layer contract.
import gptqmodel # Apply process-local Transformers compatibility patches.
from peft import LoraConfig, get_peft_model
from transformers import AutoConfig, AutoModelForCausalLM
quant_path = "organization/quantized-model"
config = AutoConfig.from_pretrained(quant_path)
config.quantization_config["backend"] = {
"gptq": "gptq_torch", "awq": "torch_awq",
}[config.quantization_config["quant_method"]]
base = AutoModelForCausalLM.from_pretrained(quant_path, config=config, device_map="auto")
model = get_peft_model(base, LoraConfig(
r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM",
))
# Train with Trainer or a training loop, then save only adapter files.
model.save_pretrained("./adapter")At inference, keep the adapter separate:
from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained(quant_path, device_map="auto")
model = PeftModel.from_pretrained(base, "./adapter")This peft.LoraConfig is the external training configuration. It is not the similarly named serialization dataclass in gptqmodel.adapter.peft.
PEFT AdaLoRA adds its adapter output in place. Trainable GPTQ Torch and Triton paths preserve backpropagation by keeping reshape views out of the custom autograd return, including in the wide-bias epilogue. Native and Triton AWQ GEMM return an owned tensor for 2-D and batched activations. See AdaLoRA regression coverage and the kernel output tests.
gptqmodel.adapter.peft.LoraConfig serializes adapter config for GPT-QModel/EoRA interchange. It does not create or train a PEFT model.
| Field | Default |
|---|---|
base_model_name_or_path |
"" |
r, lora_alpha, lora_dropout
|
8, 8, 0.0
|
target_modules, exclude_modules, modules_to_save
|
None |
fan_in_fan_out, use_rslora
|
False |
bias |
"none"; annotation allows "none", "all", "lora_only"
|
init_lora_weights |
True; annotated strings gaussian, eva, olora, pissa, pissa_niter_[number of iters], loftq are subject to validation below |
layers_to_transform, layers_pattern
|
None |
rank_pattern, alpha_pattern
|
New empty dict per instance |
megatron_config, eva_config
|
None |
megatron_core |
"megatron.core" |
loftq_config |
New empty dict per instance |
use_dora, lora_bias
|
False |
layer_replication, auto_mapping
|
None |
task_type, peft_type
|
"CAUSAL_LM", "LORA" (peft_type forced to "LORA" after init) |
__post_init__ converts target/exclude lists to sets. Regex target_modules cannot be combined with layers_to_transform or layers_pattern. A nonempty layers_pattern requires layers_to_transform to be specified; layer index 0 passes validation. It rejects LoftQ selection/config, LoRA bias, DoRA, non-default trained bias, modules_to_save, and layer replication. It also rejects EVA selection without an EVA config, an EVA config when not selecting EVA, and the combination of RSLoRA, rank/alpha patterns, and PiSSA/OLoRA initialization. Accepting a metadata field is not evidence that the minimal runtime implements the corresponding PEFT behavior.
to_dict() removes None, empty dict, and empty list values, converts sets to lists, and retains false/zero values. save_pretrained(self, save_dir: str) writes adapter_config.json. The loader signature is:
@classmethod
def from_pretrained(
cls,
path: str,
filename: str,
*,
allow_remote: bool = False,
revision: Optional[str] = None,
):
...It resolves the file through adapter.remote.resolve_path and rejects a declared non-LoRA peft_type and enabled QALoRA, aLoRA, target_parameters, CORDA, and trainable_token_indices features before filtering unknown/empty fields and constructing the dataclass. Other unknown metadata is warned about and ignored. Adapter tensor loading separately rejects recognized non-LoRA tensor layouts, DoRA magnitude vectors, saved extra modules, trained-token deltas, and bias tensors even if JSON flags are missing. This restriction applies to GPT-QModel's file-backed loader, not the external PEFT training API above. See EoRA feature limits. Remote config resolution is explicitly opt-in and requires an immutable 40-character Hub commit revision. This config-only loader has no weights-digest parameter; it is not the complete adapter-loading security boundary.
Full adapter loading through GPT-QModel distinguishes a caller's explicit adapter= from an adapter path embedded in a checkpoint:
| Input/control | Contract in adapter/remote.py and models/auto.py
|
|---|---|
allow_remote_adapter |
Defaults to False; remote loading requires explicit opt-in and an explicitly selected remote adapter. |
adapter_revision |
Requires exactly 40 hexadecimal characters identifying an immutable Hub commit, not a branch or tag. |
adapter_sha256 |
Requires exactly 64 hexadecimal characters for adapter_model.safetensors; downloaded weights are checked against this digest. |
| Explicit local adapter directory | Resolved locally; remote opt-in/revision/hash options are rejected for this input. A nonexistent absolute or explicitly relative local path is not reinterpreted as a remote repository. |
| Explicit remote reference | Supports Hub repository IDs, nested adapter subfolders, and supported HTTPS Hugging Face blob URLs. A URL's revision must match the caller's pinned commit. |
| Checkpoint-manifest adapter path | Must resolve to an existing directory inside the checkpoint. Absolute paths, Windows drive paths, .., backslashes, and symlinked directory components are rejected. Adapter-file symlinks are checked separately, with a narrow exception for the checkpoint's Hub-cache blob storage. |
These controls apply to GPT-QModel's adapter resolver, not to the separate external peft.PeftModel.from_pretrained API. Do not treat a checkpoint's embedded adapter path as authorization to fetch arbitrary external files. See EoRA for GPT-QModel adapter use.
The writer's EoRA path saves adapter_model.safetensors using PEFT-style lora_A.weight/lora_B.weight keys and writes a matching config (lora_alpha equals adapter rank). Dynamic adapter ranks become rank_pattern. See EoRA.
docs/peft.md records a recipe validated with PEFT 0.21.3.dev0, Optimum 2.3.0, Transformers 5.20.2, and PyTorch 2.14, using an 8B Llama 3.1 GPTQ checkpoint on four RTX 4090 GPUs. This is a recorded recipe, not a universal compatibility matrix.
Use PEFT's maintained examples/sft training script and set gptq_torch before constructing the model. The repository's docs/peft/fsdp_config.yaml specifies:
| Setting | Value |
|---|---|
distributed_type, num_machines, num_processes
|
FSDP, 1, 4
|
mixed_precision, downcast_bf16
|
bf16, no
|
| Sharding, state dict |
FULL_SHARD, SHARDED_STATE_DICT
|
| Auto-wrap, backward prefetch |
TRANSFORMER_BASED_WRAP, BACKWARD_PRE
|
fsdp_cpu_ram_efficient_loading, fsdp_sync_module_states
|
true, true
|
fsdp_use_orig_params, fsdp_offload_params, fsdp_forward_prefetch
|
false, false, false
|
Before training, install PEFT's FSDP wrapping policy:
from peft.utils.other import fsdp_auto_wrap_policy
if model_args.use_peft_lora and trainer.is_fsdp_enabled:
trainer.accelerator.state.fsdp_plugin.auto_wrap_policy = fsdp_auto_wrap_policy(trainer.model)With the versions recorded in the recipe, trainer.save_model() produced an empty adapter weights file. The documented workaround gathers a full state dict and writes normalized LoRA keys on the main process:
import os
from safetensors.torch import save_file
if trainer.is_fsdp_enabled:
trainer.accelerator.state.fsdp_plugin.set_state_dict_type("FULL_STATE_DICT")
state_dict = trainer.accelerator.get_state_dict(trainer.model)
trainer.save_model()
if trainer.is_world_process_zero():
adapter_state_dict = {
key.replace(".default.weight", ".weight"): value.cpu().contiguous()
for key, value in state_dict.items()
if key.endswith((".lora_A.default.weight", ".lora_B.default.weight"))
}
if not adapter_state_dict:
raise RuntimeError("FSDP full state dict contains no GPTQ LoRA tensors")
save_file(adapter_state_dict, os.path.join(training_args.output_dir, "adapter_model.safetensors"))
else:
trainer.save_model()The launch recipe uses BF16, max length 2048, batch size 1 per GPU, gradient accumulation 4, gradient checkpointing, and LoRA rank 8/alpha 16/dropout 0.1. Keep --use_4bit_quantization False for the already-GPTQ checkpoint: that option would enable a separate bitsandbytes QLoRA path. Reduce sequence length if memory is insufficient. Consult the source recipe for all SFT script arguments; they belong to that external example script, not GPT-QModel's quantization API.
PEFT cannot add dense LoRA deltas directly into packed GPTQ/AWQ weights. The project guide documents ValueError: Cannot merge LORA layers when the model is gptq quantized for GPTQ and NotImplementedError for AWQ. Keep adapters separate, or load the original dense base, merge there, save it, and quantize the merged model again. Do not relabel the original packed checkpoint as merged.
Reading the diagram: Independent concerns, not consecutive lanes. Forwarded Transformers push_to_hub is not GPTQModel's quantized writer; Hub hosting does not guarantee runtime compatibility.
Text version of the workflow
- Transformers loading: Import gptqmodel → Read checkpoint config → Choose method backend → AutoModel loading.
- Publish a local artifact: GPTQModel writer → Check artifact contents → Reload for verification → Upload with Hub client.
Use the compatibility sections when a load fails before execution; use the packing API only when you own the layer lifecycle. A copied API name does not imply identical kernel implementation or device support.
For integrators, gptqmodel.utils.hf_load exposes loading hooks (also re-exported from gptqmodel.utils.model) that preserve native checkpoint structure instead of replacing every linear with one homogeneous kernel:
| Hook | When and what it does |
|---|---|
hf_gptqmodel_prepare_model_for_load(model, checkpoint_files=None, device_map=None, backend=None, dtype=None) |
Before assigning checkpoint tensors: reads the local quantization manifest, rebuilds fused blocks/expanded MoE experts through Defuser, and installs kernels only for modules with serialized .qweight tensors. Honors the full dynamic configuration, including per-module bits/group sizes; rules that also name dense modules do not quantize those modules. Returns HFGPTQModelLoadContext, or None when the checkpoint should keep the caller's path. |
hf_gptqmodel_post_init_for_load(model, context, max_input_length=None) |
After tensor assignment: performs GPTQ v1-to-v2 conversion where needed, attaches the full quantization config, initializes kernels, restores rotation hooks, and applies the paged-attention CUDA-graph safety policy. |
is_native_gptqmodel_checkpoint(model, checkpoint_files=None) checks meta.quantizer provenance in quantize_config.json, quant_config.json, or config.json. GPTQModel provenance without Optimum provenance is native; a checkpoint naming Optimum keeps the Optimum path even if it also names GPTQModel. The prepare hook additionally recognizes a GPTQModel URI, dynamic settings, or GPTQ_V2 format when explicit provenance is absent. Missing manifests return None; native manifests with no .qweight tensors or unmatched module names raise actionable errors.
The bridge supports CPU/meta model shells and chooses a compatible kernel per serialized module. Saved W4A activation policy selects its corresponding backend; rotated GPTQ/GPTQ_V2 checkpoints default to GPTQ_TORCH and require a rotation-aware backend. _patch_optimum_gptqmodel_load_bridge() wraps GPTQQuantizer.convert_model and post_init_model when Optimum's GPTQ API is available. Installation is import-order independent: if optimum.gptq imports GPT-QModel before defining GPTQQuantizer, the install is deferred and retried from Transformers' GptqHfQuantizer pre/post weight-loading hooks and the entry points Optimum calls. Plain AutoModelForCausalLM.from_pretrained() therefore preserves native per-module dynamic widths without requiring an earlier import gptqmodel. Checkpoints owned by Optimum retain its original path when prepare returns None. Optimum remains optional; if the bridge is unavailable, use GPTQModel.load() for native mixed-bit or expanded per-expert checkpoints rather than accepting missing/unexpected expert tensors.
Native AWQ checkpoints use a second bridge: _patch_transformers_awq_gptqmodel_load_bridge() wraps AwqQuantizer._process_model_before_weight_loading and _process_model_after_weight_loading, without routing AWQ through Optimum. Installation is retried when the AWQ quantizer becomes available. If Transformers first imports gptqmodel while replacing layers, the post-load hook checks native AWQ modules against their serialized per-module bits, group size, activation order, and symmetry. It rebuilds mismatched modules around the loaded packed tensors before native initialization. This preserves dynamic settings instead of silently decoding every layer using the global group size or failing Marlin packing with a qzeros shape error.
Transformers 5.23 checks tensor shapes before the post-load hook runs. The bridge skips that check for qweight, qzeros, scales, g_idx, or bias on a native GPT-QModel AWQ quantized linear only when its live settings differ from the module settings resolved from the checkpoint. This lets the post-load hook rebuild a temporary module that was created from global settings instead of the saved per-module configuration. The comparison uses the effective group size: per-channel group_size=-1 is equivalent to the layer's in_features. Transformers still reports shape mismatches for modules that already match the saved settings or whose saved configuration cannot be resolved. Dense parameters and third-party AWQ checkpoints retain Transformers' normal behavior.
Before Transformers assigns native AWQ safetensors, the bridge reads the saved bias dtype from the checkpoint header. It widens the receiving slot if the model dtype would narrow a checkpoint-owned bias. Activation-side slots listed in MODEL_DTYPE_BUFFERS still follow the requested dtype. This prevents a load and save_pretrained() round trip from rounding a stored FP32 bias through BF16.
Rebuilt modules retain the loaded model's training/eval mode. A model already in eval therefore reaches the packed CPU INT4 path on its first eligible forward instead of selecting the training path. See the AWQ repair hook.
For fused/per-expert AWQ, import gptqmodel before AutoModelForCausalLM.from_pretrained(), or use GPTQModel.load(): expansion must happen before tensor assignment. If the first load missed preparation and Transformers dropped serialized packed modules, repair raises ValueError with those remedies rather than accepting randomly initialized experts. Non-native checkpoints and modules already matching their saved configuration retain Transformers' original path. Saving native AWQ with dynamic overrides also warns about loaders that bypass this bridge.
Both the Optimum GPTQ bridge and native AWQ bridge forward quantization_config["backend"] to kernel preparation. An explicit trainable backend such as gptq_torch or torch_awq is honored instead of silently becoming AUTO; an incompatible explicit choice raises during selection.
When the GPTQ bridge loads a gptq_v2 checkpoint, it marks the live qzeros as v2 without converting the already-v2 words. This prevents a subsequent save_pretrained() from incrementing the packed zero points as though they came from a v1 kernel. The runtime marker is kept in raw state dicts, not in the saved Transformers checkpoint. A v1 checkpoint still converts only when required by a loaded kernel.
The native GPTQ post-load bridge also copies model-wide rotation into the Transformers quantization config. A supported rotated checkpoint saved with Transformers' save_pretrained() retains the setting needed to rebuild its online Hadamard hooks on reload. This is not a model-wide setting for MACaQ's per-layer transform. Post-load bridge
Backend lookup first uses the quantizer's own non-None backend, then quantization_config.backend. The latter is essential for Transformers' AwqQuantizer, which does not have a direct backend attribute. Thus a backend stored only in an AwqConfig is honored too; an impossible request such as Marlin on CPU raises rather than silently falling back.
For GPTQ, RTN, and AWQ, the Transformers bridge applies the offload initialization check to CPU and disk placements under AUTO and AUTO_TRAINABLE. When Accelerate resolves a keyword map such as device_map="auto" from max_memory, the bridge rebuilds incompatible offloaded quantized layers with eligible kernels before assigning checkpoint tensors. GPU-resident layers can keep their original kernels. Disk-backed layers with offload-capable kernels temporarily materialize through their Accelerate hook for post_init() and return to meta afterward. See the bridge implementation and offloaded layer initialization.
An offload-safe kernel may still require a GPU for its forward pass. If the resolved map contains only CPU and disk, no accelerator executes those layers: AUTO and AUTO_TRAINABLE reselect kernels that cannot run on CPU, even if those kernels support offload to a GPU execution device.
This reselection also covers third-party AutoGPTQ and AutoAWQ checkpoints when the bridge can read their quantization metadata. Only the incompatible CPU/disk layers are replaced. Transformers/Optimum still owns the third-party load and post-initialization; the bridge does not treat these checkpoints as native GPT-QModel exports.
For third-party AWQ under AUTO, this works on the first from_pretrained() call even when Transformers imports GPT-QModel during that load. CPU and disk spill discovered after resolving device_map="auto" can still trigger reselection before weights are assigned. See AWQ first-load offload.
An explicit backend selection stays in force. On a mixed CUDA/CPU map, explicit GPTQ or AWQ Marlin can repack offloaded tensors when Accelerate provides a writable in-memory weights map. If initialization reports disk-backed entries it cannot rewrite, use offload_state_dict=False when the issue is the temporary state-dict spill, or select an offload-compatible backend for layers intentionally placed on disk. See CPU/offload dispatch.
On hosts with mixed GPU compute capabilities, an Accelerate keyword such as device_map="auto" is not a placement yet. The bridge initially selects a kernel compatible with all visible devices, then checks model.hf_device_map after Accelerate applies max_memory. A budget that places the model on one GPU capability group no longer fails the placement check merely because other GPU generations are visible; a resolved map that actually spans groups still raises. Use an explicit one-group map or restrict visible devices to select a kernel specific to that group. See Hugging Face interoperability.
Without that import/bridge, Transformers' built-in GPTQ/AWQ linear replacement does not rebuild fused 3-D experts from experts.<n>.*.qweight: a load can report missing/unexpected tensors, leave experts randomly initialized, and still return a model. GPT-QModel emits a save-time warning for that checkpoint layout; do not treat a completed load as proof of expert coverage. Use GPTQModel.load() if the bridge is inactive. See HF interoperability and the bridge implementation.
GPTQ exports with dynamic -: exclusions also record the exact quantized module set in modules_in_block_to_quantize, so plain Optimum loading does not allocate uninitialized quantized replacements for excluded dense modules. Quantized loading uses definition-specific key conversion when declared, otherwise the live model's Transformers conversion map; this remaps legacy VL model.layers.*/visual.* keys to current runtime trees and leaves already-current names unchanged.
GPT-QModel installs the following compatibility hooks during package initialization when their optional dependencies are available. Registration is lock-protected and repeat application is guarded.
def _patch_transformers_gptq_device_map_compat():
...
def _patch_transformers_gptqmodel_gguf_quantizer_compat():
...
def _patch_optimum_gptqmodel_load_bridge():
...| Patch | Exact behavior | Why it matters |
|---|---|---|
| GPTQ device map | Wraps GptqHfQuantizer._process_model_before_weight_loading. If device_map is a nonempty dict with exactly one unique value, and the model has no hf_device_map, assigns a copy of the map before invoking the original method. |
Preserves a single-device map for Optimum's later packing step. It does not invent a map for "auto", an empty dict, or a multi-value dict, and does not overwrite an existing attribute. The wrapper itself does not validate that shared value as a supported device. |
| Packed GGUF guard | Registers config/quantizer handlers for gptqmodel_gguf with setdefault. |
Rejects packed safetensors with an actionable load instruction instead of invoking Transformers' standalone .gguf quantizer. |
| CPTC loader | Registers config and quantizer handlers for cptc with setdefault. |
Loads existing CPTC safetensors checkpoints through AutoModelForCausalLM.from_pretrained() after importing gptqmodel. Use GPTQModel.quantize() to create the checkpoint. CPTC example
|
| Native GPTQ bridge | Wraps Optimum convert_model and post_init_model. |
Rebuilds the checkpoint's actual quantized module manifest, including expanded experts, before tensor assignment. |
The previous registration of Paro as a generic GPTQ quantizer was removed: it could enter an incompatible homogeneous loader. Use GPTQModel.load() with a compatible Paro backend; the rotations and auxiliary state are not ordinary GPTQ weights.
After importing gptqmodel, Transformers explicitly refuses GPT-QModel checkpoint markers bitsandbytes, exl3, macaq, paro (including legacy paroquant), qqq, nf12, tt-blockfloat, gptqmodel_gguf, and legacy gptqmodel_fp8. Load these through GPTQModel.load(). The EXL3 refusal specifically names the ExLlamaV3 backend, whose hardware restrictions still apply.
FP8 is load-only in GPT-QModel. After importing gptqmodel, Transformers also refuses quant_method="fp8" with weight_scale_semantics="inverse" before FineGrainedFP8 drops the marker and misinterprets the scales. Use GPTQModel.load() for these checkpoints. Standard multiplier-scale FineGrainedFP8 remains on the Transformers path. See FP8 loading.
The bitsandbytes guard distinguishes GPT-QModel's layout from genuine HF configurations sharing the same method name. A dictionary is refused only when neither load_in_4bit nor load_in_8bit is truthy; a config object with that method is refused only when it is not a real Transformers BitsAndBytesConfig. Genuine HF objects, flagged dictionaries, and bitsandbytes_4bit/bitsandbytes_8bit dispatch therefore remain available after importing GPT-QModel. Adding flags to a GPT-QModel checkpoint does not convert its tensors into HF's layout. Native block-float checkpoints carrying the shipped auto_map loader retain their dedicated loading path. See the layout discriminator and format-specific remedies.
Another optional import-time patch broadens OpenVINO's GPTQ layer-type recognition by replacing its supported_quant_types membership container. This compatibility change alone is not a guarantee that every GPT-QModel kernel or checkpoint can be exported to OpenVINO.
| Layer | Source | Responsibility |
|---|---|---|
| Public factory | gptqmodel/models/auto.py |
GPTQModel.load, from_pretrained, from_quantized, method/backend dispatch, GGUF input normalization |
| HF compatibility | gptqmodel/utils/hf.py |
Config dtype migration, native-vs-remote-code decisions, remote model compatibility, tokenizer loading, generation config sanitation, meta-device shells |
| Hub facade | gptqmodel/utils/hub.py |
Shared download, discovery, and repository helpers |
| Alternative model host |
gptqmodel/utils/modelscope.py, models/loader.py, __init__.py
|
Explicitly enabled ModelScope download and HF Hub patching |
| Transformers GPTQ quantizer | gptqmodel/__init__.py |
Compatibility around GptqHfQuantizer and registration of Paro |
| Native load bridge | gptqmodel/utils/hf_load.py |
Manifest-driven per-module replacement and post-load initialization for Transformers/Optimum integrators |
| Stable packing API |
gptqmodel/utils/packing.py, utils/importer.py, nn_modules/qlinear/__init__.py
|
External kernel selection, feasibility, packing, repacking, and initialization |
| Local Hub kernels | gptqmodel/hf_kernels/ |
Causal convolution and Mamba-compatible Torch ports |
| MiniMax implementation | gptqmodel/hf_minimax_m2/ |
HF config/model classes and historical class-name aliases |
| Adapter interchange |
gptqmodel/adapter/peft.py, adapter/remote.py, models/writer.py
|
PEFT-compatible LoRA config and EoRA serialization; explicit remote adapter resolution |
| Checkpoint writer | gptqmodel/models/writer.py |
Local packed safetensors, model/quantization config, tokenizer, processor, and remote-code files |
These signatures are from gptqmodel/utils/hf.py:
def get_hf_config_dtype(config: Any) -> Optional[torch.dtype]:
...
def set_hf_config_dtype(config: Any, dtype: torch.dtype) -> None:
...
def normalize_torch_dtype_kwarg(
kwargs: dict[str, Any],
*,
api_name: str,
explicit_dtype: Any = _MISSING,
) -> Any:
...
def normalize_hf_config_compat(config: Any, *, trust_remote_code: bool = False) -> None:
...get_hf_config_dtype prefers modern config.dtype; if absent it reads legacy torch_dtype. When modern dtype is None, it checks genuinely stored legacy metadata in __dict__ rather than reading a deprecated property. A resulting None is returned unchanged. It accepts torch.dtype values and resolves strings by literal getattr(torch, value); a missing attribute or an unsupported non-string value raises ValueError. This helper does not strip a "torch." prefix or check that a resolved attribute is itself a dtype. Use a real dtype such as torch.float16 or an unprefixed dtype name such as "float16".
normalize_torch_dtype_kwarg consumes deprecated torch_dtype, warns once, and normalizes to dtype. A missing, None, or "auto" modern dtype can be replaced by the legacy argument. Otherwise any unequal legacy value raises ValueError, including legacy None or "auto"; the function compares values directly rather than resolving dtype names first. With explicit_dtype supplied it returns the chosen value instead of inserting kwargs["dtype"]. Use dtype= in new code.
def resolve_trust_remote_code(model_id_or_path: Optional[str], *, trust_remote_code: bool) -> bool:
...
def has_native_transformers_causallm_support(model_id_or_path: Optional[str]) -> bool:
...
def ensure_qwen_drive_registered(model_id_or_path: Optional[str] = None) -> bool:
...The native-support probe loads AutoConfig with trust_remote_code=False and checks AutoModelForCausalLM._model_mapping[type(config)]. When native support exists, GPT-QModel reduces a caller's trust_remote_code=True to False and warns once per model/type/class tuple, unless the checkpoint declares a recognized remote-only layout. A differently spelled auto_map class name alone does not establish a different layout: native Transformers loaders can convert checkpoint key names. The current exception is model_type="bailing_hybrid" with a truthy q_lora_rank and auto_map.AutoModelForCausalLM; it preserves explicit trust for Ling-2.6-flash's remote MLA projections. Without that marker, natively supported checkpoints load through the native class. If native support cannot be established, the request remains enabled; False is never automatically elevated to True. See the trust resolver.
Qwen-Drive is deliberately not bundled. A local config with model_type="qwen_drive" triggers import of the official qwen_drive package before AutoConfig; missing support produces an installation hint pointing to https://github.com/QwenLM/Qwen-Drive-1.0. Unrelated model loads do not eagerly import that optional stack.
def load_hf_tokenizer(
tokenizer_or_path,
*,
model_config: Any = None,
trust_remote_code: bool = False,
**kwargs,
):
...
def load_tokenizer(tokenizer_or_path, *, model_config: Any = None, **kwargs):
...Both delegate construction and compatibility normalization to Tokenicer.load. load_hf_tokenizer returns the underlying native tokenizer (tokenicer.tokenizer), while deprecated load_tokenizer returns the Tokenicer object and emits DeprecationWarning. The runtime wrapper is applied by BaseQModel; these return types are not interchangeable.
These are implementation compatibility shims, not new public configuration knobs. They are applied conditionally according to installed APIs and model config.
| Area | Compatibility behavior in utils/hf.py
|
|---|---|
| Imports | Restores removed FX availability probe, legacy tokenizer enum/type imports, Flash Attention version probe, and check_model_inputs when missing. |
| RoPE | Restores an unscaled "default" RoPE initializer for legacy remote code that looks it up in ROPE_INIT_FUNCTIONS. A rotary module's own default initializer takes precedence during native Transformers weight initialization, preserving model-specific layouts such as ERNIE-4.5-VL's spatial rotary embeddings. Old rope_scaling/rope_theta values normalize into rope_parameters, including partial rotary factors and original position limits for applicable scaling types. |
| RoPE exceptions | HunYuan-VL normalizes only its text config; unscaled Nanbeige retains None legacy RoPE; GLM5-Next zero-rotary NoPE configs avoid synthetic RoPE when no genuine parameters exist. |
| Cache API | Aliases removed SlidingWindowCache/HybridCache to StaticCache when available, restores legacy max/usable-length access and tuple conversion on DynamicCache, and bridges custom generation-cache registry entries. |
| Tied weights | Converts list/tuple/set _tied_weights_keys into target-to-input-embedding mappings where embeddings really are tied; handles remote models missing all_tied_weights_keys; adapts old tie_weights(self) overrides to new keyword arguments. |
| Model helpers | Supplies missing is_parallelizable and get_head_mask defaults and bridges historical Flash Attention support flags. |
| Local code cache | Backfills missing top-level Python files from a local checkpoint into existing dynamic-module cache directories without overwriting existing files. |
| DeepSeek-VL2 | Before local config loading, restores old Llama attention/import interfaces, including old cache arguments and attention-output behavior. |
| ChatGLM | Bridges max_length to seq_length when needed and restores missing use_cache; removes the temporary max-length alias when saving. |
| Other remote configs | Restores missing Unlimited-OCR fields, LocateAnything text RoPE metadata, Ouro padding config, Brumby config compatibility, Instella legacy RoPE keys, and Hymba attention selection. |
| Vision-language init | Registers runtime Ovis visual config/model classes; adapts InternVL drop-path construction for meta init; bridges MiniCPM/Ovis Flash Attention flags and Ovis tokenizer backend recognition. |
| OpenVLA/Prismatic | Bridges removed AutoModelForVision2Seq mappings to AutoModelForImageTextToText, restores GenerationMixin where missing, and adapts remote-local timm version checks and single-layer list outputs. It does not alter timm's global version metadata. |
| LocateAnything attention | Strips unsupported allow_all_kernels from legacy overrides and keeps supported SDPA/magi paths; nested vision/text configs are kept on SDPA. |
| Hymba init | Avoids a hard-coded CUDA device during remote RoPE construction under meta initialization. |
| Hymba attention fallback | Normalizes the legacy flex setting before Transformers validates it. Hymba uses Flash Attention 2 when available, or SDPA otherwise. |
| Phi-4 multimodal | Handles specific scalar/positional-buffer meta-init cases, infers missing language/vision/speech input mode, supplies an inner generation-input hook for PEFT, and narrows AutoAWQ availability checks to awq.modules.linear. |
| Deci | Bridges mismatched remote config/model classes during AutoModel registration. |
When Ouro's legacy rotary class lacks compute_default_rope_parameters, GPT-QModel installs a bridge that uses ROPE_INIT_FUNCTIONS["default"]. The default stays available to that bridge during Transformers weight initialization, fixing the KeyError: 'default' seen when loading Ouro. See the Ouro load hook and RoPE compatibility wrapper.
For Transformers 5.22, Cache.get_max_length(layer_idx=0) is restored whenever that legacy method is missing, even when the cache-level get_max_cache_shape was also removed. It queries the selected layer's get_max_length() first, then its get_max_cache_shape(), and falls back to cache-level get_max_cache_shape(layer_idx) only when available. Missing queries, out-of-range layers, and negative/unbounded limits return None. This supports legacy remote decoders without assuming an obsolete cache-level API or recursing through older compatibility aliases. See the cache compatibility flow.
For a legacy remote generation hook such as Hymba's, GPT-QModel supplies an all-ones mask when the hook declares attention_mask and Transformers omits it. If the hook does not declare position_ids, the adapter drops that keyword so the model can build positions for its own memory tokens. Legacy caches with list/tuple key_cache storage and no layers report their length from that storage and report None for the maximum length. See legacy generation compatibility.
Both GPTQModel.load(...) and the Transformers quantized-load bridge run the same post-load repairs for affected remote-code models on Transformers 5. For models such as EXAONE-3.0 and DeciLM, the repair restores constructor-owned rotary buffers that can be left uninitialized when a checkpoint omits the buffers and the model overrides _init_weights. The loaders also install the legacy generation-cache adapter where needed. Even without a prepare context, the AWQ bridge runs these repairs and initializes native GPT-QModel AWQ checkpoints exactly once from their serialized config. This avoids a second Marlin repack of already-repacked weights. Third-party post-initialization still delegates to the Transformers hook. See the shared fix-ups and bridge post-init.
normalize_hf_config_compat always normalizes RoPE. Remote-code shims and legacy config aliases are used when remote code is trusted or a causal-LM remote auto_map exists. prepare_remote_code_compat(config) invokes that normalization with remote-code compatibility enabled. prepare_remote_model_init_compat pre-imports and patches the actual remote class before construction.
These shims do not make arbitrary remote code safe. Only enable it for code you trust. They are narrow compatibility paths, not a substitute for a supported model definition or for checking the installed Transformers version.
build_shell_model(loader, config, trust_remote_code=True, **model_init_kwargs) creates the architecture under accelerate.init_empty_weights(include_buffers=True). Its caller must provide device_map and _fast_init in model_init_kwargs, because the helper deletes these keys before from_config. Meta parameters preserve topology but do not contain usable weights. A narrow stale dynamic-code-cache failure can trigger one refresh/retry, including a known configuration-Llama shim.
suspend_hf_weight_init() temporarily disables selected Torch initializers and HF initialization, restoring global state in finally. It is an internal loading context, not a context to leave active during unrelated model creation.
Generic generation-config sanitation preserves the checkpoint's declared do_sample: an absent value stays unset, and explicit True or consistent False values survive loading and saving. The compatibility exception is an explicit False combined with contradictory sampling fields; load-time sanitation and the post-save file sanitizer repair that combination to True. See generation defaults for the checked fields. Specialized GenerationConfig subclasses are preserved by autofix_hf_model_config; rebuilding those as generic configs would lose model-specific fields.
When re-saving a loaded quantized checkpoint through a save-time shell, the writer deep-copies the live model's generation_config onto that shell before serializing generation_config.json. Sampling settings and generation-only EOS lists therefore survive instead of reverting to config.json defaults, while the live object is preserved. Ordinary saves and EXL3 use the live model directly. See the generation-config round-trip contract.
GGUF-over-HF loading is input normalization plus Transformers GGUF compatibility, not a claim that a GGUF file is GPTQ-packed safetensors.
from gptqmodel import GPTQModel
# An explicit local artifact disambiguates a directory with several GGUF files.
model = GPTQModel.load("/models/example/model-q4_0.gguf")
# Also accepted: a directory or Hub repo with one unambiguous GGUF artifact.
model = GPTQModel.load("organization/single-gguf-model")def normalize_model_id_or_path_for_hf_gguf(
model_id_or_path: Optional[str],
kwargs: dict[str, Any],
*,
api_name: str,
) -> Optional[str]:
...
def get_hf_gguf_load_kwargs(kwargs: dict[str, Any]) -> dict[str, str]:
...| Input or condition | Behavior |
|---|---|
Local .gguf file |
Normalizes to its absolute parent directory plus internal GGUF filename. |
| Local directory | Recursively inventories files; infers GGUF only if exactly one exists and no dense model files exist. |
| Hub repo ID | Uses list_repo_files(repo_id=...) and the same uniqueness rule. Discovery errors fall back to the original input. |
| Dense model files | Any .safetensors, .bin, .pt, .pth, or .ckpt prevents automatic single-GGUF inference, even alongside one GGUF file. |
| Multiple GGUF files | No automatic selection; use a local explicit file rather than expecting a quantization variant to be chosen. |
Public gguf_file=
|
A non-None value raises TypeError in GPT-QModel's normalizer. Put the artifact in model_id_or_path instead. |
| Internal handoff |
_gptqmodel_hf_gguf_file is translated to Transformers' gguf_file only at the internal HF boundary. Do not use this private key in application code. |
Artifact discovery is lru_cache-cached. A repository's contents changing within one process is not guaranteed to trigger rediscovery.
The compatibility path installs GPT-QModel's internal GGUF runtime and, if missing upstream, supplies the local Prism Q1_0_g128 symbols/dequantizer to Transformers. A separate optional direct-to-Torch GGUF loading path avoids a NumPy tensor staging path only when all of its guards hold: tensor loading requested, a target model supplied, the internal loader enabled, model_type Qwen3, and the ordinary TensorProcessor rather than an architecture-specific processor. Other cases delegate to the original loader; exceptions also fall back.
Shared load-policy field (RuntimeConfig / KernelConfig) |
Default and effect |
|---|---|
gguf_native_quantized_loader |
True; permit native packed-checkpoint detection before dense import. |
gguf_torch_dequant_loader |
None; follow whether gguf_dequant_device is set. Explicit False disables the direct Torch helper even with a device set. |
gguf_dequant_device |
None; keep the NumPy reference path by default. Accepts Torch cpu / cuda[:INDEX] execution targets, including Torch's ROCm spelling. Invalid targets are rejected; unavailable accelerators raise when resolved. |
gguf_dequant_chunk_bytes |
268435456; positive integer byte budget for a sign-only NumPy-facing conversion chunk, with at least one row. |
Pass these through runtime=RuntimeConfig(...) or legacy kernel=KernelConfig(...)/a field mapping. Typed GGUF backend configurations tune execution and do not expose the load-policy fields. Before dense GGUF import, the loader installs the call's resolved policy for the process-wide Transformers hook under the existing model-load lock. The four former GPTQMODEL_INTERNAL_GGUF_* switches are retired: they warn once per present name and have no effect. These controls concern checkpoint loading, not native MLX GGUF packing or inference-kernel selection. See GGUF load-policy examples, the source contract, and MLX (Apple Silicon).
gptqmodel/utils/hub.py exposes aliases rather than maintaining a separate cache or authentication implementation:
| Name | Provider |
|---|---|
cached_file, has_file
|
transformers.utils.hub |
create_repo, hf_hub_download, list_repo_tree, snapshot_download
|
huggingface_hub |
disable_progress_bar |
Transformers logging |
list_repo_files(*args, **kwargs), model_info(*args, **kwargs), repo_info(*args, **kwargs)
|
The existing bound Hub API client obtained from list_repo_tree.__self__
|
Use the Hub library's supported credentials, access controls, revisions, and cache options at its API boundary. The facade does not add an upload_folder wrapper or a GPT-QModel-specific authentication flow.
ModelScope is opt-in through GPTQMODEL_USE_MODELSCOPE; default is disabled. The shared environment parser accepts 1, true, yes, on, or y, case-insensitively and with surrounding whitespace removed.
pip install modelscope
GPTQMODEL_USE_MODELSCOPE=1 python quantize.pyensure_modelscope_available() returns False when not requested, returns True after a successful import when requested, and raises an actionable ModuleNotFoundError otherwise. models/loader.py chooses ModelScope's snapshot_download when enabled. Package initialization also calls modelscope.utils.hf_util.patcher.patch_hub(). Set the environment before importing GPT-QModel: this changes process-wide Hub integration, not just one model-call parameter. Failure while importing or patching ModelScope is surfaced as an installation error, so check the underlying exception if the package is already present.
External integrators should use the hf_ entry points documented in docs/hf_packing_api.md and tested in tests/test_hf_packing_api.py, not internal pack_module or select_quant_linear contracts.
def hf_check_packing_feasibility(
bits: int,
group_size: int = -1,
desc_act: bool = False,
sym: bool = True,
*,
format: Union[str, FORMAT] = FORMAT.GPTQ,
quant_method: Union[str, METHOD] = METHOD.GPTQ,
backend: Optional[Union[str, BACKEND]] = BACKEND.AUTO,
device=None,
dtype: Optional[torch.dtype] = None,
pack_dtype: Optional[torch.dtype] = None,
dynamic: Optional[Dict[str, Dict[str, Any]]] = None,
in_features: Optional[int] = None,
out_features: Optional[int] = None,
allow_marlin: bool = True,
is_sharded: bool = False,
repack: bool = False,
) -> bool:
...
def hf_check_best_packing_format(
bits: int,
group_size: int = -1,
desc_act: bool = False,
sym: bool = True,
*,
quant_method: Union[str, METHOD] = METHOD.GPTQ,
device=None,
dtype: Optional[torch.dtype] = None,
pack_dtype: Optional[torch.dtype] = None,
dynamic: Optional[Dict[str, Dict[str, Any]]] = None,
in_features: Optional[int] = None,
out_features: Optional[int] = None,
formats: Optional[Sequence[Union[str, FORMAT]]] = None,
allow_marlin: bool = True,
) -> FORMAT:
...Feasibility returns False for unsupported contracts, but argument normalization happens before that probe: unknown format/method/backend names raise ValueError, and a backend argument with an unsupported type raises TypeError. Supply shape arguments to check actual layer divisibility. Use repack=True for kernels that accept an already-packed checkpoint and convert in post_init() rather than implementing raw-weight packing; Marlin and ExllamaV2 are examples. Best-format selection ranks available kernels by declared priority and deterministically breaks format-name ties; it is not a runtime microbenchmark of your model.
The Transformers-facing selector also has an explicit stable signature:
def hf_select_quant_linear_v2(
bits: int,
group_size: int,
desc_act: bool,
sym: bool,
format: Union[str, FORMAT],
quant_method: Union[str, METHOD],
zero_point: Optional[bool] = None,
dtype: Optional[Union[str, torch.dtype]] = None,
meta: Optional[Dict[str, any]] = None,
pack: Optional[bool] = True,
device_map: Optional[Union[str, dict]] = None,
backend: Optional[Union[str, BACKEND]] = None,
) -> Type[BaseQuantLinear]:
...It normalizes backend aliases and method/format strings, accepts dtype strings with or without torch., and uses meta["pack_dtype"] when supplied. Packing storage otherwise defaults to int32, except AWQ GEMV_FAST defaults to int16. A format that normalizes to FORMAT.LLM_AWQ (for example, "llm-awq" or "LLM-AWQ") forces torch.int16 even if meta["pack_dtype"] requests a different dtype. zero_point expresses AWQ asymmetry and overrides sym as not bool(zero_point). With no device_map, selection starts from CPU. AWQ version names must already be mapped to GPT-QModel's format/quant_method concepts by the caller.
def hf_pack_layer(
module: BaseQuantLinear,
linear: nn.Module,
scales: torch.Tensor,
zeros: Optional[torch.Tensor] = None,
g_idx: Optional[torch.Tensor] = None,
*,
pack_impl: Optional[str] = None,
device=None,
block_in: int = 8192,
workers: int = 1,
scales_extra: Optional[torch.Tensor] = None,
post_init: bool = False,
post_init_kwargs: Optional[Dict[str, Any]] = None,
checkpoint_format: Optional[Union[str, FORMAT]] = None,
) -> BaseQuantLinear:
...
def hf_post_init(module: nn.Module, **post_init_kwargs) -> nn.Module:
...
def hf_repack_layer(
module: BaseQuantLinear,
backend: Optional[Union[str, BACKEND]] = None,
*,
device=None,
dtype: Optional[torch.dtype] = None,
post_init: bool = True,
post_init_kwargs: Optional[Dict[str, Any]] = None,
) -> BaseQuantLinear:
...| Contract | Details |
|---|---|
| Quantizer scale/zero layout |
[out_features, num_groups], not the transposed runtime-kernel layout |
| Missing zero points | Symmetric modules synthesize 2 ** (bits - 1); asymmetric modules require explicit zeros
|
| Activation ordering |
g_idx has length in_features
|
| Packing implementation |
None resolves to original packing; explicit values include original, auto, block/cpu, and gpu. Unavailable accelerated/block paths fall back to original packing; a selected GPU/block packer raising ValueError or NotImplementedError also falls back. Other exceptions are not caught by this fallback. Unknown implementation names warn and select original packing. |
| QQQ/AWQ | Their single pack() paths ignore GPTQ pack_impl; QQQ consumes scales_extra
|
| Storage device | Packed buffers are registered on CPU even when GPU accelerates the packing math; move to the runtime device afterward |
| Return | Packing modifies and returns the same module; module.hf_pack(...) delegates to this function |
| GPTQ zero-point versions | Raw packers always write logical (v2) zero points; the module's format flag is reset to 2 before deciding whether a conversion applies. A checkpoint_format="gptq" (v1) request converts to the zp-1 convention only for kernels declaring REQUIRES_FORMAT_V2 — other kernels keep the raw packer words for both checkpoint flavors, matching pack_module. When post_init runs, such a v1-requested module is converted back to v2 so the live module stays usable; the saved tensors still carry the requested v1 flavor |
| Post-initialization |
hf_post_init handles a single layer or quantized children of an nn.Module; it is not a replacement for all model-level scratch/activation-order initialization |
| Repacking | Uses native repack_from_gptq/repack_from_awq hooks where present, otherwise copies compatible checkpoint tensors and calls target post_init()
|
| Same target | Returns source unchanged when the requested backend is already in use |
| Private runtime ABI | Marlin/Machete/Swordfish/BitBLAS-style ABI-transforming sources are rejected for copy repacking; repack from a checkpoint-layout kernel instead |
| Errors | Invalid source type raises TypeError; unsupported layout conversion raises NotImplementedError; unresolved target/backend raises ValueError
|
Example with an existing dense linear, quantizer-native scales, and optional g_idx:
from gptqmodel.utils.importer import hf_select_quant_linear_v2
from gptqmodel.utils.packing import hf_pack_layer
kernel_cls = hf_select_quant_linear_v2(
bits=4, group_size=128, desc_act=False, sym=True,
format="gptq_v2", quant_method="gptq", backend="auto", device_map="cpu",
)
module = kernel_cls(
bits=4, group_size=128, desc_act=False, sym=True,
in_features=linear.in_features, out_features=linear.out_features,
bias=linear.bias is not None,
)
hf_pack_layer(
module, linear, scales, zeros=None, g_idx=g_idx,
pack_impl="original", checkpoint_format="gptq_v2",
)For ExllamaV2 repacking, defer initialization until scratch is available:
from gptqmodel.utils.exllamav2 import ScratchSpace
from gptqmodel.utils.packing import hf_post_init, hf_repack_layer
target = hf_repack_layer(source, backend="gptq_exllama_v2", device="cuda:1", post_init=False)
hf_post_init(target, scratch_space=ScratchSpace(target.temp_dq_size(), dev="cuda:1"))_patch_transformers_causal_conv1d_hub_kernel_compat wraps Transformers' hub_kernels.lazy_load_kernel and exported transformers.integrations.lazy_load_kernel if available. Hyphens in kernel names become underscores. Resolution checks gptqmodel.hf_kernels.<name> first, then a project-root module, and falls back to Transformers' original loader if neither exists. The resolved module is stored in Transformers' kernel mapping; causal-conv1d additionally gets a legacy sys.modules["causal_conv1d"] alias without overwriting an existing one.
These are local Torch implementations. The ops/triton/ directory name in the Mamba port does not mean these files contain Triton JIT kernels.
| Package/API | Behavior and limitations |
|---|---|
causal_conv1d (__version__="0.0.local") |
Exports causal_conv1d_fn, causal_conv1d_update, causal_conv1d_varlen_states; supports activation None, silu, swish only. |
causal_conv1d_fn |
Input [batch, dim, seqlen], weights [dim, width]; casts compute input to weight dtype and returns input dtype. Supports initial/final states; sequence-index masking cannot be combined with initial states or returning final states. |
causal_conv1d_update |
Accepts a token or sequence, mutates the supplied convolution state, supports selected state rows and circular-buffer cache_seqlens. |
causal_conv1d_varlen_states |
Extracts right-aligned sequence tails from [total_tokens, dim] and cumulative lengths into [batch, dim, state_len]; documented forward-only. |
mamba_ssm (__version__="2.2.4-local") |
Exports selective scan, inner function, falcon_mamba_inner_fn alias, state update, chunk scan, and split convolution/scan interfaces. Uses Torch and einops. |
selective_scan_fn |
Delegates to a reference recurrence; supports variable/grouped B/C, optional D, gate z, delta bias/softplus, complex A, and optional last state. |
selective_state_update |
Mutates selected state rows; checks grouped-head and tensor-shape contracts and returns activation dtype. |
mamba_chunk_scan_combined |
Reference per-token recurrence, optional sequence-index state resets, FP32 state for FP16/BF16 activations, optional final state. Accepts but discards chunk_size, cu_seqlens, and return_varlen_states; do not assume optimized chunking or a returned varlen-state structure. |
mamba_split_conv1d_scan_combined |
Splits projection payloads, runs local convolution and scan, optional gated/grouped RMS normalization and output projection. |
These ports prioritize compatible interfaces. Their sequential Torch loops do not promise the throughput or full specialized-kernel behavior of external fused packages.
gptqmodel/hf_minimax_m2/ exports MiniMaxM2Config, MiniMaxM2PreTrainedModel, MiniMaxM2Model, and MiniMaxM2ForCausalLM, plus legacy MiniMax* aliases/classes. The config's model_type is "minimax", not inferred from the package directory name. Importing the package exposes classes; its __init__.py does not itself register them with AutoConfig or AutoModelForCausalLM.
Selected constructor defaults:
| Parameter | Default |
|---|---|
vocab_size, hidden_size
|
200064, 3072
|
intermediate_size, mlp_intermediate_size
|
1536, 8192
|
num_hidden_layers |
62 |
num_attention_heads, num_key_value_heads, head_dim
|
48, 8, 128
|
num_local_experts, num_experts_per_tok
|
256, 8
|
max_position_embeddings, rope_theta, rotary_dim
|
196608, 5000000.0, 64
|
use_qk_norm, qk_norm_type
|
True, "per_layer"
|
scoring_func, use_routing_bias, use_grouped_topk
|
"sigmoid", True, True
|
router_aux_loss_coef, router_jitter_noise, output_router_logits
|
0.001, 0.0, False
|
attn_window_size, sliding_window, swa_rope_theta
|
None, None, -1.0
|
use_mtp, num_mtp_modules, mtp_transformer_layers
|
True, 3, 1 (serialized config fields, not an implemented MTP generation promise) |
rms_norm_eps, attention_dropout, tie_word_embeddings, use_cache
|
1e-6, 0.0, False, True
|
The model implements partial rotary attention, optional Q/K normalization, grouped-query heads, sparse expert routing, optional router-logit/auxiliary-loss outputs, and HF Cache/DynamicCache generation. Attention is explicitly chunked per head in 1024-token query chunks, and expert execution is chunked at 1024 routed tokens. The model declares Flash Attention, SDPA, and attention-backend support flags false; it is not a Flash Attention implementation merely because it follows HF interfaces.
load_state_dict(self, state_dict, strict=True) filters FP8-related scale metadata (weight_scale_inv, weight_scale, input_scale, scales, amax, and keys containing fp8) and splits fused qkv_proj.weight into Q/K/V projection weights. Filtering scale metadata is not FP8 dequantization: use a correctly converted dense checkpoint for this implementation rather than expecting the filter to reconstruct FP8 payloads.
The bundled test_minimax_m2_hf.py is a manual streaming checkpoint smoke script using a chat template and BF16 auto placement. Its structural assertions are commented out; it is not proof that all MiniMax config features execute or that quantized loading is numerically validated.
- Import GPT-QModel before relying on its process-local Transformers monkeypatches. Availability checks are defensive and may skip patches for missing/incompatible imports.
- Do not equate method (
gptq,awq,paro), checkpoint format, and execution backend. These are separate contracts. - A single concrete device map is not the same as
device_map="auto"; the compatibility patch only backfills the former. - Prefer modern
dtype=and avoid conflicting legacytorch_dtype=arguments. - Tokenicer wrappers and native HF tokenizers have different return contracts.
- GGUF auto-discovery will not choose among quantization variants or mixed dense/GGUF artifacts.
- A local Mamba compatibility port may intentionally ignore optimized-kernel options. Check its implementation before depending on varlen-state return behavior or throughput.
- Do not train through inference-only quantized kernels or merge LoRA directly into packed weights.
- Check nonempty adapter weights after FSDP saving, and record the actual library versions used.
- Do not use forwarded HF
push_to_hubas a substitute for GPT-QModel's checkpoint writer.
gptqmodel/utils/hf.py-
gptqmodel/utils/hf_load.py,gptqmodel/utils/model.py— native checkpoint prepare/post-init hooks. -
tests/test_hf_gptqmodel_load_bridge.py,tests/test_integration.py— bridge contracts and integration coverage. gptqmodel/utils/hub.py-
gptqmodel/utils/modelscope.py,gptqmodel/utils/env.py gptqmodel/__init__.py-
gptqmodel/models/auto.py,gptqmodel/models/loader.py,gptqmodel/models/base.py,gptqmodel/models/writer.py -
gptqmodel/utils/importer.py,gptqmodel/utils/packing.py,gptqmodel/nn_modules/qlinear/__init__.py -
gptqmodel/hf_kernels/causal_conv1d/__init__.py,causal_conv1d_interface.py,causal_conv1d_varlen.py -
gptqmodel/hf_kernels/mamba_ssm/__init__.py,ops/selective_scan_interface.py,ops/triton/selective_state_update.py,ops/triton/ssd_combined.py -
gptqmodel/hf_minimax_m2/__init__.py,configuration_minimax_m2.py,modeling_minimax_m2.py,test_minimax_m2_hf.py -
gptqmodel/adapter/peft.py,gptqmodel/adapter/remote.py docs/hf_packing_api.md-
docs/peft.md,docs/peft/fsdp_config.yaml tests/test_hf_packing_api.py-
pyproject.toml,requirements.txt
Documentation home · Quickstart · Troubleshooting · Repository · PyPI
Public source d4712ee13 declares package version 7.12.4. Individual pages identify their source revisions. Choose a method · Dtypes and formats. Maintained by ModelCloud.ai.
🧭 1 · Choose and understand
🚀 2 · Use it
- Installation and Quickstart
- Loading and Inference
- Evaluation
- Serving
- Troubleshooting and FAQ
- Memory, Offload, Multi-GPU
- MLX / Apple Silicon
- Tenstorrent / Block Floats
- Hugging Face Integration
- Multimodal and MoE Models
🎛️ Methods and representations
Each guide: choose → examples → math / advanced
- GPTQ / GPTAQ / RTN
- AWQ
- QQQ
- MACaQ / MXFP4
- EXL3
- CPTC
- VPTQ
- Paro
- FP8 and NVFP4
- NF12: lossless BF16 storage
- GGUF families
- BitsAndBytes: NF4 / FP4 / INT8
🧰 Refinement and execution
🧠 3 · Theory and advanced reference