Multimodal and MoE Models

Multimodal and MoE Models

Documentation home · Quickstart · Lifecycle

On this page

• Overview
• Workflow at a glance
• Multimodal concepts
• Architecture coverage and boundaries
• Image and audio utilities
• MiMo mixed checkpoint sources
• MoE concepts and module handling
• MoE configuration and examples
• MoE lifecycle internals
• Multimodal usage example
• Pitfalls and troubleshooting
• Source files
• See also

Overview

GPT-QModel supports architectures whose language model is embedded inside a larger vision/audio model, architectures whose feed-forward blocks contain routed experts, and models combining both. These are separate dimensions: a vision-language model need not be MoE, and a MoE model need not be multimodal.

The model definition—not the presence of a vision encoder, audio tower, or Linear module—determines which weights are quantized. In the VL/Omni definitions below, the ordinary decoder quantization tree targets language/thinker layers. Towers, projectors, speech generators, routers, and normalization modules may still execute during calibration while remaining outside that plan.

This page is verified against revision eb90dbdd5416c0c8e50d1b64f1ac562012afe4aa. HF layouts, Defuser conversions, and remote model implementations are version-sensitive. See Model Definitions and Module Tree for the definition contract, parser flags, registry, and loading/saving internals.

Workflow at a glance

Reading the diagram: The lanes are independent calibration concerns, not a required combination. Media support varies by definition; temporary routing changes do not set inference policy.

Text version of the workflow

• Multimodal input capture: Select supported inputs → Prepare model inputs → Capture language inputs → Apply decoder plan.
• Optional MoE coverage: Choose routing policy → Collect expert inputs → Restore normal routing.

Open full-size SVG

Multimodal concepts

Quantized decoder versus dense prerequisites

There are three distinct roles:

Role	Examples	What happens
Quantization targets	Decoder Q/K/V/output projections; dense MLP or individual expert projections.	Enumerated by the definition tree, filtered by dynamic exclusions and architecture hooks, then replaced with quantized modules.
Decoder passthrough/capture entries	Norms marked :!, router gates marked :!, parent MoE blocks marked :?.	Participate in execution/capture/scaling but are excluded from ordinary quantized weight replacement.
Outside the decoder tree	Vision/audio towers, multimodal projector, visual token table, talker, waveform decoder.	Used/materialized when needed; not quantized by the listed language-layer plan. Some adapters explicitly add them to base-module discovery.

Quantizing lm_head and embeddings has separate configuration/implementation paths. Do not infer they are quantized merely because they belong to the language model. Likewise, modality is metadata: its enum in utils/model.py contains TEXT="text", IMAGE_TO_TEXT="image_to_text", and VIDEO_TO_TEXT="video_to_text", but no audio enum in this revision. An audio-capable definition may inherit the text modality default.

Calibration must preserve the modality path

Multimodal calibration converts conversations into both tokens and media tensors, runs towers/projectors to obtain media features, merges them into the language input, and captures inputs at the first decoder layer. Subsequent quantization can replay decoder layers directly using captured hidden states and kwargs. The tower need not be quantized to affect decoder calibration.

batched_conversations() first applies normalize_chat_calibration_sample() and then an adapter's preprocess_dataset(). Dataset preparation is adapter-specific: some use AutoProcessor.apply_chat_template(..., tokenize=True), some separately format text and call a processor, and Ovis versions use model-specific preprocessing. Do not feed arbitrary token dictionaries into every multimodal override and expect extra vision fields to survive.

Architecture coverage and boundaries

The paths below are runtime layer-container paths in this source, not checkpoint key prefixes. Reversed HF conversion rules or explicit path resolvers may accept older layouts.

Definition / registered family	Language-layer container	Capture prerequisites and exclusions
Qwen2VLQModel, Qwen2_5_VLQModel	model.language_model.layers or language_model.layers via BaseQwen2VLGPTQ.extract_layers_node()	Dense visual tower, embeddings, and rotary modules are materialized for multimodal input capture; no visual-layer quantization tree.
Qwen3_VLQModel, Qwen3_VL_MoeQModel	model.language_model.layers or language_model.layers via BaseQwen3VLGPTQ.extract_layers_node()	Base discovery includes the visual tower. Dense tree skips Q/K norms; MoE tree additionally skips router and expands defused experts.
Qwen3_5QModel, Qwen3_5_MoeQModel	model.language_model.layers; MoE also inherits the Qwen3-VL layout resolver	Hybrid self-attention/linear-attention decoder trees; dense and MoE classes have different preprocessing inheritance. MoE requires processor-produced mm_token_type_ids for multimodal batches. Both preserve auxiliary mtp tensor prefixes.
OvisQModel (ovis)	llm.model.layers	Original Ovis uses visual_tokenizer, vte, a generation-based capture path, and model-specific conversation preprocessing.
Ovis2QModel (ovis2)	model.language_model.layers	AutoModelForImageTextToText; dense model.vision_tower and model.visual_embeddings_table.
Ovis2_5QModel (ovis2_5)	llm.model.layers	AutoModelForCausalLM; dense visual_tokenizer and vte; calls model-specific preprocess_inputs().
Ovis2_6_MoeQModel, Ovis2_6_NextQModel	llm.model.layers	Inherit Ovis2.5 preprocessing; quantize MoE decoder experts, not vision tokenizer. Next adds optional hybrid linear-attention/shared-expert paths.
InternVLQModel (internvl)	model.language_model.layers	Native HF image-text loader/processor; dense model.vision_tower and model.multi_modal_projector; batch quantization disabled.
InternVLChatQModel (internvl_chat)	language_model.model.layers or language_model.layers	Remote-code AutoModel; dense vision_model and mlp1, explicit image patch/token insertion; batch quantization disabled.
MLlamaQModel (mllama)	model.language_model.layers or language_model.model.layers	AutoModelForPreTraining; entire MllamaCrossAttentionDecoderLayer layers remain dense. Vision model/projector are also outside the decoder plan.
MLlamaTextQModel (mllama_text_model)	model.layers	CausalLM text layout; retains architecture skip hooks.
Qwen2AudioGPTQ (qwen2_audio)	model.language_model.layers	AutoModelForPreTraining; dense model.audio_tower, model.multi_modal_projector, embeddings, and rotary modules.
Qwen2_5_OmniGPTQ (qwen2_5_omni)	thinker.model.layers or thinker.model.language_model.layers	AutoModelForTextToWaveform; quantization/capture targets thinker; visual/audio towers, optional talker/token2wav, and speaker data are not a speech-generator quantization plan.
Qwen3OmniMoeGPTQ (qwen3_omni_moe)	thinker.model.layers or thinker.model.language_model.layers	Text-to-waveform loader; decoder MoE experts quantized, router/QK norms skipped; visual/audio towers and optional speech generators materialized outside that plan.
Gemma3ForConditionalGenerationGPTQ	model.language_model.layers	Vision-language decoder tree, processor-based calibration; batch quantization disabled. gemma3_text instead maps to a Llama-derived text definition.
Gemma3nForConditionalGenerationGPTQ	model.language_model.layers	Custom capture/replay handles additional per-layer inputs; non-strict module matching and batch size one. gemma3n_text uses model.layers.
NemotronOmniQModel (nemotron_omni)	language_model.backbone.layers	Hybrid text-backbone plan; base discovery adds top-level vision_model, mlp1, sound_encoder, sound_projection. Text-only capture/forward delegates to the language backbone.
MimoV2QModel (mimo_v2)	model.layers	Supports mixed FP8/MXFP4 checkpoint decoding, defused experts, inherited model-forward capture and model-specific replay kwargs; preserves model.mtp tensors outside the decoder plan. Its source-decoding utilities are not generic audio preprocessing.

Both Qwen Omni definitions declare their decoder tree as two complete variants because transformers 5.22 moved the thinker decoder from thinker.model to thinker.model.language_model and the visual/audio encoders from thinker to thinker.model. The framework resolves whichever variant exists and de-duplicates the result, so the same definition covers releases before and after the move; thinker_text_model()/thinker_multimodal_model()/thinker_text_prefix() in utils/model.py resolve the moved parents for the materialize/offload hooks and the pre-lm-head norm.

For the complete model-type/class mapping, see the registry table in Model Definitions and Module Tree. An architecture registration alone does not establish support for every multimodal calibration path or inference backend.

Qwen-VL

Qwen2VLQModel and Qwen2_5_VLQModel are thin subclasses of BaseQwen2VLGPTQ. Both use AutoModelForImageTextToText, require a processor, and declare text/image-to-text modalities. Capture locates the multimodal core, embeds text, computes visual features, replaces image/video placeholder embeddings, prepares multimodal positions/rotary inputs, and calls the first language layer. This avoids going through an incompatible full wrapper during lazy first-layer capture.

The Qwen2/2.5 helper named process_vision_info() is image-only in this implementation: it loads image/image_url items and rejects other extracted entries. Although capture handles already-present video fields, that helper does not fetch videos for conversation calibration. Do not assume that a model's video capability means its calibration adapter supports raw video conversations.

BaseQwen3VLGPTQ adds explicit conversation/processor validation, contextual batch/sample/media errors, and lazy video loading through qwen_vl_utils.vision_process.fetch_video. The helper's optional arguments default to return_video_kwargs=False, return_video_metadata=False, and image_patch_size=14.

Helper call	Actual return contract
Default flags, no videos	Image list or None.
Videos present, or return_video_metadata=True	(images, videos); either component can be None.
return_video_kwargs=True	(images, videos, video_kwargs); this branch takes precedence. The kwargs contain do_sample_frames=False and, unless metadata is requested, fps with the sampled FPS list.

prepare_dataset() calls the helper with default flags and sends the resulting images/videos to the processor; it does not request or forward the optional video_kwargs dictionary. Do not assume this adapter forwards every sampling option supported by qwen_vl_utils.

Its prepare_dataset(..., batch_size=1, **kwargs) accepts an iterable, rejects a single string/mapping and sized empty datasets, and retains the base text path only when every sample is recognized as pretokenized text: tensors, integer token sequences, or dictionaries containing input_ids with no keys beyond input_ids/attention_mask. Other samples take the conversation/processor path. This is not a promise that arbitrary precomputed processor outputs with vision keys are accepted as ready-to-replay examples.

For Qwen3-VL MoE, Defuser exposes fused gate/up expert storage as numbered gate_proj, up_proj, and down_proj modules before traversal. num_experts controls expansion; the router has :!, and GateUpDownMoELifecycleHooks supplies all-expert calibration when requested.

Qwen3.5: hybrid trees and different preprocessing inheritance

The qwen3_5, qwen3_5_text, qwen3_5_moe, qwen3_5_moe_text, and zdtaichu5_0 registrations are enabled only when auto.py observes Transformers >=5.2.0. This is the registry gate, not the current package minimum: this source revision's requirements.txt requires Transformers >=5.20.2.

Qwen3_5QModel uses AutoModelForImageTextToText and requests a processor, but inherits from LlamaQModel, not BaseQwen3VLGPTQ; it does not inherit that base's conversation/video dataset preparation. Qwen3_5_MoeQModel does inherit BaseQwen3VLGPTQ and sets require_mm_token_type_ids=True. Its processor validation requires pixel_values/image_grid_thw for image batches, pixel_values_videos/video_grid_thw for video batches, and mm_token_type_ids whenever either modality is present. Loading a processor alone does not make the two adapters' calibration contracts interchangeable.

Both trees include optional self_attn and linear_attn paths with non-strict matching. Linear-attention norm, conv1d, in_proj_b, and in_proj_a stay dense; in_proj_qkv, in_proj_z, and out_proj are candidates for quantization. The MoE tree also skips the router/shared-expert gate and orders shared-expert projections before routed-expert projections. Both classes preserve auxiliary checkpoint tensors under mtp using out_of_model_tensors.

Ovis versions are not interchangeable

Adapter	Distinct implementation behavior
Original ovis.py	Non-strict layer matching; a monkey patch casts visual_tokenizer and vte to the language model dtype and sets language generation max_length=8192; model-specific text/image preprocessing and labels, specialized capture movement, and generate() to reach the first decoder layer.
ovis2.py	Formats conversations with a processor chat template, loads images using utils/image.py, and calls the processor with images=..., videos=None.
ovis2_5.py	Replaces image URLs with PIL images through the shared load_image_from_url() helper (see the image section below); calls self.model.preprocess_inputs(messages=..., add_generation_prompt=True), constructs attention masks from the text tokenizer, casts pixels to the visual tokenizer dtype, and retains grid_thws. It is not a universal local-path/PIL loader like fetch_image().
ovis2_6_moe.py	Declares defuser_module_paths=("llm",), expert counts and lifecycle hooks; restores checkpoint-omitted SigLIP2 post-layernorm weight/bias to their default ones/zeros before parent materialization.
Ovis2.6 Next	Optional self-attention/linear-attention paths; skips the linear-attention norm, convolution, and in_proj_ba; includes routed and shared experts with non-strict matching and a method-specific AWQ tree override.

Ovis2.5 also reconstructs VisionRotaryEmbedding explicitly using the vision configuration because generic nonpersistent-buffer reconstruction cannot rebuild it correctly. These are architecture-specific materialization rules, not reasons to quantize the visual tokenizer.

InternVL: native versus chat remote code

Native InternVLQModel uses processor.apply_chat_template(..., add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt"); its reversed HF mapping translates runtime model.language_model.*, vision tower, projector, and head names to released checkpoint names.

InternVLChatQModel requires trusted remote code and does its own image tiling/normalization, conversation-history formatting, and <img>, </img>, <IMG_CONTEXT> expansion. Its _dynamic_preprocess() defaults are min_num=1, max_num=12, image_size=448, use_thumbnail=False; callers select actual sizes/options from the model. run_input_capture() sets img_context_token_id, builds a generation configuration with max_new_tokens=1 and do_sample=False, and calls the remote wrapper's generate(pixel_values=..., input_ids=..., attention_mask=..., generation_config=...) to reach the decoder. It does not implement a direct local feature-merge/inputs_embeds forward. Both InternVL definitions set support_batch_quantize=False; increasing the public batch argument does not enable batching there.

Mllama: a deliberate cross-attention exclusion

The Mllama definition states that cross-attention decoder quantization is unsupported. should_quantize_layer() detects MllamaCrossAttentionDecoderLayer by class name and returns False. should_quantize_module() resolves a candidate module's owning decoder layer and applies the same decision. Thus cross-attention weights remain dense during quantization and quantized checkpoint reconstruction/re-saving; the rest of such a layer is not selectively quantized by the ordinary self-attention tree.

First-layer capture materializes language embeddings/rotary state, constructs language hidden states/positions, and directly calls layer zero, avoiding the multimodal wrapper's mask-building path over meta tensors. This capture implementation is a text-language path; do not infer full vision-conditioned cross-attention calibration support from the model loader's ability to construct the multimodal architecture.

Audio and Omni

Qwen2AudioGPTQ.prepare_dataset() and Qwen2.5 Omni both call process_audio_info(..., use_audio_in_video=False), so their conversation preparation does not automatically extract a video's audio. They format text with the processor and pass decoded waveforms via audio=...; Qwen2.5 Omni additionally passes images, but videos=None.

Qwen2.5 Omni captures and exposes textual forward() through thinker, not the whole speech-generation wrapper. It fixes missing HF embedding getters before loading, uses integer attention masks, and conditionally materializes talker/token2wav. Its capture-start hook unconditionally calls load_speakers() with <model_local_path>/spk_dict.pt; even if optional speech modules are absent, this adapter does not skip the speaker-file load. Preserve that asset and the remote/model-specific loader's requirements.

Qwen3 Omni MoE sets INPUT_EMBEDDING_EXTRA_ARGS={"return_audio": False}, uses integer masks, and declares its language MoE tree under thinker.model.layers (with thinker.model.language_model.layers as the transformers >=5.22 variant). The inherited capture method therefore calls the full wrapper's generate(..., return_audio=False), not Qwen2.5's direct thinker call. It does not inherit Qwen2.5 Omni's conversation/audio dataset override: its class derives directly from BaseQModel. Its capture-start hook loads spk_dict.pt only if the file exists and handles optional talker/code2wav modules. Its after_model_load() retains an AutoProcessor for unquantized models so save includes processor/chat-template files. Use the actual adapter's preparation contract rather than copying a Qwen2.5 audio example unchanged.

Gemma3n likewise has specialized first-layer positional/per-layer-input capture and replay, but neither Gemma3n class overrides prepare_dataset(). Its multimodal class requests a processor without inheriting Gemma3's conversation adapter. Treat processor loading, supported calibration sample formats, and replay support as separate capabilities.

Gemma 3 checkpoint save/load compatibility

Gemma3ForConditionalGenerationGPTQ ships dedicated checkpoint hooks that keep saved artifacts round-trippable between GPT-QModel, Hugging Face, and vLLM:

• checkpoint_key_for_save()/checkpoint_key_for_load() translate between released HF checkpoint names and GPT-QModel's runtime tree through the reversed HF conversion map. Load additionally restores the missing vision_tower.vision_model.* hop: HF tensors live under vision_tower.vision_model.* while the shell exposes model.vision_tower.*.
• prepare_quantize_config_for_save() injects dynamic exclusions r"-:vision_tower\..*" and r"-:multi_modal_projector\..*" so a reload without the definition's module tree (for example under vLLM) does not try to interpret dense vision weights as quantized.
• copy_checkpoint_auxiliary_files_for_save() copies preprocessor_config.json, processor_config.json, and tokenizer_config.json next to the checkpoint so the saved directory loads standalone.

Its prepare_dataset() override also accepts pretokenized calibration samples — mappings containing input_ids (with an optional attention_mask, no other keys) — in addition to conversation-formatted samples.

Image and audio utilities

Images: gptqmodel/utils/image.py

def extract_vision_info(conversations: list[dict] | list[list[dict]]) -> list[dict]:

Source: gptqmodel/utils/image.py.

def fetch_image(
    ele: dict[str, str | Image.Image],
    image_patch_size: int | None = None,
) -> Image.Image:

Source: gptqmodel/utils/image.py.

extract_vision_info() accepts one conversation or a list of conversations. It scans list-valued message content and selects dictionaries with image/image_url/video keys or matching type names. It does not decode video, load files, or validate a complete conversation schema. These low-level helpers expect nonempty, structurally valid input; the Qwen3 adapter adds higher-level validation.

fetch_image() input	Behavior
PIL Image.Image	Return the supplied image object.
HTTP(S) string	Download through load_image_from_url() (bounded fetch, see below), then decode with PIL.
file://...	Strip the seven-character prefix and open locally.
data:image...base64,...	Base64-decode and open a BytesIO.
Other string	Open as a local path.
Unrecognized/unusable input	Raise an error; the explicit unsupported-input path raises ValueError.

image_patch_size=None is accepted for call compatibility and discarded. This function does not resize, normalize, tile, or convert to RGB; the processor/adapter owns those operations. A bare image_url value is the image source; this helper does not unpack an OpenAI-style {"url": ...} value.

Remote image downloads are bounded and diagnosed by load_image_from_url() in utils/image.py, which fetch_image() and the InternS1/Ovis2.5 adapters all route through:

Bound / behavior	Contract
Connect / read timeout	(5.0, 30.0) seconds by default; pass a number or (connect, read) pair as timeout.
Wall-clock budget	120.0 seconds total across every attempt (total_timeout); a watchdog aborts the socket so a slow-drip response cannot hold the run forever.
Download cap	32 MiB (max_bytes), enforced against Content-Length and while streaming.
Decode cap	64 Mpx (max_pixels), checked from declared dimensions before Image.load() allocates, so decompression bombs fail instead of spiking RSS.
Retries	3 attempts (attempts) with exponential backoff starting at 0.5 s (backoff_seconds), only for transient faults: timeouts raised mid-request, HTTP 408/425/429/5xx, and truncated bodies. URL/schema errors, other HTTP statuses, oversized bodies, and undecodable bytes are not retried.
Failure	ImageFetchError(RuntimeError) carrying .url (shortened to 200 characters in the message) and .reason, so a failed calibration download is diagnosable instead of hanging or surfacing as an opaque PIL error.

GPTQMODEL_IMAGE_FETCH_MAX_BYTES and GPTQMODEL_IMAGE_FETCH_MAX_PIXELS override the byte/pixel limits when the call leaves them unset; non-integer values warn and fall back to the defaults, and resolved limits must be positive. Calibrating with untrusted remote media still deserves source control: bounded fetch limits resource exhaustion, not content suitability.

Audio: gptqmodel/utils/audio.py

def process_audio_info(conversations: list[dict] | list[list[dict]],
                       use_audio_in_video: bool,
                       sample_rate: int=16000
                      ):

Source: gptqmodel/utils/audio.py.

Parameter/content field	Default	Behavior
conversations	Required	Single conversation or batch of conversations. Only list-valued content is inspected.
use_audio_in_video	Required	Whether video entries are considered as audio sources.
sample_rate	16000	Target rate for file decoding and index rate used for NumPy-array slicing.
audio / audio_url	Required for type="audio"	NumPy array, local path, file URL, HTTP(S), or base64 audio data URI.
audio_start, audio_end	0.0, None	Time offset/end; arrays are sliced using int(sample_rate * time).
video / video_url	Required for video audio extraction	Path/URL with an audio track, checked using PyAV.
video_start, video_end	0.0, None	Bounds for video audio decoding.

The result is a list of waveforms or None when no audio is found. NumPy inputs with more than one dimension raise ValueError("Support only mono audio"); the array path does not resample or infer its original rate, so the caller must supply mono data at the requested rate. Other sources use librosa.load(sr=sample_rate, offset=..., duration=...)[0]. Remote sources use audioread.ffdec.FFmpegAudioFile; enabling video audio asserts that an audio stream exists. This helper decodes waveforms, not model-specific mel/features.

Dependencies and hardware constraints

Component	Verified requirement / constraint
Core source revision	requirements.txt declares torch>=2.14.1, transformers>=5.20.2, defuser>=0.0.28, and tokenicer>=0.0.14; older releases can have different layouts/APIs.
Images	PIL/Pillow and requests are imported by utils/image.py; this revision requires requests>=2.34.2, Pillow >=12.3.0 on Python <3.11 or >=13.0.0 on Python >=3.11.
Audio files/resampling	librosa>=0.11.0 is the utility's install hint; a missing library raises ValueError when decoding is required.
Remote audio	audioread>=3.1.0 install hint plus the selected audioread FFmpeg backend.
Video audio-stream check	av>=16.0.1 install hint. Qwen2.5 Omni declares all three audio package versions in require_pkgs.
Qwen3-VL video	Lazy import of qwen_vl_utils.vision_process.fetch_video; source suggests pip install qwen-vl-utils and does not pin its version here. Image-only preparation does not require this import.
Remote model implementations	Definitions such as InternVL Chat and MiMo v2 require trust_remote_code=True; inspect remote code before enabling it.
Accelerator placement	PyTorch CUDA/ROCm paths use cuda:N device names. Placement pools must be subsets of visible compute devices; they do not override backend/kernel compatibility.
Apple Silicon	These Torch adapter/looper details are not an MLX execution contract. GPTQModel.quantize_mlx() is a separate path with a lazy mlx_lm import; see MLX Apple Silicon.

MiMo mixed checkpoint sources

gptqmodel/utils/mimo.py is a weight source decoder, not an audio/video preprocessing library. It recognizes MiMo v2 configurations combining FP8 and packed MXFP4 storage and reconstructs floating weights for the shell's current operation.

def is_mimo_mixed_source(config: Any) -> bool:

Source: gptqmodel/utils/mimo.py.

def decode_mxfp4(
    weight: torch.Tensor, scale: torch.Tensor, *, target_dtype: torch.dtype
) -> torch.Tensor:

Source: gptqmodel/utils/mimo.py.

def decode_mimo_weight(
    config: Any,
    name: str,
    weight: torch.Tensor,
    lookup: Callable[[str], torch.Tensor | None],
    *,
    target_dtype: torch.dtype,
) -> torch.Tensor | None:

Source: gptqmodel/utils/mimo.py.

Check / operation	Source contract
is_mimo_mixed_source(config)	Requires model_type == "mimo_v2", quantization quant_method == "fp8", and store_dtype == "mxfp4".
is_mimo_encoded_weight(config, full_name, tensor)	Recognizes FP8 E4M3 weights and uint8 expert MXFP4 weights; ignores non-weight names and model.mtp.*; validates supported dense dtypes.
MXFP4 source	Weight and scale tensors must be uint8 matrices. A packed weight of shape [out, packed_in] decodes to [out, 2 * packed_in]; logical columns must be divisible by 32 and scales must have shape [out, logical_in / 32]. Scale byte 255 is rejected. The dispatcher requires mxfp4_block_size=32 and loads <weight_name>_scale.
FP8 source	Companion <weight_name>_scale_inv must be an FP32 matrix with finite positive values. weight_block_size must contain two positive integers; the scale grid must match the block geometry.
Target dtype	decode_mimo_weight() accepts only FP16, BF16, or FP32 for recognized encoded weights. This dispatcher owns that validation; the lower-level decode_mxfp4() does not independently restrict target dtype to the same list.
Fused QKV	Decoder validates configuration-derived Q/K/V geometry and applies scale layout rules rather than treating every fused tensor as an arbitrary dense matrix.
Materialization	LazyTurtle invokes the MiMo decoder when copying encoded source tensors into a float shell. The loader requires a checkpoint-backed shell for this source type and rejects ordinary direct CPU fallback.

The MimoV2QModel tree supports fused/unfused attention names and dense/MoE MLP alternatives with non-strict matching. It handles checkpoint-omitted vision/audio tensors and wraps the inherited run_input_capture() to retain the original attention mask; the inherited method calls self.model(**example, use_cache=...), rather than a MiMo-specific text-only submodel. Capture also retains full/sliding-window rotary inputs, and replay rebuilds the selected layer's mask and positional kwargs. These accommodations do not add a raw-media conversation/processor override to its inherited dataset preparation.

MoE concepts and module handling

A routed MoE layer selects experts using a router; a shared expert may run for every token. Calibration following only normal sparse routing can leave some experts with little or no data. GPT-QModel separates three controls:

Control	Purpose	Does not mean
Expert tree/count metadata	Discover and schedule expert projection weights.	Automatically calibrate every expert on every token.
MoEConfig.routing	Preserve routing, temporarily change top-k, or bypass sparse routing for calibration.	Change the saved model's permanent inference routing policy.
moe_vram_strategy*	Place whole expert families on a selected device pool.	Set the number of experts per token or a global dataset batch size.

Definition-side expert metadata

MoE definitions declare a config count field (dynamic_expert_index), optional per-family count fields (dynamic_expert_indices), a :moe root, and nested expert "#" placeholders. The exported constant is EXPERT_INDEX_PLACEHOLDER="{expert_index}". Expansion produces paths such as mlp.experts.0.gate_proj.

Counts are resolved from text_config, thinker_config.text_config, or the root config. Per-prefix counts permit distinct expert families; the longest matching prefix wins. Dense layers and sparse layers can coexist with layer_modules_strict=False and optional branches.

For example, the Qwen2-MoE structure distinguishes the router, shared expert gate, shared expert projections, and numbered routed experts. Mixtral uses runtime/checkpoint aliases (mlp|block_sparse_moe, gate_proj|w1, up_proj|w3, down_proj|w2); the runtime tree follows the defused shell, while LazyTurtle resolves original checkpoint names. Qwen3-VL MoE and Flint similarly rely on Defuser to expose fused or stacked experts as individual linears.

Do not quantize router gates accidentally: the cited definitions mark those small routing/control projections :!. Do not share Hessian capture across routed experts: identical :in=x tags are scoped to each expanded expert's own parent, not all experts in a layer.

MoE definition reference

This table includes definitions that directly declare expert-count metadata or lifecycle hooks. Inherited classes can add further model-type aliases; see the complete MODEL_MAP table on the model-definition page.

Definition	Count field / per-family fields	Lifecycle declaration	Implementation
AXK2QModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/axk2.py
AfMoeQModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/afmoe.py
BailingMoeQModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/bailing_moe.py
Cohere2MoeQModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/cohere2_moe.py
DeepSeekOCR2QModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/deepseek_ocr2.py
DeepSeekV2QModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/deepseek_v2.py
DeepSeekV32QModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/deepseek_v32.py
DeepSeekV3QModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/deepseek_v3.py
DeepSeekV4QModel	'n_routed_experts'	Inherited / not declared here	gptqmodel/models/definitions/deepseek_v4.py
DeepSeekVLV2QModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/deepseek_vl_v2.py
DiffusionGemmaQModel	'num_experts'	Inherited / not declared here	gptqmodel/models/definitions/diffusion_gemma.py
Dots1QModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/dots1.py
Ernie4_5_MoeQModel	'moe_num_experts'	Inherited / not declared here	gptqmodel/models/definitions/ernie4_5_moe.py
FlintQModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/flint.py
GLM4MoEGPTQ	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/glm4_moe.py
GPTOSSGPTQ	'num_local_experts'	Inherited / not declared here	gptqmodel/models/definitions/gpt_oss.py
Glm4MoeLiteQModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/glm4_moe_lite.py
Glm4vMoeQModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/glm4v_moe.py
Glm5NextQModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/glm5_next.py
GlmMoeDsaQModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/glm_moe_dsa.py
GraniteMoeHybridQModel	'num_local_experts'	Inherited / not declared here	gptqmodel/models/definitions/granitemoehybrid.py
GraniteMoeQModel	'num_local_experts'	Inherited / not declared here	gptqmodel/models/definitions/granitemoe.py
GrinMoeQModel	'num_local_experts'	W1W3W2MoELifecycleHooks()	gptqmodel/models/definitions/grinmoe.py
HYV3QModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/hy_v3.py
HunYuanMoEV1QModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/hunyuan_v1_moe.py
HymbaQModel	Inherited / not declared here	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/hymba.py
InklingMMQModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/inkling.py
InternS2PreviewQModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/intern_s2_preview.py
K2HorizonQModel	'num_experts' {'mlp.experts': 'num_experts', 'self_attn.v_experts': 'mova_num_experts'}	K2HorizonMoELifecycleHooks()	gptqmodel/models/definitions/k2_horizon.py
KimiK25QModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/kimi_k25.py
KlearQModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/klear.py
LFM2MoeQModel	'num_experts'	W1W3W2MoELifecycleHooks()	gptqmodel/models/definitions/lfm2_moe.py
LLaDA2MoeQModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/llada2.py
LagunaQModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/laguna.py
Llama4QModel	'num_local_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/llama4.py
LongCatFlashQModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/longcat_flash.py
MimoV2QModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/mimo_v2.py
MiniMaxM2GPTQ	'num_local_experts'	W1W3W2MoELifecycleHooks()	gptqmodel/models/definitions/minimax_m2.py
MiniMaxM3VLGPTQ	'num_local_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/minimax_m3_vl.py
MixtralQModel	'num_local_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/mixtral.py
NemotronHPuzzleQModel	'n_routed_experts'	Inherited / not declared here	gptqmodel/models/definitions/nemotron_h_puzzle.py
OlmoeQModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/olmoe.py
Ovis2_6_MoeQModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/ovis2_6_moe.py
PhiMoEGPTQForCausalLM	'num_local_experts'	Inherited / not declared here	gptqmodel/models/definitions/phi3.py
Qwen2MoeQModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/qwen2_moe.py
Qwen3MoeQModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/qwen3_moe.py
Qwen3NextGPTQ	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/qwen3_next.py
Qwen3OmniMoeGPTQ	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/qwen3_omni_moe.py
Qwen3_5_MoeQModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/qwen3_5_moe.py
Qwen3_VL_MoeQModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/qwen3_vl.py
Qwen4ExpQModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/qwen4_exp.py
Raon1MiniQModel	'num_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/raon1_mini.py
SolarOpen2QModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/solar_open2.py
SolarOpenQModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/solar_open.py
UnlimitedOCRQModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/unlimited_ocr.py
Xing4_0QModel	'n_routed_experts'	GateUpDownMoELifecycleHooks()	gptqmodel/models/definitions/xing4_0.py

MoE configuration and examples

The following declarations are copied from gptqmodel/quantization/config.py:

class VramStrategy(str, Enum):
    """Placement strategies shared by dense and MoE device pools."""

    EXCLUSIVE = "exclusive"
    BALANCED = "balanced"


@dataclass
class ExpertsRoutingOverride(BaseMoERouting):
    num_experts_per_tok: Union[int, str] = MOE_ALL_EXPERTS


@dataclass
class ExpertsRoutingBypass(BaseMoERouting):
    # Number of modules to process in a single batch to reduce VRAM pressure during quantization
    # For example, with batch_size=10 and 20 expert modules (gate_proj + up_proj for 10 experts):
    # - First batch processes 10 modules (could be gate_proj for experts 0-9, or a mix depending on sorting)
    # - Second batch processes remaining 10 modules
    batch_size: Optional[int] = field(
        default=None,
        metadata={"help": "Number of modules to process in a single batch during MoE quantization"}
    )


@dataclass
class MoEConfig:
    routing: BaseMoERouting

MOE_ALL_EXPERTS is the string "all". BaseMoERouting is an empty marker dataclass, not an abstract interface that implements routing. MoEConfig requires a BaseMoERouting instance and exposes routing_bypass(), routing_override(num_experts), and to_dict().

Option	Default	Accepted values / behavior
QuantizeConfig.moe	None	No routing override/bypass. Accepts MoEConfig or normalized dictionary.
ExpertsRoutingOverride.num_experts_per_tok	"all"	Positive integer, numeric string normalized to integer, or case-insensitive/trimmed "all". Invalid/nonpositive values raise ValueError.
Resolved override	—	"all" resolves to available expert count; oversized integer is clamped to it with a log message.
ExpertsRoutingBypass.batch_size	None	Number of modules, not calibration examples, staged together in a qualifying MoE subset. None leaves it unchunked.
moe_vram_strategy	VramStrategy.EXCLUSIVE	"exclusive" or "balanced"; normalized from a case-insensitive string or enum. Unlike the top-k string parser, this parser does not trim whitespace.
moe_vram_strategy_devices	None	Optional nonempty list of device strings/torch.device; normalized and deduplicated. Empty/invalid lists raise ValueError.
dense_vram_strategy	VramStrategy.EXCLUSIVE	Separate pool strategy for the serial dense path, router, shared expert, and dense MLP.
dense_vram_strategy_devices	None	Explicit dense pool; same device normalization rules.
auto_forward_data_parallel	True	Enables forward replication; may increase VRAM pressure and time on MoE. Set False to serialize the ordinary forward path.
offload_to_disk	True	Release completed modules through offload machinery; source loading uses a shell/checkpoint-backed turtle where supported.
wait_for_submodule_finalizers	False	Force packing/offload finalizers to drain before advancing layers. False does not prevent the looper from waiting for method/device safety.

Change calibration coverage

from gptqmodel import GPTQModel, QuantizeConfig
from gptqmodel.quantization.config import (
    ExpertsRoutingBypass,
    ExpertsRoutingOverride,
    MoEConfig,
)

# Model IDs and paths here are placeholders for a supported local checkpoint.
config = QuantizeConfig(
    bits=4,
    group_size=128,
    moe=MoEConfig(routing=ExpertsRoutingBypass(batch_size=4)),
)
model = GPTQModel.load("/models/source-moe", quantize_config=config)
model.quantize(calibration_dataset, batch_size=1)
model.save("/models/quantized-moe")

# Alternative policies when constructing a configuration:
normal_routing = None
top_two = MoEConfig(routing=ExpertsRoutingOverride(num_experts_per_tok=2))
all_via_router = MoEConfig(routing=ExpertsRoutingOverride(num_experts_per_tok="all"))
all_via_replay = MoEConfig(routing=ExpertsRoutingBypass())

calibration_dataset must already contain examples suitable for the selected architecture. Routing bypass is not identical to setting top-k to all: the lifecycle explicitly replays expert computations for collection and then uses the original routed forward for correct outputs when collection has not early-stopped; override instead adjusts supported routing attributes during the forward context. ModuleLooper.MoERoutingOverrideContext uses set_moe_topk()/restore_moe_topk() and restores original top-k state on exit, including when an exception propagates. Recognized routing fields are top_k, moe_k, and num_experts_per_tok; the utility clamps the resolved override again to each router's local expert count, which matters for models with more than one expert family.

Dictionary normalization uses concrete routing class names:

config = QuantizeConfig(
    moe={
        "routing": {
            "class": "ExpertsRoutingOverride",
            "num_experts_per_tok": "all",
        }
    }
)

The accepted class names are ExpertsRoutingOverride and ExpertsRoutingBypass; unknown classes or non-routing values raise ValueError. MoEConfig.to_dict() emits {"routing": {"class": ..., ...dataclass_fields}}, and quantization serialization stores MoE settings in metadata for reload.

Place expert and dense pools separately

from gptqmodel import QuantizeConfig
from gptqmodel.quantization.config import VramStrategy

config = QuantizeConfig(
    bits=4,
    group_size=128,
    dense_vram_strategy=VramStrategy.EXCLUSIVE,
    dense_vram_strategy_devices=["cuda:0"],
    moe_vram_strategy=VramStrategy.BALANCED,
    moe_vram_strategy_devices=["cuda:0", "cuda:1"],
    auto_forward_data_parallel=False,
)

This example requires two visible compute devices. Device indices are relative to CUDA_VISIBLE_DEVICES (also use cuda:N under ROCm). Pool resolution rejects any requested device outside the available compute-device list. Default None pools use the available device set; they do not create virtual devices.

Explicit strategy placement in stage_subset.py groups routed expert families so gate/up/down stay together. Building those MoE groups requires at least 16 recognized MoE projection modules across the layer's candidate modules in this revision. Within that eligible plan, balanced placement assigns groups round-robin across the MoE pool; exclusive placement uses its first device. Dense balanced placement groups related calculations, keeping Q/K/V together; eligible shared-expert projections use the dense serial path rather than being split as routed families. Pool selection does not add excluded routers or other :! entries to the quantization plan. These are placement groups, not tensor-parallel splits within a projection.

The looper activates explicit pool planning when a pool's device list is supplied or its strategy differs from EXCLUSIVE. Merely passing EXCLUSIVE with no device list does not activate that branch: the looper retains separate legacy/automatic scheduling behavior. Forward parallelism, compute-device filtering, module overrides, and backend capabilities remain additional constraints.

MoE lifecycle internals

MoELifecycleHooks in gptqmodel/models/moe_lifecycle.py locates the :moe root and the relevant expert containers. Common routed container names are experts; shared containers are shared_experts and shared_expert. Definitions can provide a specialized hook implementation.

Class / method	Responsibility
MoELifecycleHooks.get_moe_block()	Default implementation resolves the first tree-reported MoE root with getattr() on the current layer/replica.
get_moe_block_for_subset()	Default delegates to get_moe_block(); specialized implementations can choose a root relevant to the current subset.
get_subset_execution_order()	Preserve the tree's shared/routed execution order rather than hard-coding shared-first or expert-first.
forward_to_all_experts()	Extension point for model-specific replay. The base implementation logs that replay is unimplemented and calls the MoE block; it is not an abstract-method guard or an all-expert replay implementation.
ExpertProjectionMoELifecycleHooks	Configurable gated expert replay using gate/up/down projection names and expert activation.
GateUpDownMoELifecycleHooks	Projection names gate_proj, up_proj, down_proj.
W1W3W2MoELifecycleHooks	Projection names w1, w3, w2.

ModuleLooper._should_use_moe_lifecycle() requires routing bypass, configured hooks, and a resolvable MoE block. Missing hooks or blocks log a warning and disable the optimization for that module; moe=...Bypass() by itself cannot implement an unsupported architecture.

For an eligible subset, MoELifecycleContext stores the original root forward, installs a wrapper calling the lifecycle hook with subset order and replica information, and restores the original method in __exit__. The projection implementation flattens hidden states, resolves the current expert/shared modules, runs needed gate/up projections and expert activation/products, and runs required down projections. It uses full inputs for expert calibration rather than the router's sparse token assignments.

After explicit expert collection, replay pauses normal collection hooks while calling the original routed forward, avoiding double collection and preserving correct layer outputs. If an expert collection hook raises StopForward, the implementation propagates that capture stop before the original routed call; it does not promise a complete layer output in this early-stop path. Both forward executors tolerate that empty result: run_single already skips such a batch, and the data-parallel run_parallel in looper/forward_executor.py now mirrors the serial contract instead of raising RuntimeError: Forward batch returned no output, so a multi-GPU quantization run that reaches an early-stopping MoE subset no longer crashes. When no expert projection was explicitly replayed, the original forward runs with hooks still active. Replica-relative resolution and thread-local hook pausing matter when forwarding across devices; directly patching an original global module can target the wrong replica.

What ExpertsRoutingBypass.batch_size actually limits

stage_subset.py classifies a subset as MoE when it contains at least ModuleLooper._moe_subset_threshold, which is 16 recognized MoE modules in this revision. Only a bypass routing object, positive non-None module batch size, and a stage requiring forward execution trigger the deterministic chunk path. It sorts subset module names and slices them into chunks of that size.

This is not an input-example batch size, not a number of experts, not a universal limit for every small MoE, and not a hard total VRAM cap. Other modules, captured inputs, Hessians, replicas, and pending finalizers still consume memory. The dataclass does not impose a positive-value validator on ExpertsRoutingBypass.batch_size; nonpositive values simply do not activate this chunk condition. Use a positive integer when requesting staged execution.

Multimodal usage example

Use real media and an architecture with the corresponding conversation adapter. This Qwen2.5-VL example uses an image, not video:

from gptqmodel import GPTQModel, QuantizeConfig

calibration_dataset = [
    [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": "/data/calibration/example.jpg"},
                {"type": "text", "text": "Describe the objects and their positions."},
            ],
        }
    ],
]

model = GPTQModel.load(
    "/models/qwen2.5-vl-source",
    quantize_config=QuantizeConfig(bits=4, group_size=128),
)
model.quantize(calibration_dataset, batch_size=1)
model.save("/models/qwen2.5-vl-quantized")

This is a structural example, not a recommended calibration corpus size. Use representative conversations/media for the deployment workload. For a supported audio adapter, content items use {"type": "audio", "audio": "/data/example.wav"} (or supported array/URL forms); the adapter must actually call process_audio_info().

Saving retains the processor after the tokenizer so multimodal chat templates survive. Preserve architecture-specific speaker assets, remote Python files when required, and passthrough tensors declared by the model. A successful decoder-weight save alone is not proof of a complete multimodal model package.

Pitfalls and troubleshooting

Symptom / misconception	Check
Vision tower weights remain dense	Expected for the decoder-only trees documented here. Check the definition rather than increasing a global bit setting.
Mllama cross-attention layers remain dense	Explicit unsupported-quantization boundary; both layer and module hooks enforce it.
Raw video rejected on Qwen2/2.5-VL	Its image helper does not load video. Qwen3's video helper is a different implementation with optional qwen_vl_utils dependency.
Processor/chat-template failure	Correct processor files/version, adapter's expected conversation shape, and exact media placeholder/content format. Tokenizer-only preprocessing is not equivalent.
Meta tensor error before decoder capture	Tower, projector, rotary buffer, embedding, or an omitted checkpoint parameter may need architecture-specific materialization. Do not blindly .to() a meta shell.
Expert receives little calibration data	Normal routing may not visit it. Choose supported bypass or override deliberately; confirm lifecycle hooks are active.
Bypass warning: no hooks/block	Check the definition's moe_lifecycle_hooks, :moe tree root, Defuser conversion, and expert container names.
OOM despite small bypass batch size	It limits modules only in qualifying subsets; inspect tower inputs, captured activations, dense pools, forward replication, and finalizers.
Invalid MoE device pool	Pool must be a nonempty normalized subset of visible compute devices. Account for visibility renumbering.
Shared-input dedup disabled	Concrete model_type must be verified on the definition. Never deduplicate distinct expert parents just because their projection names match.
MiMo direct-load failure	Mixed FP8/MXFP4 input requires a checkpoint-backed shell; generic HF CPU loading cannot preserve/interpret the encoded source correctly.
Good quantization loss but broken multimodal inference	Validate the saved processor/template and dense tower/projector/skip weights as well as quantized decoder weights. Compare against an unquantized baseline using the exact rendered inputs.

Source files

• gptqmodel/models/definitions/base_qwen2_vl.py, qwen2_vl.py, qwen2_5_vl.py, base_qwen3_vl.py, qwen3_vl.py — language trees, multimodal preprocessing, processor validation, video helper, MoE VL expert structure.
• gptqmodel/models/definitions/qwen3_5.py, qwen3_5_moe.py, qwen3_5_text.py, qwen3_5_moe_text.py — hybrid dense/MoE trees and distinct preprocessing inheritance.
• gptqmodel/models/definitions/ovis.py, ovis2.py, ovis2_5.py, ovis2_6_moe.py — version-specific vision tokenization/materialization and dense/MoE language paths.
• gptqmodel/models/definitions/internvl.py, internvl_chat.py, mllama.py — native/remote InternVL, Mllama layout variants and cross-attention exclusions.
• gptqmodel/models/definitions/qwen2_audio.py, base_qwen2_5_omni.py, qwen2_5_omni.py, qwen3_omni_moe.py — audio/thinker/talker boundaries.
• gptqmodel/models/definitions/gemma3.py, gemma3n.py, nemotron_omni.py, mimo_v2.py — additional multimodal/hybrid capture and replay implementations.
• gptqmodel/utils/image.py, gptqmodel/utils/audio.py — media loading helpers and optional dependencies.
• gptqmodel/utils/mimo.py, gptqmodel/utils/structure.py — MiMo source decoding and checkpoint-backed materialization.
• gptqmodel/models/base.py, gptqmodel/models/_const.py, gptqmodel/models/shared_input.py — expert expansion, metadata flags, shared-input isolation.
• gptqmodel/models/moe_lifecycle.py, gptqmodel/quantization/config.py — lifecycle interfaces, routing classes, normalization, placement options.
• gptqmodel/looper/module_looper.py, stage_subset.py, weight_only_looper.py — contexts, pool validation, chunking and device placement.
• gptqmodel/models/loader.py, writer.py, auto.py — architecture dispatch, Defuser/lazy source, saved processor and passthrough tensors.
• gptqmodel/utils/calibration.py, gptqmodel/utils/model.py, requirements.txt — conversation normalization, modality enum, source-version dependencies.

See also

• Model Definitions and Module Tree
• Quantization Lifecycle
• QuantizeConfig Reference
• Memory Offload and Multi-GPU
• Looper Processors
• FP8 and NVFP4
• Loading and Inference
• HuggingFace Integration
• Installation and Quickstart
• Troubleshooting and FAQ