Skip to content

Troubleshooting and FAQ

Jump to bottom
Qubitium edited this page Oct 29, 2026 · 5 revisions

Troubleshooting and FAQ

Documentation home · Quickstart · Lifecycle

On this page

Overview

Most failures occur at a boundary: the installed package set versus the requested engine, checkpoint representation versus kernel capabilities, physical GPU visibility versus logical device indices, or calibration memory versus runtime inference memory. Diagnose that boundary before changing quantization parameters or deleting caches.

This page is verified against source revision c1976918cfe3703fbcf6d1ce2440a82b26394375, which declares GPT-QModel 7.7.0. The dependency ranges and flags below are revision-specific. Older releases, external engines, and model remote code can have different requirements.

Workflow at a glance

Diagnose the failing boundary. Follow the lane matching the failure before changing data or metadata.

Reading the diagram: Lanes are independent diagnoses. Workspace budgets are not total-memory caps; an inference export is not a resume checkpoint.

Text version of the workflow
  • Import or JIT failure: Record the environment → Identify the extension → Fix the first failure → Retry the affected path.
  • Kernel selection failure: Read saved metadata → Resolve the device → Check kernel contracts → Choose a valid backend.
  • Quantization OOM: Locate the stage → Inspect live memory → Bound relevant work → Retry and validate.
  • Interrupted quantization: Reload the source → Verify resume identity → Restore committed state → Finish, then export.

Open full-size SVG

First checks

Symptom First check Relevant section
Import error or dependency resolver conflict Installed Python/Torch/Transformers and optional-engine versions; pip check. Dependency/version conflicts
Kernel build or shared-library failure The first compiler/linker error, installed toolchain, build root, CUDA/HIP version, and target architecture. Kernel build failures
No compatible quantized linear kernel Actual method/format/bits/group size/activation order/symmetry/dtype/device and shape. Backend compatibility
Quantization OOM Whether allocation comes from input capture, Hessian, dense forward, finalization, or packing. Quantization memory
Wrong device or invalid ordinal Visibility/order before Python starts; normalized device map. Device selection
Resume rejected Same original model, calibration, configuration, packing, software and device identity. Checkpoint recovery
Poor generation or evaluation Dense baseline plus exact prompts/input IDs before blaming quantization. Quality and tokenizer issues

A lightweight environment report that does not load model weights:

python -m pip check
python -m pip show GPTQModel torch transformers accelerate tokenicer optimum
python - <<'PY'
import importlib.metadata as metadata
import platform
import torch

print("Python:", platform.python_version())
for name in ("GPTQModel", "torch", "transformers", "accelerate", "tokenicer",
             "optimum", "bitblas", "triton", "vllm", "sglang"):
    try:
        print(name, metadata.version(name))
    except metadata.PackageNotFoundError:
        print(name, "not installed")
print("Torch CUDA:", torch.version.cuda, "Torch HIP:", torch.version.hip)
print("CUDA API available:", torch.cuda.is_available())
for i in range(torch.cuda.device_count()):
    print(i, torch.cuda.get_device_name(i), torch.cuda.get_device_capability(i))
PY

On ROCm, PyTorch exposes AMD devices through its torch.cuda API. This report's CUDA API availability is not evidence of an NVIDIA GPU; include torch.version.hip and actual device names. For XPU/MPS/NPU, report that platform's availability and devices separately.

Environment-variable reference

What is actually defined in utils/env.py?

gptqmodel/utils/env.py defines no named application environment variables. It provides a generic helper used throughout the package:

TRUTHFUL = {"1", "true", "yes", "on", "y"}

def env_flag(name: str, default: str | bool | None = "0") -> bool:
    """Return ``True`` when an env var is set to a truthy value."""

    value = os.getenv(name)
    if value is None:
        if default is None:
            return False
        if isinstance(default, bool):
            return default
        value = default
    return str(value).strip().lower() in TRUTHFUL

The helper returns true only for the trimmed, case-insensitive values in TRUTHFUL. An unset variable uses the caller's default; an explicitly empty value is false even if the default is true. None defaults are false. Values such as "2", "enabled", and "t" are not truthy for this helper.

The tables enumerate its named consumers, dynamic JIT controls, and related operational settings. Not every setting uses env_flag: integer conversions, permissive false-lists, and exact "1" tests are identified below. Set variables before starting Python unless the consumer explicitly reads them per operation. Many flags are module globals or cached capability probes; changing os.environ after imports may have no effect.

Build, JIT and cache controls

Variable Default / accepted value Effect and source
GPTQMODEL_KERNEL_REBUILD False; env_flag Global force-rebuild control for TorchOpsJitExtension; utils/cpp.py. Remove it after diagnosis to avoid rebuilding every new process.
GPTQMODEL_TORCH_EXTENSIONS_DIR Unset Parent root for default_torch_ops_build_root(subdir); otherwise ~/.cache/gptqmodel/torch_extensions/<subdir>. Extension-specific build roots take precedence for those extensions.
GPTQMODEL_TORCH_OPS_LOCK_TIMEOUT Unset: max(600.0, 5.0 * compile_baseline_seconds) seconds Cross-process JIT lock wait scales with the extension's compile baseline; a missing baseline is zero. An explicit numeric value replaces the adaptive wait; negative values clamp to zero. Invalid text falls back to 600.0, not the adaptive baseline.
GPTQMODEL_NVCC_COMPILE_LEVEL Unset; common compile default is O3 Case-insensitive O0, O1, O2, O3, or NONE, NOOPT, NO_OPT, OFF, DISABLE, 0; surrounding whitespace and one leading - are stripped, and an empty value also disables optimization flags. Disable spellings remove the optimization-level flag; O0 explicitly requests it. Other values raise ValueError. Despite its name, the shared helper normalizes C++ and CUDA optimization flags.
NVCC_THREADS "8" unless an explicit thread argument is passed CUDA compiler --threads setting; not the number of concurrent Ninja build jobs.
GPTQMODEL_EXT_VERBOSE False through env_flag Verbose shared extension builds. The CPU helpers load_pack_block_extension(verbose=True) and load_floatx_cpu_extension(verbose=True) in utils/cpp.py temporarily set it and restore the prior environment. The public extension.load() has no verbose parameter; set the environment variable for that API. ExLlama v3's early module-level parser uses a false-list rather than TRUTHFUL.
GPTQMODEL_EXT_BUILD Unset Build-root override shared by CPU pack-block, FloatX, and INT4 qzero-group-scaled extensions. Otherwise their roots are separate subdirectories under the default root.
TORCH_CUDA_ARCH_LIST Unset PyTorch CUDA target override. GPT-QModel may temporarily merge visible GPU capabilities into an override, then restore it. A manual target list does not bypass hardware/kernel checks.
TORCH_EXTENSIONS_DIR Torch-owned Included in failure diagnostics; distinct from GPT-QModel's own explicit build-root control. Setting it alone does not replace GPTQMODEL_TORCH_EXTENSIONS_DIR.
CUDA_PATH Unset Explicit CUDA toolkit root considered by shared build helpers in addition to Torch's detected CUDA home.
ROCM_PATH, HIP_PATH, ROCM_HOME Unset ROCm-root candidates used by shared build helpers; detection also considers the Torch-provided ROCm home and /opt/rocm.
GPTQMODEL_CACHE_DIR Unset General GPT-QModel cache root for Machete/ParoQuant cache helpers; otherwise $XDG_CACHE_HOME/gptqmodel, then ~/.cache/gptqmodel. Not the same control as the Torch extension build root.
XDG_CACHE_HOME Unset Fallback parent for those general caches.
GPTQMODEL_OFFLINE False; env_flag Suppresses Machete online precompiled-cache probing; it is not a global Hugging Face/dataset offline switch.
GPTQMODEL_CUTLASS_DIR Unset Machete CUTLASS search override; it must contain the expected CUTLASS header tree. See utils/machete.py.
GPTQMODEL_MACHETE_PRECOMPILED_LIBRARY Unset Explicit Machete prebuilt library path handled by TorchOpsJitExtension; do not point it at an incompatible binary.
GPTQMODEL_EXT_DEBUG Off ExLlama v3 debug compile behavior; accepts 1, true, on, yes after normalization, not the full env_flag truth set.

Each row below names the complete override variables. Build-root variables default to unset and rebuild variables default to false. Default subdirectories are beneath the root described above; fingerprinted build directories may exist within them.

Extension / source Build-root variable Force-rebuild variable Default subdirectory
GPTQ block / quantization/gptq_cuda.py GPTQMODEL_GPTQ_BLOCK_BUILD_ROOT GPTQMODEL_GPTQ_BLOCK_FORCE_REBUILD gptq_block
AWQ / utils/awq.py GPTQMODEL_AWQ_BUILD_ROOT GPTQMODEL_AWQ_FORCE_REBUILD awq
QQQ / utils/qqq.py GPTQMODEL_QQQ_BUILD_ROOT GPTQMODEL_QQQ_FORCE_REBUILD qqq
VPTQ / utils/vptq_cuda.py GPTQMODEL_VPTQ_BUILD_ROOT GPTQMODEL_VPTQ_FORCE_REBUILD vptq
ExLlama v2 GPTQ / utils/exllamav2.py GPTQMODEL_EXLLAMAV2_BUILD_ROOT GPTQMODEL_EXLLAMAV2_FORCE_REBUILD exllamav2
ExLlama v2 AWQ / same source GPTQMODEL_EXLLAMAV2_AWQ_BUILD_ROOT GPTQMODEL_EXLLAMAV2_AWQ_FORCE_REBUILD exllamav2_awq
ExLlama v3 / exllamav3/ext.py GPTQMODEL_EXLLAMAV3_BUILD_ROOT GPTQMODEL_EXLLAMAV3_FORCE_REBUILD exllamav3
Marlin FP16 / utils/marlin.py GPTQMODEL_MARLIN_FP16_BUILD_ROOT GPTQMODEL_MARLIN_FORCE_REBUILD marlin_fp16
Marlin BF16 / same source GPTQMODEL_MARLIN_BF16_BUILD_ROOT GPTQMODEL_MARLIN_FORCE_REBUILD marlin_bf16
Machete / utils/machete.py GPTQMODEL_MACHETE_BUILD_ROOT GPTQMODEL_MACHETE_FORCE_REBUILD machete
Swordfish / utils/swordfish.py GPTQMODEL_SWORDFISH_BUILD_ROOT GPTQMODEL_SWORDFISH_FORCE_REBUILD swordfish
ParoQuant rotation / utils/paroquant.py GPTQMODEL_PAROQUANT_BUILD_ROOT GPTQMODEL_PAROQUANT_FORCE_REBUILD paroquant
Hadamard / utils/hadamard.py GPTQMODEL_HADAMARD_BUILD_ROOT GPTQMODEL_HADAMARD_FORCE_REBUILD hadamard

Diagnostics, quantization and process behavior

Variable Default / parser Behavior and consumer
DEBUG False; env_flag Package debug mode and importer support-map diagnostics; __init__.py, utils/importer.py. Not a promise of a global file log.
GPTQMODEL_LOG_TIMES False; env_flag Opt-in block/region timing messages; does not gate aggregate QuantizationRegionTimer recording/flushing. utils/logger.py.
GPTQMODEL_LOG_HESSIAN False; env_flag Additional GPTQ Hessian diagnostics; quantization/gptq.py.
GPTQMODEL_LOG_QUANT_STATS "0"; enabled unless empty, 0, false, or False Extra quantization statistics logging; looper/loop_processor.py. This parser is not case-normalized env_flag.
GPTQMODEL_LOG_CALIBRATION_SAMPLES 0; integer, lower-bounded at zero Preview that many prepared calibration samples; utils/calibration.py. Invalid non-integer text can raise. Use with care if calibration contains private text.
GPTQMODEL_PRINT_MODULE_TREE False Opt-in model-tree dump; accepts 1, true, yes, on, y, t; models/loader.py.
GPTQMODEL_DISABLE_GPTQ_CUDA False; env_flag Skip the optional GPTQ block CUDA fast path in GPTQ/QQQ when otherwise eligible; not a global CUDA disable.
GPTQMODEL_DISABLE_BITBLAS False; env_flag Skip BitBLAS import/version probing; useful when TVM import is unstable. Set before qlinear modules import.
GPTQMODEL_MARLIN_USE_FP32 True; env_flag GPTQ Marlin FP32 reduction setting, captured by the linear instance.
GPTQMODEL_FP32_ACCUM True; env_flag FP32 accumulation selection in AWQ CUDA/Triton and ParoQuant Triton GEMM paths. Turning it off is a numerical change, not just a logging/performance change.
GPTQMODEL_USE_TORCH_REPLICATE True; env_flag Chooses Torch replication support in looper helpers; module-global setting.
GPTQMODEL_POOL_PREWARM True; env_flag Best-effort prewarm of the device worker pool during model entry-point setup.
GPTQMODEL_LINEAR_PREWARM True; env_flag Best-effort background linear-library prewarm; utils/linalg_warmup.py.
GPTQMODEL_QUANTIZE_PREWARM True; env_flag Best-effort quantization-library prewarm; same source. Unsupported devices/dtypes and failures are skipped/debug-logged.
GPTQMODEL_CPU_WORKERS Auto; integer override clamped to at least one CPU device-pool workers. Default is min(8, max(2, cpu_count // 16)) without the GIL and min(12, max(1, (cpu_count + 1) // 2)) with it; unavailable CPU count is treated as one. Invalid overrides fall back.
GPTQMODEL_MAX_PACKERS GIL: 1; free-threaded: min(8, max(2, cpu.count * cpu.cores)) Parallel module packing; integer override at least one; invalid values fall back; utils/model.py.
GPTQMODEL_DISABLE_PACK_EXT False; env_flag Skip native block pack extension and use the Python path.
GPTQMODEL_FORCE_PACK_EXT False; env_flag Re-raise an eligible native pack attempt's error rather than falling back. Does not override DISABLE_PACK_EXT or make unsupported bit widths eligible.
GPTQMODEL_PACK_THREADS Packer's positive workers argument, otherwise 1 Native pack-block threads; valid integer overrides clamp to at least one. Invalid text warns and retains the existing count.
GPTQMODEL_PACK_PY_THREADS 1 Python fallback pack workers; clamped to 1..total_blocks, invalid text warns and remains sequential. Separate from native packing threads.
GPTQMODEL_RESUME Unset Exact value "1" raises: legacy resume is retired. Use CheckpointConfig, not marker/offload files.
GPTQMODEL_USE_MODELSCOPE False; env_flag Enable ModelScope repository resolution; missing ModelScope produces an installation error. See utils/modelscope.py.
GPTQMODEL_DISABLE_GIL_WARNING Unset Exact "1" suppresses GIL diagnostic messages; does not change the interpreter's GIL.
PYTHON_GIL Interpreter-owned 0 requests disabled GIL on a supporting free-threaded Python build. Not a switch that makes ordinary Python free-threaded.
GPTQMODEL_ROCM_QUANT_BLAS "rocblas" Scoped quantization BLAS override when ROCm is using hipBLASLt; maps rocblas to Torch's hipblas name and restores the previous backend afterward. Empty/0/off/default/hipblaslt/cublaslt opt out. Unsupported platforms/API failures are skipped.
GPTQMODEL_NPU_CHOLESKY_BLOCK_SIZE 64; direct integer conversion NPU Cholesky block size; quantization/npu_linalg.py.
AWQ_BATCH_SIZE 1 in the utility's fallback Set from AWQ/ParoQuant quantization batch_size and read for post-init sizing. Prefer the public batch_size argument to manual environment changes.
BITBLAS_ENABLE_TUNING Unset: enabled for non-repack, disabled for GPTQ repack If set, values other than trimmed case-insensitive 0, false, no, off enable tuning; utils/bitblas.py.
TVM_TARGET Set by BitBLAS initialization Receives the normalized detected CUDA target. A user value is not a supported replacement for GPT-QModel's target-selection checks.
GPTQMODEL_AWQ_TRITON_SCHEDULE "auto", lowercased AWQ Triton scheduler mode: auto or legacy; other values raise in the scheduling entry point. quantization/awq/modules/triton/scheduler.py.

ParoQuant flags

These are implementation diagnostics/tuning controls, not substitutes for ParoQuant configuration. The env_flag rules apply except where indicated.

Variable Default Behavior / source
GPTQMODEL_PAROQUANT_ROTATE_AUTOTUNE True Fused rotation launch tuning. Uses a permissive parser: only 0, false, no, off are false after trimming/lowercasing; even an empty value enables it. utils/paroquant.py.
GPTQMODEL_PAROQUANT_AUTOTUNE_CACHE True Persistent rotation autotune cache; hardware/driver/Torch-specific identity.
GPTQMODEL_PAROQUANT_ROTATE_CTA_M Unset; 4, 8, 16 Explicit fused-rotation launch override. Invalid/non-integer values are ignored.
GPTQMODEL_PAROQUANT_ROTATE_ROW_PAD Unset; 0, 2 Explicit row-padding launch override. Invalid values are ignored; either valid launch override takes precedence over autotuning.
GPTQMODEL_PAROQUANT_CACHE_RUNTIME_DTYPE False Allow typed runtime metadata caching in the qlinear path.
GPTQMODEL_PAROQUANT_AUTO_CACHE_BF16_RUNTIME_DTYPE True Automatically cache runtime metadata for BF16.
GPTQMODEL_PAROQUANT_CACHE_ROTATION_DTYPE False Allow typed rotation metadata caching.
GPTQMODEL_PAROQUANT_AUTO_CACHE_BF16_ROTATION_DTYPE True Automatically cache BF16 rotation metadata.
GPTQMODEL_PAROQUANT_CPU_AVX512 True Permit the eligible CPU AVX512 packed INT4 path; actual CPU/kernel checks still apply.
GPTQMODEL_PAROQUANT_AFFINE_STE_TRITON True Enable eligible Triton affine straight-through-estimator forward path.
GPTQMODEL_PAROQUANT_AFFINE_STE_TRITON_BACKWARD False Enable its optional Triton backward path.
GPTQMODEL_PAROQUANT_SYM_STE_TRITON True Enable eligible Triton symmetric STE path.
GPTQMODEL_PAROQUANT_OPT_FUSED_ROTATION True Permit fused optimizer rotation when other eligibility checks pass.
GPTQMODEL_PAROQUANT_OPT_LARGE_TRAIN_QUANT_COMPILE True Permit compilation of the eligible large training quantization path.
GPTQMODEL_PAROQUANT_OPT_STAGE_CUDAGRAPH True Permit eligible stage CUDA-graph execution.
GPTQMODEL_PAROQUANT_TRITON_AUTOTUNE "1"; only exact "0" disables Triton qlinear autotune; not env_flag.
GPTQMODEL_PAROQUANT_TRITON_AUTOTUNE_WARMUP 2, clamped >=0 Triton tune warmup count.
GPTQMODEL_PAROQUANT_TRITON_AUTOTUNE_ITERS 4, clamped >=1 Timed tune iterations.
GPTQMODEL_PAROQUANT_TRITON_AUTOTUNE_MARGIN 0.05, clamped >=0 Relative tuning selection margin.
GPTQMODEL_PAROQUANT_TRITON_DECODE_MAX_ROWS 8, clamped >=1 Decode-path row bound.

The final four numeric controls use direct int/float conversion during module initialization; invalid text can prevent import. Runtime metadata caches do not authorize persistent dense or expanded weight caches. AGENTS.md requires quantized persistent weight storage for every backend and fallback.

CPU, Torch and GGUF diagnostics

Variable Default / parsing Behavior
GPTQMODEL_CPU_BF16X3 True; env_flag Allow eligible oneDNN BF16x3 FP32 matmul; requires the checked CPU ISA and API; utils/cpu_features.py.
GPTQMODEL_EXL3_CPU_BF16X3 True; env_flag EXL3-specific BF16x3 eligibility; nn_modules/exllamav3_torch.py.
GPTQMODEL_BITSANDBYTES_CPU_PACKED False; env_flag Opt-in CPU packed BitsAndBytes implementation eligibility.
GPTQMODEL_BITSANDBYTES_CPU_INT8_DENSE True; env_flag Named CPU INT8 fallback eligibility in qlinear/bitsandbytes.py; this name is an implementation switch, not an exception to the quantized-storage rule.
GPTQMODEL_FLOATX_CPU_THREADS Usually 8; 32 for eligible large calls with at least 64*1024*1024 elements Thread count capped to 1..min(32, os.cpu_count() or 1); invalid text uses auto default; quantization/dtype.py.
GPTQMODEL_FLOATX_CPU_FORCE_NATIVE_FP8 Off; accepts 1, true, yes, on Force the native CPU FP8 conversion candidate before normal size-based eligibility; not env_flag.
GPTQ_TORCH_STREAMING 0; bool(int(...)) Torch qlinear streaming mode.
GPTQ_TORCH_STREAM_TILE 512; integer Output-column streaming tile size.
GPTQ_TORCH_TRITON_DEQUANT Auto If unset, follows Triton availability; explicit 0, false, False disable, other values permit it only if Triton dequant is available.
GPTQ_TORCH_ROCM_GEMV, GPTQ_TORCH_ROCM_GEMM 1; bool(int(...)) Permit eligible ROCm Torch-backend optimized GEMV/GEMM paths.
GPTQ_TORCH_CACHE_WEIGHTS, GPTQ_TORCH_LOOKAHEAD 0; bool(int(...)) Retired compatibility switches in qlinear/torch.py. A nonzero value calls a shim that emits FutureWarning and leaves caching/lookahead disabled. They do not enable dense weight reuse or speculative dense prefetch in this revision.
GPTQMODEL_GGUF_CPP_THREADS Torch thread count GGML CPU thread override, clamped to at least one; invalid text falls back.
GPTQMODEL_GGUF_CPP_NATIVE_MATMUL Auto benchmarks GGML versus Torch per shape/dtype/qtype Explicit values other than 0, false, off enable native candidate; unsupported qtypes remain excluded. Failures fall back to Torch.
GPTQMODEL_GGUF_FUSED_CUDA_MAX_ROWS 32, >=0 CUDA fused-path row threshold.
GPTQMODEL_GGUF_FUSED_CUDA_MIN_MATRIX_ELEMENTS 8388608, >=0 CUDA fused-path matrix-size threshold.
GPTQMODEL_GGUF_FUSED_CPU_MAX_ROWS 64, >=0 CPU fused-path row threshold.
GPTQMODEL_GGUF_FUSED_CPU_MIN_MATRIX_ELEMENTS 0, >=0 CPU fused-path matrix-size threshold.
GPTQMODEL_GGUF_FUSED_CHUNK_BLOCKS 8, >=1 Fused dequantization chunk size in blocks.
GPTQMODEL_GGUF_FUSED_AUTOTUNE_WARMUP 1, >=0 Fused-path tuning warmups.
GPTQMODEL_GGUF_FUSED_AUTOTUNE_ITERS 2, >=1 Tuning iterations.
GPTQMODEL_GGUF_FUSED_AUTOTUNE_MARGIN 0.05, >=0 Tuning margin.
GPTQMODEL_INTERNAL_GGUF_QUANTIZED_LOADER True; false-list includes empty/0/false/off/no Internal quantized GGUF loader; utils/internal_gguf.py.
GPTQMODEL_INTERNAL_GGUF_TORCH_LOADER Unset: enabled when dequant-device override is nonempty Internal Torch GGUF loader selection; explicit false-list empty/0/false/off/no disables; utils/hf.py.
GPTQMODEL_INTERNAL_GGUF_DEQUANT_DEVICE Unset Internal GGUF dequant device; accepts available CUDA ordinals or CPU; invalid/unsupported devices return no override.
GPTQMODEL_INTERNAL_GGUF_DEQUANT_MAX_BYTES 268435456 bytes (256 MiB) Chunked Torch GGUF dequant staging budget; invalid integer text falls back. This is an internal loader control, not a general quantization VRAM cap.

Integer-based Torch flags accept numeric strings, not true/false. GGUF fused numeric controls use direct conversions; avoid invalid text. For user-facing GGUF loading and format requirements, use GGUF, not internal loader switches as a public configuration API.

Visibility, allocator and logging environment

Variable Role
CUDA_VISIBLE_DEVICES CUDA visibility/logical indexing. Set before Python/Torch initialization; optimize/_common.py also writes and verifies it for its CLI.
ROCR_VISIBLE_DEVICES ROCm visibility variable considered by optimizer GPU setup. models/auto.py deletes it when both it and CUDA_VISIBLE_DEVICES are present, preferring the CUDA visibility setting to avoid the ROCm conflict.
CUDA_DEVICE_ORDER models/auto.py sets PCI_BUS_ID if the current value is missing or empty, preserving a nonempty explicit value. CI also selects PCI bus ordering.
PYTORCH_ALLOC_CONF models/auto.py sets expandable_segments:True,max_split_size_mb:256,garbage_collection_threshold:0.7 if missing or empty, before that module's Torch import. This cannot guarantee application startup order if Torch was already imported elsewhere. Set explicitly before process startup for reliable effect; it cannot create additional VRAM.
LOGBAR_ANIMATION LogBar's animation control; CI sets 0. It is not the logger's sole pytest-capture detection mechanism.
PYTEST_CURRENT_TEST Pytest-owned signal used with non-TTY stdout to suppress live redraw during test capture.
GPU_ALLOCATOR_URL CI/optimization GPU-allocation service setting, not a qlinear backend or local device-selection API.

Kernel build failures

What to collect first

Preserve the first compiler, missing-header, linker, import, or missing-operator error. A later “kernel unavailable” or “using fallback path” line may only be the consequence. Record:

  • Python executable/version, Torch version, torch.version.cuda/torch.version.hip, compiler/toolkit version, driver, GPU/capability.
  • The extension name, requested activation dtype and backend.
  • Cache/build root, relevant overrides, target architectures, and write permissions.
  • Whether the same failure occurs in a fresh process with one supported visible GPU.

The common loader logs extension build diagnostics including base/actual root, Python/PID, TORCH_EXTENSIONS_DIR, GPTQMODEL_TORCH_EXTENSIONS_DIR, candidate binaries and build entries. CPU, CUDA and HIP source sets are not interchangeable.

Inspect or rebuild an extension without loading a model

The public named extension API in gptqmodel/extension.py includes these exact signatures:

def available_extensions() -> tuple[str, ...]:
def is_available(name: str, *, use_cache: bool = True) -> bool:
def error(name: str) -> str:
def load(name: str = "all", *, use_cache: bool = True) -> dict[str, bool]:

These are signature-only reference lines, not a runnable block of function definitions. Load one extension to isolate its diagnostics:

GPTQMODEL_EXT_VERBOSE=1 python - <<'PY'
from gptqmodel import extension
print(extension.available_extensions())
try:
    print(extension.load("marlin_fp16"))
finally:
    print(extension.error("marlin_fp16"))
PY

is_available() can load/JIT; it is not a zero-work static package probe. marlin is a group name for the FP16 and BF16 extensions, while availability/error/operator queries target concrete names. load() raises if a selected extension fails. use_cache=False clears cached build artifacts only before the library is loaded into the current interpreter; afterward it raises and requires restarting Python. It does not change the module's supported hardware.

For one diagnostic rebuild:

GPTQMODEL_KERNEL_REBUILD=1 GPTQMODEL_EXT_VERBOSE=1 python - <<'PY'
from gptqmodel import extension
try:
    print(extension.load("marlin_fp16"))
finally:
    print(extension.error("marlin_fp16"))
PY

Do not delete shared build directories while other processes are compiling/loading them. The shared loader serializes cross-process builds with an OS lock; exiting/killing a process releases that lock. A timeout can mean another legitimate build is still running. Increase the timeout only after inspecting the owner/workload; arbitrarily removing a lock file can defeat synchronization.

Common build errors

Error category Action
Compiler/Ninja absent or C++ standard unsupported Check the active environment and compiler/toolkit installation. The package requires Ninja, but it does not ship a complete host C++ compiler or NVCC/HIP compiler. Shared JIT flags select C++20 for Torch at least 2.14 and C++17 for older Torch; check the first compiler error against that requirement.
Wrong CUDA/HIP/toolkit Align the active Torch build with the intended compiler/toolkit. Do not repair a CUDA build by pointing at a ROCm directory or vice versa.
Missing runtime/cuBLAS/cuSPARSE/cuSOLVER headers Use the matching package extra/toolkit; see Marlin below. Runtime availability alone does not prove headers are installed.
Unsupported GPU target Inspect compute capability and actual kernel support. A manual TORCH_CUDA_ARCH_LIST is a build request, not an emulation layer.
Undefined symbol or incompatible cached binary Capture the symbol and environment versions, then rebuild the affected extension in a new process with matching Torch/toolchain/ABI. Shared JIT fingerprints help separate incompatible builds but cannot validate an arbitrary user-supplied prebuilt binary.
Permission/disk failure Select a writable, sufficiently large cache root; preserve build logs before cleanup.
Build OOM Reduce external build concurrency or target breadth as appropriate and inspect host RAM. NVCC_THREADS controls compiler threads, not quantization batch size or all Ninja concurrency.
Optional extension fails but program continues Determine which fallback actually ran and whether its performance/device/format is suitable; a warning is not proof the requested accelerated kernel executed.

Marlin and BitBLAS

Marlin is unavailable or tries to compile repeatedly

Current utils/marlin.py checks NVIDIA CUDA rather than ROCm, and supports compute capability at least 7.5. Do not apply an older universal “Ampere-only” statement to this source revision. The actual device is validated rather than freezing support from whichever GPU happened to be current during import.

FP16 and BF16 are separate extensions and build roots. A successful FP16 build does not prove the BF16 library is available. Compute capability 7.5 requires FP16; BF16 is rejected there. GPTQ Marlin supports symmetric 4/8-bit weights with group sizes -1, 32, 64, 128, subject to its other declarations and device/shape checks. Packed K/N dimensions must be divisible by 32 // bits, and effective activation-order weights require aligned thread tiles; ordinary tile padding cannot repair those cases.

AUTO and AUTO_TRAINABLE deliberately reject Marlin's tile-misaligned shapes so selection can try another backend. Explicit GPTQ_MARLIN can enable runtime tile padding for otherwise valid non-activation-ordered shapes. Padding does not waive packing divisibility, group boundaries, dtype, hardware, or other validation requirements. desc_act=True with a single group (group_size=-1 or group_size=in_features) is treated as no effective activation ordering.

GPTQMODEL_MARLIN_USE_FP32 defaults to true for the GPTQ Marlin class. Disabling it changes reduction precision; first reproduce with the default before attributing numerical differences to weight quantization.

For missing CUDA headers, the source's install hint selects:

Torch CUDA major Extra Dependencies at this revision
12 marlin-cuda12 Runtime 12.9.79, cuBLAS 12.9.2.10, cuSPARSE 12.5.10.65, cuSOLVER 11.7.5.82 CUDA-12 wheels.
13 or later marlin-cuda nvidia-cuda-runtime>=13.4.92, nvidia-cublas>=13.8.1.7, nvidia-cusparse>=12.8.6.72, nvidia-cusolver>=12.3.5.3.

For example, choose the one matching the active Torch CUDA build, not both blindly:

python -m pip install 'gptqmodel[marlin-cuda12]'

The wheels provide headers/libraries, not a general replacement for NVCC and the host compiler. Repeated compilation can also result from GPTQMODEL_KERNEL_REBUILD=1, GPTQMODEL_MARLIN_FORCE_REBUILD=1, changing fingerprints/toolchains, nonpersistent caches, or unwritable build directories.

Do not select format="marlin" for new quantization. BaseQModel.quantize() rejects this deprecated export format; use a compatible GPTQ checkpoint and select/allow the Marlin inference backend.

BitBLAS import/JIT/tuning fails or appears to hang

The bitblas extra pins 0.1.0.post1; runtime probing requires at least that version. _is_bitblas_available() catches import errors, attempts to preload CUDA libraries when the error mentions libcu, retries import, and verifies the version. Successful package installation alone does not establish TVM/CUDA runtime availability.

Set GPTQMODEL_DISABLE_BITBLAS=1 before import to skip unstable probing and try another compatible backend. This does not make explicit BitBLAS selection succeed; it marks BitBLAS unavailable.

The GPTQ BitBLAS class supports desc_act=False, bits 2/4/8, group sizes -1/32/64/128, and both symmetry values. It does not support activation-order GPTQ just because another backend does.

Initialization writes TVM_TARGET from its normalized detection and uses a database path suffixed with the BitBLAS version. This revision caps the normalized target SM at 90, logging a compatibility fallback on newer architectures. That fallback is not proof that every newer GPU executes correctly; include the target and GPU in reports.

GPTQ checkpoint repacking disables expensive tuning by default unless BITBLAS_ENABLE_TUNING explicitly enables it. A native serialized/non-repack load defaults differently. Distinguish import, compilation, target selection, repack, database access, and tuning time when reporting an apparent hang.

Why does no backend support my checkpoint?

Method, format, kernel and device are different settings

METHOD identifies the quantization family. FORMAT identifies the saved tensor layout. BACKEND selects execution. The selector normalizes legacy backend names with the method, validates candidate qlinear classes, and applies auto-selection priorities. It does not reinterpret arbitrary packed tensors as another format.

The following is a focused compatibility reference from class declarations, not a complete substitute for shape/device/dtype/packing/runtime validation:

GPTQ backend Bits group_size desc_act sym Device / activation dtype
GPTQ_MARLIN 4, 8 -1, 32, 64, 128 False, True True only NVIDIA CUDA; FP16/BF16, but FP16 only on capability 7.5; Linux and shape/capability checks.
GPTQ_EXLLAMAV2 4 -1, 32, 64, 128 False, True False, True CUDA; FP16/BF16. Source excludes ROCm because of accuracy issues.
GPTQ_TRITON 2, 4, 8 -1, 16, 32, 64, 128, 256, 512, 1024 False, True False, True CUDA; FP16/BF16. XPU is not declared supported by this class.
GPTQ_BITBLAS 2, 4, 8 -1, 32, 64, 128 False only False, True CUDA; FP16/BF16, with BitBLAS dependency/runtime checks.

All four declare GPTQ/GPTQ_V2 format support; Marlin also declares legacy MARLIN, and BitBLAS declares BITBLAS. Their declaration of a format does not mean all are valid raw quantization packers.

Examples outside GPTQ:

Class / family Bits and groups Ordering/symmetry Important constraints
AwqGEMMLinear / GEMM 4; -1,16,32,64,128 Both declared values CUDA; FP16/BF16.
AwqGEMVLinear / GEMV 4; -1,64,128 Both declared values CUDA/ROCm declarations; FP16 only.
AwqGEMVFastLinear / GEMV_FAST 4; -1,16,32,64,128 Both declared values CUDA/ROCm declarations; FP16; format-specific int16 packing.
QQQLinear / QQQ 4; -1,128 desc_act=False, sym=True Linux CUDA/ROCm declaration; NVIDIA capability at least 8.0; FP16/BF16 activation declarations, with the native forward converting to FP16 internally and restoring the input dtype. Portable QQQTorchLinear also exists.

What do the three common parameters change?

Parameter Meaning and practical implication
group_size Grouping of quantization parameters; -1 is the channel-wide grouping convention. Backend support lists are discrete. 128 is not universally interchangeable with 64, nor can a saved checkpoint be changed by editing JSON.
sym Symmetric versus asymmetric quantization/zero-point semantics. Marlin's GPTQ class requires symmetric weights; BitBLAS/Triton/ExLlama v2 have broader declarations.
desc_act GPTQ activation-order handling and associated indexing/permutation requirements. A kernel that does not support it cannot be fixed by relabeling the metadata.

Current base config defaults are bits=4, group_size=128, sym=True, and desc_act=None before concrete-class normalization. GPTQConfig.default_desc_act() resolves it to false. GPTQ's act_group_aware defaults to true and cannot be true together with desc_act=True; explicitly enabling desc_act with no explicit act_group_aware automatically disables the latter with a warning. Both explicitly true raises.

When diagnosing selection, include dimensions, activation dtype, packing dtype, dynamic overrides, adapters/training, format, platform, and optional-package availability. “Four-bit CUDA” is insufficient. For a controlled isolation, try BACKEND.AUTO and then a documented compatible explicit backend without modifying checkpoint metadata. See Inference Backends and Kernel Backends Reference.

CUDA OOM during quantization

Why can a four-bit output model need much more memory to quantize?

Quantization works with original weights, forward activations, method-specific statistics, temporary solver tensors, packing buffers, and sometimes multiple devices/replicas. Inference checkpoint size is not a peak-quantization-memory estimate. Identify the stage from the traceback and optional timing/Hessian/telemetry logs before choosing a remedy.

Setting Actual default Diagnostic action
quantize(..., batch_size=...) 1 Reduce a larger batch to one. Models that disallow batch quantization force one.
calibration_concat_size None Bound calibration sequence construction deliberately; changing it changes calibration data/quality and checkpoint identity.
calibration_sort "desc" Sort supports asc, desc, shuffle; ordering affects scheduling and checkpoint calibration identity.
calibration_data_min_length 10 Filters short examples; not a peak-memory limit.
offload_to_disk True Keep enabled for the supported disk-offload lifecycle; use a writable disk with sufficient capacity. Required for checkpointing.
offload_to_disk_path None before normalization Auto-generated when offload is enabled and no path was provided. Use explicit durable paths for deliberate operational control.
calibration_data_device None Review whether retained calibration is consuming scarce accelerator memory.
auto_forward_data_parallel True Review multi-device forward replicas and memory use; disable for a controlled single-device diagnosis.
dense_vram_strategy VramStrategy.EXCLUSIVE exclusive and balanced are valid strategies; choose intentionally for the dense pool rather than assuming another GPU's free memory is automatically available.
wait_for_submodule_finalizers False Waiting for packing/offload finalizers before advancing can reduce overlap-related VRAM pressure.
HessianConfig.chunk_size, chunk_bytes None, None Bound GPTQ Hessian staging chunks. They do not remove the full method's statistics/solver requirements.
HessianConfig.staging_dtype torch.float32 FP16/BF16 are accepted, but lower staging precision can change numerical results. Validate quality rather than treating it as free compression.
HessianConfig.dedup_shared_inputs True Capture explicitly tagged shared-input Hessians once; requires correct model-tree shared-input declarations.
MSEConfig.grid_budget, clip_grid_budget None, None; each resolves to 16 MiB on CPU or 256 MiB on other devices Chunk-sizing targets for GPTQ/QQQ MSE shrink-grid scratch and AWQ clipping scratch respectively. Accept nonnegative byte integers or strings such as "64M", "1G"; 0 selects single-candidate chunks and, for AWQ, one-token product tiles. These are not total-memory caps: multiple scratch buffers or even a minimum tile can exceed one budget.
AWQConfig.scale_search_chunked_activations True Stream/chunk the AWQ scale-search activation work rather than retaining full reference/reconstruction activation working sets. Separate from clip-grid workspace control.

For example, this configuration uses real fields while leaving model selection and calibration to the application:

from gptqmodel import GPTQConfig
from gptqmodel.quantization.config import HessianConfig, VramStrategy

quant_config = GPTQConfig(
    bits=4,
    group_size=128,
    offload_to_disk=True,
    offload_to_disk_path="offload/my-run",
    auto_forward_data_parallel=False,
    dense_vram_strategy=VramStrategy.EXCLUSIVE,
    hessian=HessianConfig(chunk_bytes=64 * 1024 * 1024),
)
# Load the original model with this configuration, then use:
# model.quantize(calibration_dataset, batch_size=1)

64 MiB here is an example staging budget, not a source default or a total VRAM cap. Reducing sample count/length is another resource tradeoff, but changes calibration and possibly quality. Do not silently change data when retrying a checkpointed job.

For search-workspace failures, pass budgets through the config's mse object, for example MSEConfig(grid_budget="64M", clip_grid_budget="64M") from gptqmodel.quantization.config, rather than inventing an environment flag. Preserve the intended norm value: its default is 0.0, which disables GPTQ/QQQ shrink-grid search, so replacing an existing MSE config with a default-norm object changes the algorithm as well as workspace sizing.

Also check other processes, device visibility, host RAM/disk pressure, unexpected replicas, and stale application references. PYTORCH_ALLOC_CONF=expandable_segments:True may help allocation behavior on supporting builds; it cannot solve a genuinely oversized live working set. torch.cuda.empty_cache() cannot free tensors still referenced by your program.

The loader can disable disk offload for model definitions with support_offload_to_disk=False and for the xielu activation construction path. Inspect the effective normalized config and loader warning rather than assuming an initial offload_to_disk=True was retained for every architecture.

See Memory Offload and Multi-GPU and Quantization Lifecycle.

Device selection problems

Set visibility before importing Torch/GPT-QModel. After CUDA_VISIBLE_DEVICES=2, the visible physical GPU is generally logical cuda:0, not cuda:2. Use the logical device indices visible inside the process when constructing maps. Avoid changing visibility midway through a job or between checkpoint attempts.

The selector accepts device information through normalization helpers, including torch.device, strings, the project's DEVICE values, integer indices, and device-map values. Integer values refer to the active accelerator namespace (CUDA/ROCm, XPU, or NPU), not always CUDA; without an active accelerator they raise. An automatic/unspecified hint is not a fixed physical-GPU assignment. Invalid, unavailable, out-of-range, mixed-family, or backend-incompatible devices must be diagnosed from the actual normalized map.

Problem Explanation / action
Several GPU families in one map Importer validation rejects maps spanning different accelerator families. CPU offload plus one accelerator family is not equivalent to combining CUDA/XPU/MPS/NPU families.
Different GPU compute capabilities in one map The selector also rejects heterogeneous CUDA/ROCm capability tuples, even within one accelerator family. Report each visible device's capability, not only its family.
ROCm shown as CUDA in Torch Expected API convention; GPT-QModel checks IS_ROCM and backend support separately. Do not select a CUDA-only extension based solely on torch.cuda.is_available().
CPU fallback unexpectedly used Inspect requested versus AUTO backend, optional import failures, actual tensor placement and device validation.
Wrong ordinal after masking Report original visibility variables and in-process enumerated names/capabilities.
Unsupported NPU or XPU operation Check the selected qlinear class and platform API support. hw/ascend_npu.md lists missing CUDA-equivalent capability/memory APIs.
Optimizer CLI waits for GPU optimize/_common.py has GPU allocation/idle-gate behavior beyond normal model loading. Check CLI arguments and allocator availability.

CUDA_DEVICE_ORDER and allocator defaults applied during import preserve existing user values and may be too late if Torch was imported first. Explicit process-start environment is more reliable than relying on import side effects.

trust_remote_code and Hugging Face loading

Remote code executes code supplied by a model repository. Enable it only after reviewing and trusting that repository/revision; do not automatically retry every failure with trust enabled.

The helper's exact signature is:

def resolve_trust_remote_code(model_id_or_path: Optional[str], *, trust_remote_code: bool) -> bool:

utils/hf.py first performs Qwen Drive registration. If trust is false or no model path is supplied, it preserves the value. If trust is true and the installed Transformers has native causal-LM support for the config's model type, it warns once per model/type/class and overrides trust to false. Otherwise it preserves true.

Therefore an “overriding trust_remote_code=True to False” warning can be intentional: installed native support is preferred over stale repository code. It does not mean the caller's argument was forgotten. Native support detection is tied to the installed Transformers version, not just the model repository name.

Symptom Action
Config/model type unknown to Transformers Check the model's required Transformers version, actual installed version and remote-code requirements; include config model type and traceback.
Remote code imports missing dependencies Install the model's documented dependencies in the same environment, or use native support when available.
Trust error on a local model directory Local paths can still reference HF dynamic modules through config auto_map; local storage is not a guarantee of no custom code.
Tokenizer/template mismatch after load Establish prompt/input-ID baseline and follow Tokenicer ownership rules below; trust is not a tokenizer-normalization fix.
Model version-dependent loader error loader.py calls check_versions(cls, cls.require_pkgs) for model-definition package constraints. Follow the concrete required package/version, not a generic downgrade recipe.

See Hugging Face Integration and Model Definitions and Module Tree.

Transformers, Optimum and engine version conflicts

Component Declared requirement in this revision
Python >=3.10
Build backend setuptools.build_meta, setuptools >=84.2.1,<85
Torch >=2.14.1
Transformers >=5.20.2
Accelerate >=1.15.0
Safetensors >=0.8.0
Tokenicer >=0.0.14
Optimum (hf extra) >=2.3.0; not a core requirement.
BitsAndBytes extra >=0.50.2
BitBLAS extra ==0.1.0.post1
Triton extra >=3.8.0, Python <3.15

These come from requirements.txt and pyproject.toml. There is no single checked-in universal Optimum/Transformers/external-engine compatibility matrix. The vllm and sglang extras explicitly carry their own dependency pins; use dedicated environments when they conflict. Do not repair a resolver conflict by installing with --no-deps and then assume the resulting environment is supported.

Recommended diagnosis:

  1. Confirm the Python executable used by the process and use python -m pip, not an unrelated pip.
  2. Run python -m pip check and record package versions.
  3. Distinguish GPT-QModel core loading, Optimum integration, and a separate engine's loader.
  4. Reproduce with the package's current declared dependencies in an isolated environment.
  5. If model-specific version checks conflict with broad package requirements, include both in the report. Do not invent a silent downgrade workaround.

CI pins and old changelog examples are not authoritative public dependency ranges. For example, the unit workflow's selected Torch can differ from requirements.txt; inspect the installed packages rather than copying a workflow label into a bug report.

Unknown or ignored quantization-config fields

There are several distinct cases:

Path Behavior
QuantizeConfig.from_quant_config(...) Lowercases keys, applies supported aliases, normalizes method/format, logs and ignores unknown top-level parameters.
Public QuantizeConfig(...) factory Routes to a concrete config class and filters payload to target-class init fields. A field from a different method can disappear rather than affect the result.
Nested configuration Some normalizers reject unsupported fields/types, and strict dataclasses can raise. Top-level ignore behavior is not universal.
Invalid known value Can raise validation errors for group sizes, formats, dtype, nested limits or conflicting settings.
Removed is_marlin_format Explicitly rejected; replace with an explicit format field. Legacy inspection and new export are different cases.
Manual format plus checkpoint format metadata from_quant_config rejects conflicting sources instead of silently overriding.

The serialized method field is quant_method; the public dataclass attribute is method. Use the current config API and inspect both the normalized object and serialized payload. Do not “fix” an incompatible checkpoint by editing desc_act, sym, group size, or format metadata without converting/requantizing the tensors.

The concrete config methods allowed_quant_methods() and supported_export_formats() are part of validation. Use QuantizeConfig Reference for supported fields and Packing Formats and Compatibility for layout conversion.

Checkpoint resume and interruption recovery

Use CheckpointConfig, not GPTQMODEL_RESUME=1

The current checkpoint config is a frozen dataclass with these public fields/defaults:

@dataclass(frozen=True)
class CheckpointConfig:
    path: str | Path = "auto"
    resume: str = "auto"
    interval: str = "layer:1"
    keep_last: int = 2
    strict_device_check: bool = True

The implementation additionally maintains an internal non-init _resolved_path; the fields above are the public initialization fields. Do not assign to fields after construction; create a new config for changed settings. Changing settings can still invalidate an existing checkpoint's identity.

Field Validation / meaning
path Nonempty path or "auto". Automatic location is derived from the existing offload path and retained on that config; use an explicit durable root across freshly loaded model/config objects.
resume="auto" Start if no published checkpoint exists; resume compatible progress; incompatible/corrupt published progress is an error, not silent restart.
resume="required" Require a compatible published checkpoint.
resume="never" Refuse a published checkpoint; use a new path for a deliberate fresh run.
interval layer:<positive integer> only, not seconds or arbitrary strings.
keep_last Integer at least two; retained published generations, not a cap on all attempt directories.
strict_device_check Boolean; false relaxes physical UUID/serial matching only, not GPU count/order/type, algorithm/data, or other identity checks.

Minimal recovery pattern

This pattern follows the repository's checkpoint.md; model_id and calibration_dataset must be the same original source/data on every attempt:

from gptqmodel import CheckpointConfig, CheckpointStopped, GPTQConfig, GPTQModel

quant_config = GPTQConfig(bits=4, group_size=128, offload_to_disk=True)
model = GPTQModel.load(model_id, quant_config)
try:
    model.quantize(
        calibration_dataset,
        batch_size=1,
        checkpoint=CheckpointConfig(
            path="checkpoints/my-run",
            resume="auto",
            interval="layer:1",
            keep_last=2,
        ),
    )
except CheckpointStopped:
    raise SystemExit(75)
model.save("quantized-output")

Do not save a final model from the stop handler. Reload the original model and resume, then save only after normal completion. A checkpoint directory is not an exported model and an attempt-* directory is not the original source.

Current support and identity restrictions

looper/gptq_checkpoint.py supports model types llama and qwen3_moe and exact processor chains: GPTQ, AWQ, QQQ, or ParoQuant, each alone or followed by EoRA; weight-only alone; and EXL3's own processor alone. A method enum alone does not guarantee that its chosen processor chain has a continuation adapter.

Checkpointing requires:

  • offload_to_disk=True, sequential quantization, Python main thread for signals, and a local safetensors source model.
  • No dynamic overrides/exclusions, GPTAQ/FOEM, rotation preprocessing, embedding/lm-head quantization, or unsupported adapter types. Supported adapter handling is specifically the checked LoRA type and processor chains.
  • The same source/configuration/calibration/runtime/packing/execution topology identity on resume.

Source safetensors and config are hashed. Calibration is prepared again and checked; completed transformer layers are not replayed merely to validate identity. Physical GPU checks are only one portion of identity. Using strict_device_check=False is not a general “ignore compatibility” flag.

Stops, crashes and storage

Event / artifact Behavior
SIGINT / Ctrl+C or SIGTERM Request a safe stop; finish current layer/finalizers, publish checkpoint, raise CheckpointStopped. Not immediate cancellation.
SIGKILL, crash, host failure Only the previously published generation survives; later work is repeated.
Partial layer/subset Re-run the uncommitted layer; partial Hessians are not resumed.
Ordinary exception No partial-layer checkpoint is synthesized.
CURRENT Atomically publishes retained generations; recovery does not scan unpublished directories and guess.
objects/ Content-addressed continuation/tensor objects, SHA-256 verified on recovery.
writer.lock One writer per root; separate concurrent jobs need separate roots.
Damaged newest generation Try retained generations newest to oldest; fail if none is complete.

Use a local filesystem supporting file locks, atomic rename and file/directory fsync. Object stores or network filesystems lacking those semantics do not provide the documented durability. Do not edit source weights, CURRENT, manifests, objects or attempts during a run.

For an incompatible-checkpoint error, restore the accidental difference or use a new checkpoint path for an intentional change. For corruption, preserve the root and restore a backup or start elsewhere if no retained complete generation exists. keep_last does not remove every private offload attempt; storage can remain large across retries. Remove the root only after a successful final save and when live quantization objects no longer need it.

For extra diagnostics, TelemetryConfig(device=True) records checkpoint preparation/validation/capture/commit/restore/failure metadata without tensor contents. Telemetry is excluded from algorithm identity, so it can be enabled just for a resume diagnosis. See checkpoint.md for the complete operational guide.

Where do logs go?

Log / artifact Location and behavior
General messages/progress LogBar-backed logger and process output. The common setup does not establish a universal rotating file-log directory. Capture stdout and stderr explicitly.
Processor records logs/<processor>_log_<random>_time_<timestamp>.log relative to the current working directory, from LoopProcessor._build_log_file_path(). Despite the docstring's “project-local” wording, the constant is Path("logs").
Quantization CSV <save_dir>/quant_log.csv when writer.py saves a quantized model with a nonempty quant_log. Columns are layer, module, loss, sample count, damping and processing time. Not every failing run reaches save.
JIT diagnostics Process logs plus the extension's recorded error/build directory; extension.error(name) exposes the error text.
CI per-test logs /opt/dist/GPTQModel/<run_id>/logs/<test_script>.log in the self-hosted test runner. Local pytest does not use this path automatically.
Timings GPTQMODEL_LOG_TIMES=1 enables opt-in block/region timing messages where instrumented and is checked at call time. QuantizationRegionTimer aggregates are recorded and periodically flushed independently of this flag.
Device telemetry Structured diagnostic records through the telemetry/logger path; not a promise of a separate persistent file.

For a reproducible capture:

set -o pipefail
GPTQMODEL_LOG_TIMES=1 GPTQMODEL_EXT_VERBOSE=1 python -u reproduce.py 2>&1 | tee run.log

This is a Bash example; pipefail preserves failure status through tee. Keep complete output, not only the final progress bar. Pytest capture suppression checks PYTEST_CURRENT_TEST and non-TTY stdout. LOGBAR_ANIMATION=0 can simplify CI output, but missing redraws are not evidence that no work ran.

Quality, tokenizer and evaluation issues

Before changing quantization settings, run the original dense model with the same tokenizer, template, calibration/evaluation tasks, sequence lengths, generation settings and dependency versions. Compare exact rendered prompt text, input IDs, attention masks, special tokens and template behavior as well as scores.

Reusable tokenizer/chat-template corrections belong in Tokenicer. AGENTS.md explicitly rejects accumulating GPT-QModel loader workarounds to avoid fixing that dependency. Task-specific few-shot/evaluation formatting belongs to the evaluator rather than a tokenizer normalization shim.

For kernel numerical reports, distinguish arithmetic error from final FP16/BF16 rounding, and identify both the kernel actually executed and the independent reference. A low perplexity score alone does not validate packing, every activation dtype, or fallback correctness. See Evaluation and Developer Guide.

Reporting a useful bug

Open an issue in ModelCloud/GPTQModel with a small, complete reproduction and the first failure. Include:

Category Required information
Source GPT-QModel release/version and commit for source installs; whether local edits are present.
Environment OS, Python executable/version/GIL mode, Torch build, Transformers, Accelerate, Tokenicer, Safetensors, relevant optional backend/engine versions, pip check output.
Hardware/toolchain GPU/accelerator model, in-process logical devices, capability, driver/toolkit/compiler versions, CUDA versus HIP build, host RAM/disk if relevant.
Model Model ID and revision or reproducible local layout, config model_type, trust setting, tokenizer/template revision, attention implementation.
Quantization Concrete config class and normalized settings: method, format, bits, group size, desc_act, sym, packing dtype, activation dtype, dynamic overrides, preprocessing/adapter options.
Workload Calibration source/order/count/length, batch size, concat/sort settings, device map, offload strategy, checkpoint settings, inference/generation shapes.
Backend Requested and actual selected backend/qlinear class, available extension status, JIT roots/flags, whether a fallback ran.
Failure/evidence Complete traceback and log from startup, expected versus observed behavior, minimal command/script, whether dense/reference execution succeeds.
Resume Preserved checkpoint error/identity differences, original environment, signal/failure type, published-generation status. Do not modify checkpoint metadata to make a report look compatible.

Include only relevant non-secret environment settings. Remove access tokens, private credentials, signed URLs and sensitive calibration/prompt text. Prefer a public or synthetic reproduction where possible. Do not upload proprietary weights or private datasets simply to demonstrate a failure.

Source files

  • gptqmodel/utils/env.py, logger.py, cpp.py, device_telemetry.py
  • gptqmodel/extension.py, gptqmodel/version.py, gptqmodel/__init__.py
  • gptqmodel/utils/marlin.py, machete.py, bitblas.py, paroquant.py, awq.py, qqq.py, vptq_cuda.py, hadamard.py, exllamav2.py
  • gptqmodel/utils/importer.py, backend.py, model.py, looper_helpers.py, linalg_warmup.py, rocm.py, cpu_features.py
  • gptqmodel/utils/hf.py, internal_gguf.py, modelscope.py, calibration.py, gptqmodel/utils/__init__.py
  • gptqmodel/models/auto.py, base.py, loader.py, writer.py, _const.py
  • gptqmodel/quantization/config.py, gptq.py, qqq.py, gptq_cuda.py, dtype.py, npu_linalg.py
  • gptqmodel/quantization/awq/modules/triton/, gptqmodel/quantization/paroquant/
  • gptqmodel/nn_modules/qlinear/__init__.py, torch.py, marlin.py, bitblas.py, exllamav2.py, tritonv2.py, gemm_awq.py, gemv_awq.py, gemv_fast_awq.py, qqq.py, paroquant.py, paroquant_triton.py, bitsandbytes.py, gguf.py, gguf_cpp.py
  • gptqmodel/nn_modules/exllamav3_torch.py, gptqmodel/exllamav3/ext.py
  • gptqmodel/looper/loop_processor.py, awq_processor.py, gptq_checkpoint.py, checkpoint_store.py, checkpoint.py
  • checkpoint.md
  • requirements.txt, pyproject.toml, AGENTS.md, .agents/skills/
  • optimize/_common.py, hw/ascend_npu.md, tests/pytest.ini
  • .github/workflows/unit_tests.yml, unit_tests_reusable.yml, .github/scripts/ci_tests.py, docs/changelog/archive.md

See also

Documentation home · Quickstart · Troubleshooting · Repository · PyPI

Source baseline: GPT-QModel 7.7.0 / c197691. Maintained by ModelCloud.ai.

Home

Getting started

Quantization methods

Complementary configs

Using a model

Platforms

Internals

Clone this wiki locally