Skip to content

GSQ

Jump to bottom
Qubitium edited this page Oct 28, 2026 · 4 revisions

GSQ (Gumbel-Softmax refinement and staged training)

Documentation home · Quickstart · Lifecycle

On this page

Overview

GSQ adapts the Gumbel-Softmax quantization ideas from the IST-DASLab GSQ paper (Dadgarnia et al.) and its reference implementation into two opt-in features that run during quantization:

  • Local scalar refinement (GSQConfig): after a supported method's ordinary quantizer produces an initial quantized layer, candidate code assignments are sampled through a Gumbel-Softmax relaxation and scored against a method-specific local reconstruction objective. The initializer is retained unless a hard, export-aware candidate improves that objective.
  • Staged block training (GSQTrainingConfig): for a materialized dense Llama model, decoder blocks are trained in reconstruction stages and each block is packed through an existing GPTQ or AWQ checkpoint format before the next block's inputs are captured.
  • Non-uniform bit allocation (RCO): staged blocks pack independently, so a checkpoint can mix widths across blocks. gptqmodel/quantization/rco.py ports the budget-constrained RCO search that produces the paper's 2.37/2.62 bits-per-parameter Llama rows, and the staged path accepts the resulting per-block width assignment through a restricted dynamic map.

GSQ is not a new runtime format: every path saves each method's existing scales, zero points, packing, and checkpoint layout. External runtimes and training integrations keep supporting only the formats they already accept. All options are off by default, they increase quantization time, and they are experimental adaptations: the paper's reported quality gains do not transfer automatically to any path. Measure downstream quality on your own workload before adopting GSQ broadly.

This page documents source revision 536b7f9b6af3c1e39f1e7a1b2e58a50d8793af4e. Constraint details are version-dependent; use the source matching your installed GPT-QModel version.

Workflow at a glance

Refine or retrain code assignments inside existing formats. Two opt-in paths: per-linear scalar refinement and staged Llama block training.

Reading the diagram: The two lanes are alternatives selected by different config fields; gsq refines per-linear outputs of an existing quantizer, while gsq_training replaces the ordinary quantize loop for a dense Llama model. Both lanes end at the base method's existing checkpoint format.

Text version of the workflow
  • Local scalar refinement: Calibrate and solve with the base method → Sample hard code candidates via Gumbel-Softmax → Keep improved codes → Pack as usual.
  • Staged Llama block training: Load a materialized dense Llama → Train each block's stages (Q/K, then V/O, then MLP) → Pack each block through GPTQ or AWQ → Save config and run metadata.

Open full-size SVG

Concepts

What the two paths change

gsq and gsq_training are mutually exclusive on one config: validation raises when both are enabled. Local refinement still runs the base quantizer first and then perturbs the resulting hard assignments, so the base method's solve remains the initializer. Staged training is a separate loop inside quantize() — it dispatches before the ordinary looper pipeline, so ordinary processors (rotation, smoothing, GPTAQ, FOEM, adapters, preprocessors) are not silently applied on that path.

Local objective, not end-to-end training

Per-linear refinement optimizes each calibrated nn.Linear independently against its captured calibration statistics — a calibration Hessian for GPTQ, captured moments for QQQ, native weight-only or calibration-aware objectives for FP8. It is not the paper's full block-training schedule and not a claim that downstream perplexity improves; it changes only the stored hard codes (and optionally stored scales where learn_scales is allowed).

Module selection

GSQConfig.modules is an optional tuple of regular expressions matched with pcre.search against the module's full name; a None value refines every eligible module. On the GPTQ processor path a dynamic override can also set gsq per module pattern (see QuantizeConfig reference):

dynamic={
    r".*self_attn\.(q_proj|k_proj)$": {"gsq": {"enabled": True, "steps": 50}},
    r"-:lm_head": {},                      # excluded entirely
}

Local scalar refinement

Enable it on the method's config. Every method accepts either a GSQConfig or a dictionary:

from gptqmodel.quantization import GSQConfig, QuantizeConfig

quant_config = QuantizeConfig(
    bits=4,
    group_size=128,
    gsq=GSQConfig(enabled=True, steps=100, candidates=16, learn_scales=False),
)

For GPTQ, local GSQ applies to calibrated nn.Linear layers in the scalar FORMAT.GPTQ, FORMAT.GPTQ_V2, and FORMAT.GPTQ_P exports, at 2–8 bits and all GPTQ group sizes. The GPTQ path requires calibration activations and rejects tensor-parallel padded projections; when the data-independent fallback policy triggers, refinement is skipped and gsq_diagnostics records {"status": "skipped", "reason": "data_independent_fallback"}.

max_candidate_bytes bounds the code-candidate bank per output-row chunk (64 MiB by default); the optimizer and temporary tensors need additional memory beyond that bound. learn_scales=True tunes stored scales where the method supports it — currently the GPTQ/AWQ affine paths; QQQ, FP8, and ParoQuant validation rejects it.

Per-method adapter constraints

Config Where refinement applies Validation constraints when gsq.enabled
GPTQConfig Calibrated nn.Linear layers format in GPTQ, GPTQ_V2, GPTQ_P; mock_quantization off; no gptaq/foem.
AWQConfig AWQ scalar export format in GEMM, GEMV, GEMV_FAST, LLM_AWQ; bits must be 4; non-GEMM formats also require group_size in -1, 32, 64, 128.
RTNConfig Scalar GPTQ or AWQ export format in GPTQ, GPTQ_V2, GEMM, GEMV, GEMV_FAST, LLM_AWQ; AWQ-family formats require bits 4.
QQQConfig QQQ packing format must be FORMAT.QQQ and learn_scales must stay False. Staged gsq_training is unsupported. Non-nn.Linear or TP-padded modules skip refinement.
FP8Config FP8 weight export learn_scales unsupported (fixed inverse scales required); float8_e8m0fnu is rejected — GSQ needs finite E4M3/E5M2 weight storage. gsq_calibration=True requires enabled GSQ and weight_scale_method row or block, and runs the calibration-aware looper path instead of the weight-only lifecycle.
ParoConfig 4-bit module-scope optimization bits 4, opt_scope "module", and opt_train_on_noisy_inputs False.

Each adapter keeps its method's native scales, zero points, packing, and format constraints; refinement records gsq_diagnostics (objective value before/after, or a skip reason) into the module state for writer-side provenance.

Staged Llama block training

gsq_training selects a staged reconstruction recipe instead of the ordinary loop. On GPTQConfig the initializer must be "gptq" or "rtn"; on AWQConfig pass {"initializer": "awq"} (the dict form fills it in automatically) and export must be 4-bit GEMM.

from gptqmodel.quantization import AWQConfig, FORMAT, GPTQConfig, GSQTrainingConfig

training = GSQTrainingConfig(enabled=True, epochs=20, qk_steps=2000)

gptq_config = GPTQConfig(
    bits=4, group_size=128, format=FORMAT.GPTQ_V2,
    act_group_aware=False, offload_to_disk=False,
    gsq_training=training,
)

awq_config = AWQConfig(
    bits=4, group_size=128, format=FORMAT.GEMM,
    gsq_training={"enabled": True, "epochs": 20, "qk_steps": 2000},
)

The public staged path — quantize_llama_gsq_public for METHOD.GPTQ — enforces all of the following at this revision:

  • A fully materialized, unquantized dense Llama in eval mode on one CPU or CUDA device, with attn_implementation="eager".
  • Symmetric W2/W3/W4 GPTQ scalar export (format GPTQ or GPTQ_V2) with desc_act and act_group_aware off, or AWQ W4 GEMM export using AWQ's scaled weights, integer zero points, and packer.
  • pack_dtype int32 and the Torch packing backend (AUTO/GPTQ_TORCH).
  • No rotation, adapter, gptaq, foem, mock_quantization, static_groups, adjacent_model, adaptive_clipping, lm_head, preprocessors, smoother, offload_to_disk, or adaptive damping, and no adapter/adapter_calibration_dataset/embed_quant_config/checkpoint arguments to quantize(). Unsupported options raise rather than being ignored. dynamic is the single exception — it is accepted only as a per-block width assignment, described in Non-uniform bit allocation (RCO).

The staged recipe trains all seven projections of every decoder block without dynamic skipping: query and key projections separately, value and output jointly against attention output, then the MLP projections against the decoder block output. It trains assignments and scales with the selected optimizer (Lion by default) under temperature and logit-multiplier schedules, and packs each block before capturing the next block's inputs. Block state can be checkpointed so a long W3/W2 run resumes safely; a resume validates capture metadata in the checkpoint signature. New checkpoints also record a rolling hash of their packed predecessors, and checkpoint gaps or changed predecessors are rejected before training — see Distributed execution and activation reuse for the prefix-import exception.

restore_best controls which epoch is exported:

restore_best Behavior
None (default) Automatic: selects the best held-out validation epoch when validation batches exist; exports the last epoch when they do not.
True Requires nonempty held-out validation batches for every reconstruction stage; staged runs without validation_documents raise.
False Always exports the last epoch.

Held-out validation uses validation documents supplied through the training capture path; ordinary calibration= alone does not create a held-out split for this guard.

The public quantize() staged path keeps the model on one CPU or CUDA device. Multi-rank execution is available only through the runtime entry point; see Distributed execution and activation reuse.

Kernel acceleration

Staged W3/W4 training compiles a fused scalar-relaxation kernel instead of running the eager autograd path: through NVRTC on CUDA, and through HIPRTC on ROCm since 4b36fd61 — the HIPRTC backend is selected first when torch.version.hip is set and binds libhiprtc via ctypes, so it needs no build step or new dependency. Measured on an MI350X (gfx950): ~72 ms to ~14 ms per update at W3/W4 (~5x). W2 cannot use the fused path — the kernel needs a five-choice logits tensor and a non-None initial offset, and initial is None for W2 — so W2 always runs eagerly. GPTQMODEL_GSQ_DISABLE_CUDA_RELAXATION=1 forces the eager path everywhere; GPTQMODEL_GSQ_DISABLE_HIP_RELAXATION=1 skips only the HIPRTC backend. When a backend is unusable the module returns None and training silently continues on the eager path.

Separately, when the group map is the standard contiguous arange(in) // group_size, the scale-gradient backward reduces as a segment sum instead of a colliding duplicate-index scatter_add_ — same terms, different reduction order (fp32 error ~1e-7, forward output bit-identical), ~1.45–1.54x faster per staged update at paper geometry on gfx950.

Distributed execution and activation reuse

The runtime entry point quantize_llama_gsq_model() — what run_arm.py and the reproduction scripts drive, distinct from the quantize() public path — accepts a distributed context: a GSQDistributed (gptqmodel/quantization/gsq_distributed.py) wrapping an already-initialized Gloo or NCCL torch.distributed process group (on ROCm the nccl backend name uses RCCL). Distributed execution requires enabled staged training; batch_size remains the global optimizer batch and must be divisible by world size, and microbatch_size must fit the per-rank local batch. Each global batch's documents shard into contiguous rank chunks, and dataset lengths that do not divide into complete global batches are rejected rather than truncated — so a 4096-document corpus at global batch 64 always produces the recipe's 1280 updates per stage regardless of world size.

Group rank zero is the primary: it owns GPTQ initialization and the fixed-quadratic Q/K fit and broadcasts the results, performs atomic checkpoint reads/writes, and drives the globally-weighted validation and best-epoch selection. Initial quantizer state is synchronized across ranks before the first fit evaluation; sampling RNGs are rank-local. Accumulated gradients and reported losses are weighted by reconstructed output-element counts and summed across ranks before each optimizer step — FP16/BF16 gradients reduce in bounded FP32 chunks. Capture and incremental replay are rank-local, but their failures propagate collectively. GSQDistributed never initializes torch.distributed itself; the caller owns process-group setup (e.g. torchrun), device selection, and optimizer steps.

Three memory knobs ride along on the same entry point:

Parameter Effect
target_cache — 'none' (default) | 'cpu' | 'device' Retain exact fixed stage targets — teacher outputs and the fixed post-attention residual — in host or device memory instead of recomputing them inside each update. A memory policy requires SDPA attention; the SDPA disk-capture path can still create temporary target caches even when the explicit policy is none.
activation_checkpointing — False (default) Recompute student activations during backward with a non-reentrant checkpoint, reducing activation storage; it does not redraw noise during backward or share a sample between microbatches.
reuse_capture — False (default) Advance train, calibration, and validation captures through each newly packed block — including its stored-scale rounding — instead of replaying the whole prefix for the next block. Requires memory-backed (CPU or device) capture; incompatible with the capture_directory disk spool. After a resumed prefix, the first unfinished block is still captured from the loaded prefix before incremental advancement continues.

resume_prefix_directory performs an explicit compatible-prefix import into a checkpoint_directory (required): completed blocks from another checkpoint tree are verified against the model, all document digests, bit widths, layer selection, the determinism setting, and the full GSQ recipe including microbatch size and seed — while device, distributed_world_size, target_cache, and activation_checkpointing may differ. Imported blocks are rewritten atomically under the destination signature and record imported_from; completed destination blocks take precedence over the import source, and the plan selects only one contiguous completed prefix — gaps are rejected before any training starts. Import carries completed blocks only, not an interrupted block's optimizer state. New checkpoints hash their preceding packed blocks into packed_prefix, so a changed predecessor is rejected; legacy checkpoints without that hash are accepted only when their earlier source files reproduce the packed prefix actually installed. Ordinary resume still requires the full execution signature.

The run record exposed as model.gsq_training_runs additionally stores world_size, target_cache, activation_checkpointing, reuse_capture, and resume_prefix_directory.

Non-uniform bit allocation (RCO)

A staged checkpoint stores one width per decoder block, so blocks may mix widths. The GSQ paper's 2.37 and 2.62 bits-per-parameter Llama rows are exactly such non-uniform assignments, produced by RCO — Riemannian budget-constrained optimization (Helcig & Alistarh, arXiv 2605.00649, cited from the GSQ paper's Sec. 4.2 and Table 1). gptqmodel/quantization/rco.py is a dependency-free port of the IST-DASLab/RCO reference: its tangent projection, retraction, and momentum transport match the reference alpha bit-identically at tolerance 0.05 and 1e-6, and its knapsack rounder matches the reference solver exactly at 16 and 112 groups.

An assignment is expressed per decoder block:

  • On the public GPTQ staged path (quantize_llama_gsq_public), QuantizeConfig.dynamic accepts only whole-block width entries — {'model.layers.<n>': {'bits': 2|3|4}} — one width per selected block. Any other module name or any override field besides bits raises; a non-uniform export writes per-projection dynamic entries for blocks whose width differs from the base bits and records the full map in meta['gsq_layer_bits'] so loaders rebuild each layer at its assigned width.
  • Internally, quantize_llama_gsq_model() accepts bits as a {layer_index: width} dict, and resolve_projection_bits/initialize_llama_gptq accept per-projection maps; projection_bits_for_budget builds a greedy exact-budget assignment interpolating two widths.

scripts/gsq_paper/rco_allocate.py builds the width database from staged per-block checkpoint arms, runs the projected Gumbel search on held-out calibration documents, and rounds the soft solution to an exact-budget assignment; rco_materialize.py installs each block from the arm of its assigned width and runs the normal staged export — no retraining:

python scripts/gsq_paper/rco_allocate.py --model meta-llama/Llama-3.2-1B-Instruct \
    --corpus /models/datasets/gsq-bench/fineweb-paper \
    --out /workspace/rco-2375 --target-bits 2.375 \
    --arms '/workspace/w{bits}gsq4096/ckpt' --layers 16 --steps 200
python scripts/gsq_paper/rco_materialize.py --model meta-llama/Llama-3.2-1B-Instruct \
    --assignment /workspace/rco-2375/assignment.json \
    --arms '/workspace/w{bits}gsq4096/ckpt' --out /workspace/rco-2375/model

Measured on Llama 3.2 1B (16 blocks, {W2, W3} options, held-out NLL on all 128 validation sequences): both budgets are hit exactly — 2.3750 and 2.6250 bpp, max constraint drift 9.5e-7, with the paper's 6-of-16 and 10-of-16 promotions — but the allocation does not beat a uniform mix: RCO 2.375 gives NLL 5.157 and 2.625 gives 4.631, versus 5.808 for uniform W2 and 3.835 for uniform W3, which is worse than a linear interpolation of those endpoints (+0.089 and +0.056 NLL). The paper's near-W3 accuracy at 2.37 bpp depends on the per-layer sensitivity structure of 8B/70B models, which a 1B model with 16 near-uniform blocks does not have. Context and next steps: docs/experiments/gsq-paper-reproduction.md.

Configuration reference

GSQConfig

Field Declared type Declared default Meaning and validation
enabled bool False Master switch; must be bool. None/absent disables refinement entirely.
steps int 100 Refinement steps per layer; integer >= 1.
candidates int 33 Hard-code candidates sampled per row chunk; integer >= 2.
seed int 7 Refinement RNG seed; integer >= 0.
learning_rate float 0.1 Candidate optimizer rate; finite positive number.
temperature_start float 1.0 Initial Gumbel-Softmax temperature; finite positive.
temperature_end float 0.1 Final temperature; finite positive.
max_candidate_bytes int 64 * 1024**2 Candidate-bank byte budget per output-row chunk; integer >= 1. Not a total-memory cap.
learn_scales bool False Tune stored scales where the method permits; rejected for QQQ, FP8, and ParoQuant.
modules Optional[Tuple[str, ...]] None Nonempty sequence of PCRE regular expressions restricting refinement to matching module full names; each pattern must compile.

Configs accept a GSQConfig, a dictionary (normalized to GSQConfig), or None.

GSQTrainingConfig

Field Declared type Declared default Meaning and validation
enabled bool False Master switch; must be bool.
initializer str 'gptq' gptq, rtn, or awq. GPTQ export requires gptq/rtn; AWQ export requires awq.
optimizer str 'lion' lion or adamw.
seed int 7 Training RNG seed; integer >= 0.
epochs int 20 Reconstruction epochs per stage; integer >= 1.
batch_size int 64 Documents per training batch; integer >= 1.
microbatch_size int 2 Microbatch documents; integer >= 1 and <= batch_size.
qk_steps int 2000 Update steps for the Q/K stage; integer >= 1.
damp_percent float 0.01 Hessian damping fraction for staged fitting; finite nonnegative.
assignment_lr float 1e-4 Assignment (logit) learning rate; finite and > 0.
scale_lr float 5e-5 Scale learning rate; finite and > 0.
weight_decay float 1.0 Optimizer weight decay; finite nonnegative.
betas tuple[float, float] (0.9, 0.95) Two optimizer moments, each in [0, 1).
temperature tuple[float, float] (2.0, 0.05) (start, end) sampling temperature endpoints; both positive.
multiplier tuple[float, float] (100.0, 500.0) (start, end) logit-scale multiplier endpoints; both positive.
warmup_steps int 0 LR warmup steps; integer >= 0.
min_lr float 0.1 Final LR fraction of peak; finite, <= 1.
decay str 'cosine' LR decay: cosine, linear, or constant.
restore_best bool | None None Best-validation-epoch export; see table above. None is auto.

training_kwargs() drops enabled and renames damp_percent to qk_damp_percent for the fitting entry points. to_dict() re-validates before serializing.

Serialization, diagnostics, and saved metadata

When a gsq/gsq_training option is set, to_dict() writes it into the quantize config payload:

Key Emitted by
gsq GPTQ, AWQ, RTN, QQQ, ParoQuant, FP8 configs when gsq is not None (as a plain dict).
gsq_calibration FP8Config always (bool).
gsq_training GPTQConfig and AWQConfig when set, via GSQTrainingConfig.to_dict().

A staged run additionally records meta["gsq_training"] (per-block run metrics) and meta["gsq_training_recipe"] = "experimental_staged_llama_v1" in the saved checkpoint, and the model wrapper exposes gsq_training_run["blocks"]. A non-uniform run also records meta["gsq_layer_bits"] (the {layer_index: width} map) and publishes per-projection dynamic entries for blocks not at the base width. Per-module refinement leaves gsq_diagnostics in module state — before/after objective values, learn_scales, or a skipped status with its reason. These diagnostics are provenance, not a quality guarantee: a recorded after below before is a local-objective improvement, not evidence of better downstream accuracy.

Paper reproduction scripts and measured results

scripts/gsq_paper/ reproduces the paper's staged-GSQ comparison (App. G.1) on a single Llama 3.2 1B Instruct checkpoint — a GPTQ-only arm versus a GPTQ-initialized arm that also runs staged reconstruction training:

python scripts/gsq_paper/build_corpus.py --model meta-llama/Llama-3.2-1B-Instruct \
    --out /models/datasets/gsq-paper
scripts/gsq_paper/run_all.sh --model meta-llama/Llama-3.2-1B-Instruct \
    --corpus /models/datasets/gsq-paper --out /models/gsq-paper-out \
    --gpus 0,1,2,3,4,5 --bits "2 3 4"
python scripts/gsq_paper/compare_arms.py --corpus /models/datasets/gsq-paper \
    --models /models/gsq-paper-out/w{2,3,4}-gsq-s7/model \
             /models/gsq-paper-out/w{2,3,4}-gptq-s7/model \
    --dense meta-llama/Llama-3.2-1B-Instruct --n 128

Measured against matched GPTQ baselines on that model (group size 128, full 4096/512/128 splits, 20 epochs, 2000 Q/K steps): W2 −3.395 NLL / +1.19 avg accuracy, W4 −0.537 NLL / +0.30, while W3 is noise-dominated at this scale — three runs straddle the GPTQ baseline within a 0.219-NLL spread against a nominal 0.05 effect, below the evaluation's resolution. Documented deviations: the paper's final 2-bit scale-only end-to-end stage is absent (from both this implementation and the released reference), Q/K is fit on the GPTQ-initialization split rather than the training split, and the paper's Llama 3.1 8B/70B absolute numbers do not transfer to a 1B model. The scalar relaxation's scale-gradient backward reduces as a segment sum when the group map is contiguous — the standard staged case — and the collision-prone scatter_add_ remains only for non-contiguous maps. run_arm.py --deterministic still enables torch.use_deterministic_algorithms(True) (off by default) for bit-identical Q/K reruns.

For distributed and memory-bounded runs, run_arm.py additionally exposes --device (default cuda:0), --distributed (requires a torchrun launch with at least two ranks; --batch-size stays the global batch and must divide world size, and --microbatch must fit the local batch), --target-cache none|cpu|device, --activation-checkpointing, --reuse-capture, and --resume-prefix-from PATH (explicit compatible-prefix import; requires --resume). See Distributed execution and activation reuse for their semantics.

run_distributed_pair.sh is the detached driver behind the paper-model Llama 3.1 8B reproduction: it runs two independent four-rank torchrun arms (W2 and W3 on separate GPU groups) against the full 4096/512/128, 20-epoch, 2000-Q/K-step budget, imports each arm's completed prefix, mirrors checkpoints while training, and then runs the RCO allocation/materialization and NLL comparison jobs. Its arguments are MODEL CORPUS OUTPUT MIRROR ORIGINAL_ARMS, and status.txt, driver.pid, and per-arm run.log/exit-code.txt expose progress.

bench_distributed_stage.py times complete MLP-stage updates against real 8B weight shapes. Measured on an MI350X with global batch 64 × 4096 tokens, microbatch 16, BF16 weights/activations/logits, device-cached targets and activation checkpointing (synthetic activations, RTN initialization):

Ranks W2 ms/update W3 ms/update
1 394.16 345.23
2 259.06 248.89
4 139.47 137.21

An eight-rank correctness run against an independent float64 oracle reports max absolute error 4.4e-16 (W2) and 2.7e-15 (W3) across losses, gradients, parameters, and optimizer state. These are update-timing and numerics results, not completed-model quality claims; the full 8B GSQ reproduction remains pending. Details: docs/experiments/gsq-paper-reproduction.md.

The same harness covers the paper's non-uniform budgets via rco_allocate.py/rco_materialize.py; see Non-uniform bit allocation (RCO) for usage and measured results.

Full recipe tables, per-document analysis, and statistical-power notes: docs/experiments/gsq-paper-reproduction.md and docs/experiments/gsq-paper-budget-1b-w3.md.

Pitfalls and edge cases

  • gsq and gsq_training cannot both be enabled on one config; staged and per-linear are separate paths.
  • GSQ requires enabled configs: a GSQConfig() default (or gsq set without enabled=True) is accepted but does nothing.
  • Refinement increases quantization time and allocates optimizer/candidate tensors beyond max_candidate_bytes.
  • The staged path runs inside quantize() before the ordinary looper; ordinary transforms and adapters are rejected, not silently applied. Passing calibration-free options or adapter arguments raises.
  • A dynamic map on the staged path is a width assignment, not a general override: model.layers.<n> block names setting only bits. Per-projection overrides must go through the internal bits dicts instead, and any other dynamic shape raises.
  • FP8 gsq_calibration=True flips FP8 off the weight-only lifecycle and requires calibration data; adapter generation is not supported on that path.
  • GPTQ local GSQ needs calibration activations, calibrated nn.Linear modules, and rejects tensor-parallel padded projections; data-independent fallback skips refinement rather than failing.
  • Staged GSQ validates resumable block checkpoints by capture metadata; checkpoint payloads are version-tagged, so resumes must match the producing configuration. Checkpoints additionally hash their packed predecessors, and gaps or changed predecessors are rejected before training; only an explicit resume_prefix_directory import may relax device, world size, target_cache, and activation_checkpointing.
  • An accepted config is not a quality promise: the README's measured gains are model- and data-specific, and the paper recipe is only partially reproduced here.

Source files

See also

Documentation home · Quickstart · Troubleshooting · Repository · PyPI

Source baseline: GPT-QModel 7.7.0 / c197691. Maintained by ModelCloud.ai.

Home

Getting started

Quantization methods

Complementary configs

Using a model

Platforms

Internals

Clone this wiki locally