Evaluation

Evaluation

Documentation home · Quickstart · Lifecycle

On this page

• Overview
• Workflow at a glance
• Choose the appropriate measurement
• Dependencies, devices, and reproducibility
• Evalution: the current benchmark path
• Perplexity and compact quality comparisons
• Data preparation helpers: utils/data.py
• Calibration normalization: utils/calibration.py
• Standalone MMLU-Pro utility
• lm-eval compatibility and migration
• How maintainers validate quantization accuracy
• W4A GSM8K evidence: what the artifact does and does not prove
• Pitfalls and troubleshooting
• Source files
• See also

Overview

Quantization quality needs more than a successful load or plausible generated sentence. GPT-QModel's current model-test evaluation delegates benchmark execution to Evalution, while repository utilities provide calibration preparation, a separate MMLU-Pro runner, compact perplexity comparisons, and model-specific regression gates.

This page documents source revision c1976918. Repository test helpers are described as test tooling, not as stable installed-package APIs. Historical validation artifacts retain the protocol and versions from their own run; they are not evidence that a benchmark was rerun on this revision.

Workflow at a glance

Reading the diagram: Independent measurements. CLI JSON contains aggregated metrics, not RunResult; disjoint windows omit cross-window transitions.

Text version of the workflow

• Evalution task scoring: Fix the protocol → Build the session → Score each task → Close and report.
• Compact perplexity: Tokenize held-out text → Slice disjoint windows → Sum next-token loss → Compute perplexity.

Open full-size SVG

Choose the appropriate measurement

Measurement	What it establishes	Relevant implementation
Downstream benchmark accuracy	Correctness on a named task, prompt protocol, dataset, and answer parser.	Evalution; tests/eval.py; scripts/eval_model.py.
Perplexity	Average next-token prediction loss on a fixed token corpus/windowing protocol.	scripts/benchmark_vptq.py::_perplexity.
Calibration quality/data fidelity	What examples and token/mask distributions the quantizer actually consumed.	utils/calibration.py; utils/data.py.
Numerical kernel validation	Packed-format decoding, arithmetic, dtype, and layout correctness against an independent reference.	Kernel tests and .agents/skills/gptqmodel-torch-oracle-numerics/SKILL.md.
Model lifecycle regression	Source load, quantize, save, reload, generation, and expected task metrics.	tests/models/model_test.py and model-specific subclasses.
Paired same-weight activation experiment	Effect of activation policy while controlling the saved GPTQ weights and prompt protocol.	tests/models/w4a_quality_regression.py and W4A evidence artifacts.

Do not substitute one row for another. A kernel can pass numerical tests while downstream accuracy regresses; a short perplexity run is not a full reasoning benchmark; a metadata-only runtime comparison is not fresh quantization validation.

Dependencies, devices, and reproducibility

Component	Requirement/boundary at this revision
Python	>=3.10 in pyproject.toml.
Evalution	Optional eval extra declares Evalution>=0.0.17; tests/eval.py imports it on demand and raises an installation error when absent.
Core tokenizer/data dependencies	tokenicer>=0.0.14, datasets>=5.1.0, pypcre>=0.6.4 in requirements.txt.
Core Torch/Transformers declarations	torch>=2.14.1, transformers>=5.20.2 in the current requirements. Historical artifact versions below differ intentionally.
Model tests	test extra includes pytest>=9.1.1, pytest-timeout>=2.4.0, parameterized>=0.9.0. Large model fixtures and dataset access are separate requirements.
Engine evaluations	vLLM/SGLang optional environments and accepted checkpoint formats apply; see Serving.
CUDA/ROCm/MLX	Use a backend supported by the actual hardware/build. No single CUDA/ROCm minimum belongs to the generic evaluation helper. The legacy MMLU-Pro runner hard-codes .cuda() during prompt-length checks, and the VPTQ comparison script explicitly requires a CUDA device. Neither is a device-neutral MLX evaluation harness.

pip install 'gptqmodel[eval]' --no-build-isolation
# For repository regression tests as well:
pip install 'gptqmodel[test]' --no-build-isolation

Record at least checkpoint revision/hashes, runtime revision, dependency versions, device/backend, dtype, tokenizer/template version, dataset revision/split/hash, selected rows, seed, batch size, prompt limits, few-shot construction, generation settings, answer parser, and exact metric key. Seeded greedy generation does not remove every protocol or floating-point dependency.

Evalution: the current benchmark path

Relationship to the external repository

The package is spelled Evalution, imported as evalution, and maintained at ModelCloud/Evalution. GPT-QModel's README points users there for post-quantization benchmarking. Evalution owns its benchmark factories, engines, sessions, results, and most task-specific prompting/scoring. GPT-QModel's repository adds a compatibility/test adapter and a local ARC-Challenge loglikelihood suite; it does not vendor the entire benchmark library.

The README's native-engine usage is:

import evalution as eval

run = (
    eval.GPTQModel(
        backend="marlin",
        device="cuda:0",
    )
    .model(eval.Model(path="ModelCloud/Llama-3.2-1B-Instruct-gptqmodel-4bit-vortex-v1"))
    .run(eval.benchmarks.gsm8k_platinum(apply_chat_template=True, batch_size=16))
)
print(run.to_dict()["tests"][0]["metrics"])

Those explicit backend/device/batch values are an example, not defaults for every task or machine. Consult the installed Evalution version for its broader benchmark API.

Repository evaluate() signature

The current compatibility helper lives in tests/eval.py, not gptqmodel/utils/eval.py:

def evaluate(
        model_or_id_or_path: Any = None,
        tokenizer: Any = None,
        tasks: Any = None,
        batch_size: int | str = 1,
        trust_remote_code: bool = False,
        output_path: Optional[str] = None,
        llm_backend: str = "gptqmodel",
        backend: BACKEND | str | None = BACKEND.AUTO,
        model_args: Optional[Dict[str, Any]] = None,
        **args,
):

Parameter	Default	Meaning
model_or_id_or_path	None	Model path/ID or supported loaded model/wrapper. None is not a valid evaluation target.
tokenizer	None	Used to resolve tokenizer-path configuration; a live-model session still requires a tokenizer attached to the model.
tasks	None	Defaults to ("arc_challenge",); list/tuple or one task identifier accepted.
batch_size	1	Engine batch policy. String "auto" becomes None at suite level, not a fixed universal batch size.
trust_remote_code	False	Model/config custom-code permission.
output_path	None	Result writing only occurs for a path whose suffix is .json. A directory path does not automatically create a result filename.
llm_backend	"gptqmodel"	Only gptqmodel, vllm, sglang, normalized case/whitespace. Chooses the Evalution engine family.
backend	BACKEND.AUTO	GPT-QModel inference backend for the native-engine path; distinct from llm_backend.
model_args	None	Split into known engine options and remaining model-load kwargs.
gen_kwargs in **args	None	Mapping or comma-separated key=value settings, interpreted by the suite builder.
apply_chat_template in **args	False	Requested chat formatting for suites that consume it.
suite_kwargs in **args	{}	Passed to the selected benchmark factory/local suite, with task-specific defaults filled via setdefault.

Any other top-level **args cause TypeError. This helper is not a drop-in implementation of every old lm-eval option.

# Run from a checkout where the repository's tests package is importable.
from tests.eval import evaluate, format_eval_result_table

result = evaluate(
    model_or_id_or_path="/path/to/quantized-checkpoint",
    tasks=["arc_challenge"],
    batch_size=1,
    apply_chat_template=True,
    model_args={"device": "cuda:0", "dtype": "bfloat16", "seed": 42},
    suite_kwargs={"max_rows": 32, "stream": True},
    output_path="results/arc-smoke.json",
)
print(format_eval_result_table(result))

A max_rows=32 result is a smoke check, not the full benchmark score.

Tasks, suites, and defaults

SUPPORTED_TASKS exposes these identifiers to the CLI:

Listed task identifiers	Special behavior in tests/eval.py
arc_challenge	Local multiple-choice loglikelihood suite; removes num_fewshot.
arc_easy, boolq, gpqa, hellaswag, openbookqa	Generic callable lookup on evalution.benchmarks; default suite batch size is supplied.
gsm8k_cot	benchmarks.gsm8k(variant="cot", ...).
gsm8k_platinum_cot	benchmarks.gsm8k_platinum(variant="cot", ...).
mmlu, mmlu_stem	benchmarks.mmlu; STEM sets subsets="stem"; both default stream=True.
mmlu_pro, mmlu_pro:math	benchmarks.mmlu_pro; colon suffix becomes subsets.

The Python builder also accepts nonempty mmlu_pro:<subset> values and callable generic benchmark names. The CLI's fixed supported-task list is narrower; for example, an arbitrary colon subset is not automatically allowed by its _resolve_task().

Setting	Builder behavior
Generative GSM8K/MMLU-Pro defaults	max_new_tokens=256, do_sample=False, temperature=0.0, unless provided through the relevant settings/suite kwargs.
Explicit suite_kwargs	Usually win because task defaults use setdefault. Do not accidentally specify contradictory generation settings in two places.
streaming	Normalized to stream when stream is absent; removed when both appear. Here this means dataset streaming, not token streaming.
Local MMLU/GSM8K fixtures	Uses /monster/data/model/dataset/hails-mmlu_no_train or /monster/data/model/dataset/gsm8k when they exist; otherwise leaves normal dataset defaults. These are maintainer fixture paths, not prerequisites for all users.
gen_kwargs string	Comma-separated items containing =; scalar coercion handles booleans, null/none, integers, floats, otherwise strings. Items without = are ignored. Prefer a mapping for complex values.

The builder does not forward every parsed gen_kwargs field to every task. Its explicit generative defaults read max_new_tokens, do_sample, and temperature; additional task-specific settings should be supplied through an appropriate suite_kwargs field supported by that factory. Generic suites do not automatically inherit all chat/generation flags.

Engine and live-model behavior

Target/engine	Construction
Path + native gptqmodel	Builds evalution.GPTQModel with backend, device/map, dtype, attention policy, seed, batch size, trust, padding side (default left), and supported graph settings; then builds an evalution.Model.
Loaded Transformers/PEFT model or GPT-QModel-like wrapper	Requires a resolvable model path and attached tokenizer; builds a TransformersCompatSession retaining the wrapper so composite-model forward shims remain active.
vllm	Builds Evalution's VLLM engine; wrapper defaults include tensor parallel size 1 and GPU memory utilization 0.9, unless supplied. This is separate from GPT-QModel's utils/vllm.py Python-generation adapter.
sglang	Builds Evalution's SGLang engine with translated engine options. This is also separate from the direct GPT-QModel runtime wrapper.

Known engine options include device/map, dtype, attention implementation/backend, seed/random seed, padding side, tensor/pipeline/data parallel settings, memory/context limits, tokenizer configuration, quantization/load format, and engine-specific kwargs. Reserved model-identity/backend/tokenizer entries are dropped rather than blindly passed as loader kwargs. safe_kwargs_call() is used for selected engine constructors to tolerate changed optional signatures.

For live models, tokenizer preparation uses Evalution's _clone_prepare_tokenizer; the requested padding side is applied to both tokenizers and missing pad IDs are filled from EOS IDs when available. Sessions are closed in finally, with session.gc() between tasks. Results use Evalution's RunResult structure with model, engine/execution metadata, and tests.

ARC-Challenge scoring and metric aliases

The local _ArcChallengeLoglikelihoodSuite defaults to allenai/ai2_arc, configuration ARC-Challenge, split test, stream=True, no row cap, and optional chat formatting. Each choice is scored as a continuation, with a leading space added when it lacks whitespace. It selects both the maximum total logprob and maximum token-normalized logprob:

Metric	Definition
accuracy,loglikelihood	Correctness of the choice with greatest continuation logprob.
accuracy,loglikelihood_norm	Correctness of the choice with greatest logprob / max(token_count, 1).

Per-sample results retain prompts, choice scores, gold index, raw/normalized predicted indices, and normalized-choice text. This is not free-form letter-generation accuracy. The chat-render helper falls back to the original prompt if template rendering fails; inspect saved prompts instead of assuming a requested template was necessarily used.

get_eval_task_results() extracts task metrics from result["tests"]; get_eval_task_metrics() also resolves colon MMLU-Pro names to underscore result names. resolve_eval_metric_alias() preserves these legacy metric spellings:

Legacy metric	Current lookup
acc, acc,none	accuracy,loglikelihood
acc_norm, acc_norm,none	accuracy,loglikelihood_norm

CLI helper

Run from the checkout with its imports available:

PYTHONPATH=. python scripts/eval_model.py --list-tasks
mkdir -p results
PYTHONPATH=. python scripts/eval_model.py \
  --model /path/to/quantized-checkpoint \
  --tasks arc_challenge \
  --chat-template-tasks arc_challenge \
  --batch-size 1 \
  --output results/arc.json

Flag	Default/behavior
--model	Required unless listing tasks.
--backend	auto; parsed from lowercase BACKEND member names.
--tasks	arc_challenge, comma-separated.
--chat-template-tasks	None; comma-separated task subset.
--batch-size	auto; integer or auto.
--dtype	auto, passed to model loading.
--gen-kwargs	None; generation settings for evaluation.
--model-arg KEY=VALUE	Repeatable evaluation model arguments.
--load-arg KEY=VALUE	Repeatable GPTQModel.load() kwargs.
--trust-remote-code	Opt-in boolean.
--output	evalution_results.json; - disables saving.
--table-format	github.

The CLI does not create output parent directories; the example creates results/ first. In contrast, the Python helper creates parent directories when writing a .json output_path.

The script loads a BaseQModel first, then evaluates selected tasks and prints metrics. Unlike direct evaluate(..., output_path="results.json"), this CLI saves aggregated task metrics only, not the complete Evalution RunResult with per-row samples and execution metadata. Use the Python helper or Evalution directly when those records are required. The CLI evaluates its live wrapper with the helper's default llm_backend="gptqmodel"; --backend controls checkpoint loading and is not a switch to Evalution's VLLM/SGLang engine classes. For external-engine evaluation, call evaluate() with a model path, suitable model_args, and either llm_backend="vllm" or llm_backend="sglang" instead.

The script sets CUDA device ordering and allocator defaults; on Darwin it sets MPS fallback if unset. Do not infer that every model/backend in the test suite works on MPS from that convenience setting.

Perplexity and compact quality comparisons

There is no current generic tests/perplexity.py harness in this checkout. The concrete compact implementation is scripts/benchmark_vptq.py, alongside task-driven model tests. Its default comparison covers dense, VPTQ, and GPTQ at nominal 2/4/8 bits.

PYTHONPATH=. python scripts/benchmark_vptq.py \
  --model /path/to/source-model \
  --output-root /path/to/benchmark-artifacts \
  --configs dense,gptq4 \
  --device cuda:0

This can quantize and save models; it is not a read-only evaluation command. Choose writable output storage and a workload appropriate for the GPU. Its important defaults are:

Flag	Default
--configs	dense,vptq2,vptq4,vptq8,gptq2,gptq4,gptq8
--device	cuda:0; non-CUDA device types rejected
--calibration-rows	128
--calibration-concat-size	512
--eval-tokens	8192
--eval-seq-len	1024
--decode-tokens	32
--warmup, --trials	1, 1
--arc-batch-size, --arc-tasks	32, arc_easy
--with-speed, --with-arc	False; opt-in additional work

Calibration takes deterministic complete blocks from the beginning of WikiText-2 raw train, joining text with "\n\n", disabling added special tokens, and truncating to the calibration budget. Evaluation uses the test split with the same join and special-token policy, capped by --eval-tokens. The script uses AutoTokenizer.from_pretrained() directly; record that tokenizer path/version when comparing it with Tokenicer-normalized model-test results.

_perplexity(model, tokens, args) moves tokens to the requested device, iterates non-overlapping windows of eval_seq_len, disables KV cache, computes FP32 next-token cross entropy with summed reduction, accumulates scored token counts, and returns:

$$\operatorname{PPL} = \exp\!\left(\frac{\text{total next-token negative log likelihood}}{\text{total scored tokens}}\right).$$

Windows shorter than two tokens are skipped. This is not a sliding-window full-context likelihood: each window's first token is unscored and cross-window transitions are omitted. Changing sequence length, token budget, tokenizer, BOS/EOS behavior, or corpus concatenation changes the measurement. Empty/insufficient token input leaves the scored-token count zero and raises division by zero; it does not return a sentinel perplexity. Compare only matched protocols.

Data preparation helpers: utils/data.py

These are lower-level block/batch helpers, not complete accuracy or perplexity scorers.

make_data_block

def make_data_block(
    samples: Dict[str, List[str]],
    prompt_col_name: str,
    label_col_name: str,
    tokenizer: PreTrainedTokenizer,
    preprocess_fn: Optional[Callable] = None,
    sample_max_len: int = 1024,
    block_max_len: int = 2048,
    add_eos_token: bool = False,
    truncate_prompt: bool = True,
    merge_prompt_label: bool = False,
) -> Dict[str, List[LongTensor]]:

Option	Behavior
samples, column names	Batched prompt/label strings, optionally transformed by preprocess_fn.
sample_max_len=1024	Combined prompt/label length budget.
truncate_prompt=True	Removes excess tokens from the front of the prompt. False removes from the end of the label. Empty labels are dropped.
add_eos_token=False	When true, appends tokenizer EOS to each label before length handling.
merge_prompt_label=False	False keeps prompt inputs and separate label targets. True concatenates prompt+label and masks prompt/pad targets with -100.
block_max_len=2048	Budget based on padded rows in each block, not a count of independent text examples.

Rows are sorted by input length, packed into blocks, and left-padded internally. The actual return keys are input_ids, attention_mask, and labels with nested Python lists, despite the broad return annotation and an older singular-label docstring. Prompt-only truncation does not guarantee the combined budget if the label itself is longer than the budget; validate unusually long examples. A sample exceeding the block budget is also not robustly split by this packer.

collate_data

def collate_data(
    batch: List[Dict[str, List[List[int]]]],
    pad_token_id: int,
    padding_side: str = "right",
) -> Dict[str, Tensor]:

It flattens rows across outer items, validates ID/mask lengths, pads globally on the requested side, and returns input_ids as long tensors and attention_mask as boolean tensors. It does not return labels, even when upstream blocks contain them. Do not feed this collator to a loss evaluator assuming targets survived. Empty input produces (0, 0) tensors.

get_dataloader

def get_dataloader(
    data_path_or_name: str,
    prompt_col_name: str,
    label_col_name: str,
    tokenizer: PreTrainedTokenizer,
    load_fn: Optional[Callable] = None,
    preprocess_fn: Optional[Callable] = None,
    num_samples: int = 128,
    sample_max_len: int = 1024,
    block_max_len: int = 2048,
    add_eos_token: bool = False,
    truncate_prompt: bool = True,
    merge_prompt_label: bool = False,
    load_fn_kwargs: Optional[dict] = None,
    preprocess_fn_kwargs: Optional[dict] = None,
    **kwargs,
) -> DataLoader:

The loader uses a caller load function or datasets.load_dataset(). A dataset dictionary chooses evaluation, then test, otherwise train. It randomly selects up to num_samples with Python random.sample, maps the entire selection through make_data_block with one process and cache reuse disabled, and overrides DataLoader batch_size=1, shuffle=False, and its collator regardless of kwargs. One outer item can contain many padded rows, so batch_size=1 is not necessarily one text example.

Implementation caveats:

• Seed Python's RNG yourself for repeatable selection; there is no seed argument.
• The current implementation unconditionally constructs partial(preprocess_fn, ...); although the signature defaults to None, supply a callable (an identity function is sufficient when no preprocessing is needed).
• It uses len(), .select(), and .map(); a streaming iterable without those operations is not a general supported path merely because IterableDatasetDict is imported.
• Sampling, mapping, and block materialization are in memory. An empty selection or oversized sample needs caller-side validation.
• Its fixed collator drops labels. Use an appropriate target-preserving collator/scorer for actual loss evaluation.

Calibration normalization: utils/calibration.py

def prepare_calibration_dataset(
    qmodel,
    calibration_dataset: CalibrationInputType,
    calibration_dataset_concat_size: Optional[int] = None,
    calibration_dataset_sort: Optional[str] = None,
    batch_size: int = 1,
    calibration_data_min_length: int = 10,
    calibration_concat_separator: Optional[str] = None,
    logger=None,
):

This is the normalization path used by model calibration preparation, not a downstream task metric. It materializes the incoming iterable, rejects empty datasets and a single dataset string, and uses qmodel.tokenizer and qmodel.support_batch_quantize.

Input/policy	Behavior
List/iterable of raw strings	Requires a tokenizer; tokenizes with added special tokens and PT tensors. A bare string is not the whole dataset.
Dictionary with messages	Prefers callable apply_template; otherwise requires an actual chat template for apply_chat_template. The latter is requested with tokenize=False, add_generation_prompt=False; apply_template is requested with tokenize=False only. On TypeError, the helper retries with messages alone.
messages without a usable template	Falls back to that row's text if present, with a warning; otherwise fails.
Dictionary with text	Raw-text tokenization.
Dictionary with input_ids	Normalizes supplied IDs and optional masks.
Integer sequence/tensor	Normalizes IDs; absent attention mask becomes all ones.
Attention masks	Normalized into sequence keep masks, including supported higher-rank masks; incompatible shapes raise.
Position limits	Uses qmodel.model.config.max_position_embeddings and supported fallback names (max_sequence_length, max_seq_len, n_positions, seq_length); trims rows according to padding side and logs trimming.
Minimum length	Removes examples whose first normalized row length is less than or equal to calibration_data_min_length (default 10).
Concatenation size	Optional target-size token blocks, spanning source examples/chunks; final partial buffer is padded. A nonempty separator can force an earlier short block to flush without padding. Requires tokenizer access.
Concatenation separator	None means empty string, not automatic newline or EOS. A nonempty separator is tokenized and inserted by the concatenation loop.
Sort	asc, desc, shuffle; any other value/None retains native order. Shuffle uses Python RNG.
Batch-quantization support	Uses collate_data with tokenizer padding side and logs padded/nonpadded totals. Otherwise returns individual blocks with input_ids only.

GPTQMODEL_LOG_CALIBRATION_SAMPLES is an integer count (default 0), not env_flag boolean syntax. Positive values preview the exact sorted token rows before batching, decode up to 128 IDs, and limit displayed text. Previewing can reveal calibration text; avoid it for sensitive datasets. Invalid noninteger values fail integer parsing.

The optional concatenation loop uses only the first row of each normalized example (input_ids[0] and its mask). Supply one row per example if concatenation is enabled; a multirow tensor/dictionary can otherwise lose later rows. Position-limit trimming happens before concatenation, with no second context-limit clamp afterward, so choose a concatenation target compatible with the model. If minimum-length filtering removes everything, the helper can return an empty batch list; only an initially empty input has the explicit empty-dataset error. With batch quantization enabled, use a positive batch_size and a valid integer pad ID when rows need padding.

Calibration and evaluation must remain separate. Changing concatenation, separators, truncation, tokenizer/template, or ordering changes the quantization input distribution. A dataset described as streaming can still be materialized by this helper, so bound the iterable and memory budget before calling it.

GPTQMODEL_RESUME=1 is retired and raises during quantization; checkpoint resumption now uses explicit CheckpointConfig. AWQ_BATCH_SIZE is set from quantization batch size by the AWQ/ParoQuant processor setup; it is not the benchmark evaluation batch-size control. See Quantization Lifecycle.

Standalone MMLU-Pro utility

gptqmodel/utils/mmlupro.py is a separate chain-of-thought runner; it is not the same code path as Evalution's mmlu_pro factory.

def mmlupro(model: PreTrainedModel,
            tokenizer: PreTrainedTokenizerBase,
            ntrain: int = 5,
            selected_subjects: str = "all",
            save_dir: str = "results",
            global_record_file: str="eval_record_collection.csv",
            batch_size: int = 1,
            seed: int = 12345,
            max_samples: int | None = None):

Parameter/constant	Behavior
model, tokenizer	Native-compatible model/generation API and tokenizer. Caller should use the intended evaluation state/device.
ntrain=5	First category-matched validation examples for few-shot CoT prompting, reduced when the prompt is too long.
selected_subjects="all"	Otherwise comma-separated substring matching after space/underscore normalization; categories are sorted.
save_dir="results"	Default is redirected to logs/mmlupro/results.
global_record_file	Bare relative filename is placed under logs/; CSV rows are appended.
batch_size=1	Static DataLoader batch size. String "auto" is replaced by 4 with a warning, not measured auto-batching.
seed=12345	Calls Python random.seed, not a full Torch/CUDA deterministic setup.
max_samples=None	Optional cap per selected subject.
max_model_length / max_new_tokens	Module constants 4096 / 2048.
stop_string	"Question:".

It loads TIGER-Lab/MMLU-Pro test and validation splits, removes "N/A" options, builds CoT demonstrations from validation, and asks the model to finish with the answer is (X). The length-check loop reduces few-shot count until prompt length is strictly below 4096-2048; the batch tokenizer also enables truncation at 4096 and left padding.

Generation uses do_sample=False, temperature=0.0, 2048 new tokens, and stop_strings=["Question:"], passing tokenizer= to generation. It passes input_ids but does not forward the batch attention mask. Only tokens after padded prompt width are decoded; the stop marker is removed from returned text.

Answer extraction tries, in order, answer is (A-J), an answer-line pattern, then the last standalone A-J letter. Although the option-label list goes through P, the extraction regexes only accept A-J. An unparsed response is scored using a random option, not automatically marked wrong. eval_cot() calls save_res() twice, so those fallback random draws can differ between its logged and returned counts. This utility's scores therefore need their own parser/protocol identification.

Other important limits:

• Prompt-length checking calls .cuda() directly, even though batched inputs later use model.device; CPU/MPS-only use is not supported by that check.
• There is no terminal guard for an intrinsically oversized question that cannot fit even without demonstrations; inspect such inputs rather than expecting bounded few-shot reduction.
• A larger model context length does not change these fixed module constants automatically.
• Output consists of per-subject result JSON (including predictions and raw model outputs), summary text, and appended CSV records. mmlupro() returns the summary string, not an Evalution result dictionary.
• Results are not directly interchangeable with Evalution MMLU-Pro or lm-eval MMLU-Pro scores without matching prompts, row set, parser, generation, and fallback policy.

lm-eval compatibility and migration

Current tests/eval.py evaluates through Evalution; it does not call lm_eval.simple_evaluate() or export the old lm-eval harness schema. Legacy model tests and metric names still appear in the tree, but names alone do not establish which engine actually ran.

models/auto.py retains _hide_unsupported_quantization_config_for_eval(model) and the compatibility alias _hide_unsupported_quantization_config_for_lm_eval(model). The context manager tries Transformers' AutoQuantizationConfig.from_dict(). If that metadata is unsupported, it temporarily clears config.quantization_config and restores the original value in finally. It changes metadata visibility, not tensors or quantization math; it is a private helper, not blanket lm-eval support for all quantization formats/models.

Composite-model compatibility also exists in utils/hf.py and model definitions such as base_qwen2_5_omni.py for text-only tooling. That does not mean a text-only benchmark evaluates audio/vision behavior. Check Multimodal and MoE Models and record whether the task used the composite wrapper or a text submodel.

When migrating old scripts:

1. Choose Evalution's engine and benchmark factory, or the repository helper's supported arguments.
2. Translate exact metric names rather than comparing arbitrary acc fields.
3. Verify chat rendering and token IDs against the old protocol.
4. Preserve few-shot examples, row order, generation cap, padding, batch size, and parser.
5. Save per-row results and execution metadata, not only the final scalar.

How maintainers validate quantization accuracy

tests/models/model_test.py::ModelTest is a lifecycle and quality harness. Model-specific subclasses define source model/fixtures, quantization settings, backend, calibration policy, expected generation, and task metrics.

Harness control	Base declaration
EVAL_BATCH_SIZE, QUANT_BATCH_SIZE	"auto", 1
LOAD_BACKEND, QUANT_BACKEND	BACKEND.MARLIN, BACKEND.AUTO
INPUTS_MAX_LENGTH, MODEL_MAX_LEN	2048, 4096
DATASET_SIZE, DATASET_SORT	512, "desc"
BITS, GROUP_SIZE, SYM, DESC_ACT	4, 128, True, False
APPLY_CHAT_TEMPLATE	False; per-task/model overrides are important.
GPTQMODEL_MODEL_TEST_MODE	fast by default; accepts fast/slow; empty selects fast and full aliases slow.
EVAL_TASKS_FAST, EVAL_TASKS_SLOW	Model-specific task/baseline policies, not a universal score table.

evaluate_model() delegates tasks to Evalution, supports per-task chat, suite kwargs, engine args, batch size, and path-based evaluation, and aggregates exact metric keys. Old USE_VLLM=True model-test routing now raises and directs evaluation to Evalution; select an appropriate Evalution engine rather than toggling that obsolete switch.

check_results() requires configured baselines, validates task/metric presence, resolves metric aliases, and compares relative ranges:

$$100(1-\text{floor\_pct}) \le 100\frac{\text{observed}}{\text{expected}} \le 100(1+\text{ceil\_pct}).$$

These are relative fractions of the baseline, not percentage-point deltas. The harness also has a current-native-baseline fallback path; inspect the printed diffs and effective baseline before treating a passing test as proof of an unchanged frozen score.

Model-specific protocol matters. For example, tests/models/test_llama3_2.py explicitly enables chat templates for its instruct-model task policies; its fast and slow baselines and generation/evaluation settings differ. Fast runs are useful compatibility checks, not automatically full-model/full-protocol acceptance. Many tests reference maintainer-local model paths; provision the intended model rather than treating a missing fixture as an accuracy result.

Recommended validation sequence

1. Establish a non-quantized baseline and compare exact tokenizer/template outputs before diagnosing quantization.
2. Run independent numerical packed-format/kernel checks for the chosen dtype and device.
3. Quantize with recorded settings and, for calibration-based methods, recorded calibration data. Weight-only quantization does not require calibration forwarding. Save and reload through the intended inference backend.
4. Check generation and a bounded task subset for gross errors.
5. Run the full acceptance task with a frozen row set/protocol and matched baseline.
6. Preserve checkpoint hashes, per-row outputs, parser results, statistical comparisons, and memory observations. Do not hide expanded weight caches or swap/OOM behavior to make a run pass.

W4A GSM8K evidence: what the artifact does and does not prove

docs/gptq-w4a-gsm8k-validation.json records an October 7, 2026 campaign at runtime commit 7b6814c21e6774ad1b78b5b723d28d7f187e09d8. Its scope explicitly says saved full-model GPTQ checkpoints, no fresh weight quantization. It is historical evidence, not a claim that all experimental policies remain supported by current loading code.

Protocol item	Recorded value
Model	Llama-3.2-1B-Instruct
Dataset	madrylab/gsm8k-platinum, main, test, 1209 rows
Dataset fingerprint	056daafd03ad723e
Canonical rows SHA-256	8a0fdfe240dcc009ae774af302535ec8066db56e77bf4248957fac30faf0a593
Seed / model dtype / padding	42 / BF16 / left
Few-shot / rendering	Eight-shot, few-shot as multiturn, chat template enabled
Generation	Greedy, max_new_tokens=256
Runs	Four, 1209 rows each
Versions	Torch 2.11.0+cu130, Transformers 5.17.0, Triton 3.6.0, Tokenicer 0.0.14, Evalution 0.0.17, datasets 5.0.1

Each activation lane has its own same-weight W4A16 control. The lanes are not a direct A4-versus-A8 comparison:

Lane	W4A16 correct	Activation correct	Delta	Paired 95% interval	Recorded verdict
A4, batch 8	502/1209 (41.5219%)	474/1209 (39.2060%)	-2.3160 pp	[-4.7289, 0.0970] pp	inconclusive, budget 2.32 pp
A8, batch 32	432/1209 (35.7320%)	438/1209 (36.2283%)	+0.4963 pp	[-1.6182, 2.6107] pp	within_budget, budget 2 pp

The A4 historical point score was reproduced, but its statistical budget test remained inconclusive; “not a statistically detectable drop” is not the same as proving noninferiority. The A8 lane was within its matched-control budget. Report each lane's frozen protocol, not a combined headline implying identical controls.

The artifact additionally records weight hashes, activation policy, answer/correctness flips, exact McNemar p-values, historical-result comparisons, full-coverage dtype/transport audits, 75 passed audit tests, and artifact hashes. Its memory record includes a 2 GiB reserve, minimum observed available memory of 109,536,626,688 bytes, zero observed swap, and no increment in the recorded OOM/memory-limit counters.

The named full artifacts live under a maintainer host directory recorded in the JSON. That path is provenance, not a downloadable file included in every checkout. The checked-in JSON and explanatory notes are the portable evidence summary.

Related tools and historical notes

Source	Role/limits
tests/models/w4a_quality_regression.py	Constructs same-weight reference views, checks tokenizer/protocol alignment, evaluates full rows, compares paired per-row results, and enforces task-specific acceptance.
tests/models/w4a_quality_regression.py::compare_adapted	Distinguishes frozen/candidate controls for adapted checkpoints rather than conflating weight changes with activation policy.
tests/models/w4a_dtype_audit.py	Runtime dtype/transport audit tooling; complement with numerical kernel tests and full task evaluation.
tests/models/run_w4a_gb10_safe.sh	GB10-specific guarded campaign entry point, not a generic cross-device test runner.
docs/gptq-w4a-gb10.md	Historical architecture/campaign guide with the linked evidence summary.
docs/gptq-w4a-accuracy-notes.md	Detailed experiment log, including failed recipes and unresolved/continuation work. Not every experiment is an accepted production recipe.

The paired tool labels full GSM8K Platinum as an acceptance task; other diagnostic tasks do not automatically become acceptance gates. Passing an operator audit, reproducing old outputs, and satisfying a statistical budget are separate statements. Historical descriptions of expanded inference-weight operands must not be generalized into current storage policy: current inference rules require weights to remain quantized between operations.

Pitfalls and troubleshooting

• Bad prompt versus bad quantization: establish the dense/native baseline first; tokenizer normalization belongs in Tokenicer.
• Calibration leakage: do not use held-out task questions for calibration/adaptation and then present those task scores as held-out accuracy.
• Batch/protocol mismatch: preserve padding side, mask behavior, few-shot messages, model dtype, token limits, batch size, and prompt dates when comparing frozen artifacts.
• Wrong metric: distinguish free-form generated-answer accuracy, total loglikelihood choice accuracy, token-normalized accuracy, and perplexity.
• Dropped labels: the shared collate_data returns only IDs/masks; it is not a target-preserving language-model loss collator.
• Silent differences: ARC chat rendering can fall back; generic suite factories do not receive every global generation flag; inspect saved prompts/results.
• Legacy runner quirks: MMLU-Pro random fallback, double scoring, missing forwarded masks, and CUDA-only length checks can alter interpretation.
• Memory: bound materialized calibration/evaluation inputs, release large models/tensors between comparisons, and record OOM/swap observations. Full-corpus logits can dominate memory even for a small quantized model.
• Test fixtures/network: dataset downloads, gated models, optional engines, and maintainer-local paths are prerequisites, not evidence of a quantization regression when absent.

Source files

Path	Key responsibilities
README.md	Public Evalution relationship and native-engine example.
tests/eval.py	Current repository evaluator, task factories, ARC scoring, session construction, result/metric compatibility.
scripts/eval_model.py	CLI around loading and Evalution-backed evaluation.
tests/test_eval.py, tests/test_eval_runtime.py	Model/path evaluation smoke/regression examples.
tests/test_eval_loader_args.py, tests/test_evalution_suite_stream_defaults.py	Engine-argument and dataset-streaming compatibility tests.
tests/models/model_test.py	Lifecycle, evaluation configuration, expected-metric gates, current-native fallback.
tests/models/test_llama3_2.py	Concrete fast/slow task and chat-template policies.
scripts/benchmark_vptq.py	CUDA compact quality comparison and non-overlapping-window perplexity.
gptqmodel/utils/data.py	Prompt/label block packing, collator, dataset loader.
gptqmodel/utils/calibration.py	Calibration normalization, masks, trimming, concatenation, sorting, batching/previews.
gptqmodel/utils/mmlupro.py	Standalone MMLU-Pro CoT prompting, generation, extraction, result files.
gptqmodel/models/auto.py, gptqmodel/utils/hf.py	Quantization-metadata evaluation compatibility and composite/text-only compatibility helpers.
tests/models/w4a_quality_regression.py	Same-weight paired accuracy validation.
docs/gptq-w4a-gsm8k-validation.json	Machine-readable historical scores/protocol/audit/provenance.
docs/gptq-w4a-gb10.md, docs/gptq-w4a-accuracy-notes.md	Campaign explanation and experiment history.
pyproject.toml, requirements.txt	Evaluation/test/core dependencies and Python requirements.

See also

• Loading and Inference
• Quantization Lifecycle
• QuantizeConfig Reference
• Developer Guide
• Hugging Face Integration
• Serving
• Multimodal and MoE Models
• GPTQ
• VPTQ
• FP8 and NVFP4
• Troubleshooting and FAQ