Evalmetry 1.1.0

Evalmetry 1.1.0 makes the few-shot count explicit, corrects the tool version recorded in run manifests, adds phase timings, and ships a verification report with supporting data. It is published on PyPI.

Installation

pip install evalmetry==1.1.0

Python 3.10 or later is required. Model evaluation uses one CUDA GPU; install a compatible CUDA build of PyTorch for your system.

Breaking change

--num-fewshot is now required, and RunConfig.num_fewshot has no default. Choose the shot count explicitly for the intended benchmark configuration:

evalmetry run --model-args pretrained=Qwen/Qwen3-0.6B,dtype=bfloat16,device=cuda \
    --tasks arc_easy --num-fewshot 0 --limit 8 --batch-size 1 \
    --output results/quickstart

Python callers must likewise pass num_fewshot, for example RunConfig(model_args="pretrained=...", tasks=["gsm8k"], num_fewshot=5).

Fixes and additions

• Generation with automatic batching probes once per generation loop instead of repeating the probe for each document. The traced generation batch remains 1.
• Run manifests now record tool version 1.1.0, matching the distribution version. Version 1.0.0 incorrectly recorded tool version 0.1.0.
• module-stats --pass collection now explains which passes exist and how to create a missing collection pass.
• A package docstring no longer refers to development-only review files.
• Run manifests record phase timings for model loading, evaluation, collection and end-to-end execution; collect-research-data records collection timings.
• The repository includes a verification report and a prediction-depth research showcase with supporting data.

Verification

• Ten checkpoints across five tasks passed all 50 stock-versus-traced comparisons. Three multiple-choice tasks used their full datasets with bit-identical request log probabilities; two generation tasks used 200 documents each.
• Six additional comparisons exercised optional heavy collection across three architectures and two tasks.
• Generation equality is established at the same actual batch size of 1. A separate 24-run experiment confirmed all twelve matched-batch comparisons. Changing stock batching can change outputs and scores: Qwen3 GSM8K changed from 46.875% at batch 1 to 43.75% at batch 8 on 32 documents.

Published artifacts

• The final wheel and sdist passed strict Twine validation. Rebuilding the sdist produced identical uncompressed wheel contents. Independent installation, dependency, import, CLI and offline local CPU scoring/collection checks passed. These installation checks are separate from the retained GPU parity evidence.
• The exact verified files were uploaded, then downloaded from PyPI and checked against their recorded SHA-256 values. See the artifact record for filenames, hashes, metadata and validation details.

Full changelog: v1.0.0...v1.1.0 · Release notes in the repository