Evalmetry 1.1.1

Evalmetry 1.1.1 corrects the documentation links that broke after the 1.1.0 release and records the matching tool version in run manifests. No evaluation code changed.

Installation

pip install evalmetry==1.1.1

Python 3.10 or later is required. Model evaluation uses one CUDA GPU. Install a compatible CUDA build of PyTorch for the target system.

Changes

• The README evidence links pointed at the release-1.1.0 branch, which no longer exists, so both returned 404. They now point at the version tag and no longer depend on a branch.
• Run manifests record tool version 1.1.1, matching the distribution version.
• The README, verification report, research showcase and release notes were rewritten with one paragraph per line and plainer sentences. Their numbers, tables, links and code examples are unchanged.

Verification

No evaluation code changed between 1.1.0 and 1.1.1, so the 1.1.0 verification evidence applies unchanged. No new GPU verification was run for this release.

The 1.1.0 page on PyPI keeps the broken documentation links, because the description of an uploaded release cannot be edited.

Full changelog: v1.1.0...v1.1.1

Evalmetry 1.1.0

Evalmetry 1.1.0 makes the few-shot count explicit, corrects the tool version recorded in run manifests, adds phase timings, and ships a verification report with supporting data. It is published on PyPI.

Installation

pip install evalmetry==1.1.0

Python 3.10 or later is required. Model evaluation uses one CUDA GPU; install a compatible CUDA build of PyTorch for your system.

Breaking change

--num-fewshot is now required, and RunConfig.num_fewshot has no default. Choose the shot count explicitly for the intended benchmark configuration:

evalmetry run --model-args pretrained=Qwen/Qwen3-0.6B,dtype=bfloat16,device=cuda \
    --tasks arc_easy --num-fewshot 0 --limit 8 --batch-size 1 \
    --output results/quickstart

Python callers must likewise pass num_fewshot, for example RunConfig(model_args="pretrained=...", tasks=["gsm8k"], num_fewshot=5).

Fixes and additions

• Generation with automatic batching probes once per generation loop instead of repeating the probe for each document. The traced generation batch remains 1.
• Run manifests now record tool version 1.1.0, matching the distribution version. Version 1.0.0 incorrectly recorded tool version 0.1.0.
• module-stats --pass collection now explains which passes exist and how to create a missing collection pass.
• A package docstring no longer refers to development-only review files.
• Run manifests record phase timings for model loading, evaluation, collection and end-to-end execution; collect-research-data records collection timings.
• The repository includes a verification report and a prediction-depth research showcase with supporting data.

Verification

• Ten checkpoints across five tasks passed all 50 stock-versus-traced comparisons. Three multiple-choice tasks used their full datasets with bit-identical request log probabilities; two generation tasks used 200 documents each.
• Six additional comparisons exercised optional heavy collection across three architectures and two tasks.
• Generation equality is established at the same actual batch size of 1. A separate 24-run experiment confirmed all twelve matched-batch comparisons. Changing stock batching can change outputs and scores: Qwen3 GSM8K changed from 46.875% at batch 1 to 43.75% at batch 8 on 32 documents.

Published artifacts

• The final wheel and sdist passed strict Twine validation. Rebuilding the sdist produced identical uncompressed wheel contents. Independent installation, dependency, import, CLI and offline local CPU scoring/collection checks passed. These installation checks are separate from the retained GPU parity evidence.
• The exact verified files were uploaded, then downloaded from PyPI and checked against their recorded SHA-256 values. See the artifact record for filenames, hashes, metadata and validation details.

Full changelog: v1.0.0...v1.1.0 · Release notes in the repository

Evalmetry 1.0.0

Evalmetry 1.0.0 packages the lm-eval research toolkit for installation from PyPI.

Installation

pip install evalmetry

Python 3.10 or later is required. Model evaluation uses one CUDA GPU; install a compatible CUDA build of PyTorch for your system.

Included capabilities

• Collect per-layer logit lens and layer similarity from the forward passes used for lm-eval scoring.
• Optionally capture attention, hidden states, custom hooks, and module statistics.
• Use custom benchmarks, LLM judges, and modified-model adapters.
• Resume runs, read saved research data, and generate comparison reports.

Packaging

• Publish wheel and source distributions with MIT licensing and repository metadata.
• Keep the default lm-eval 0.4.13 and transformers 5.16.1 dependency pair.
• Validate distribution metadata, installed imports, and CLI commands before publishing through GitHub Actions and PyPI Trusted Publishing.

Local validation passed for wheel installation with dependencies, dependency consistency, all package module imports, and CLI help. CUDA model evaluation was not run as part of this packaging release.