Skip to content

Evalmetry 1.0.0

Choose a tag to compare

View all tags
@minguinho26 minguinho26 released this 15 Sep 08:15
· 12 commits to main since this release
v1.0.0
76ff80c

Evalmetry 1.0.0 packages the lm-eval research toolkit for installation from PyPI.

Installation

pip install evalmetry

Python 3.10 or later is required. Model evaluation uses one CUDA GPU; install a compatible CUDA build of PyTorch for your system.

Included capabilities

  • Collect per-layer logit lens and layer similarity from the forward passes used for lm-eval scoring.
  • Optionally capture attention, hidden states, custom hooks, and module statistics.
  • Use custom benchmarks, LLM judges, and modified-model adapters.
  • Resume runs, read saved research data, and generate comparison reports.

Packaging

  • Publish wheel and source distributions with MIT licensing and repository metadata.
  • Keep the default lm-eval 0.4.13 and transformers 5.16.1 dependency pair.
  • Validate distribution metadata, installed imports, and CLI commands before publishing through GitHub Actions and PyPI Trusted Publishing.

Local validation passed for wheel installation with dependencies, dependency consistency, all package module imports, and CLI help. CUDA model evaluation was not run as part of this packaging release.

Assets 4
Loading