Skip to content

evaluation-harness v0.2.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 20 Oct 20:09
· 76 commits to main since this release
v0.2.0
e100d85
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

The first version meant to be installed rather than copied.

Added

  • Installable package. pip install "evaluation-harness @ git+https://github.com/chipi/eval-harness@v0.2.0#subdirectory=harness"
    gives the evh command: every Makefile target, running the same scripts.
  • EVAL_HOME / EVAL_DATA_DIR. The project's data lives in the project, never in the
    installed package; the data root can be nested (for example data/eval/stats).
  • The harness identifies itself in every run. fingerprint.instrument.harness gains
    version, code_sha256 and installed.
  • evh import-run. Per-item scores produced by another tool become a run the
    significance tests read, with the source, the scorer and the sha256 of the imported bytes
    on the record.
  • --asc for pair-test and family-test. For lower-is-better metrics, "wins" and
    "ahead" count the lower score. The p-values were already two-sided and do not change.

Changed

  • validate's V9 evidence floors are read from <data>/validate_floors.json (project
    policy) instead of being hardcoded for this repository's run directories.
  • The family-test cache lives under the project root, not beside the code.

Install

pip install "evaluation-harness @ git+https://github.com/chipi/eval-harness@v0.2.0#subdirectory=harness"
Assets 4
Loading