Skip to content

evaluation-harness v0.6.2

Latest

Choose a tag to compare

@github-actions github-actions released this 03 Nov 18:36
· 1 commit to main since this release

Faster answers from evh and its MCP server. No result moves.

Added

  • evh mcp --write-answers FILE and evh mcp --public --answers FILE: a release's answers written ahead. Each
    call to the MCP server runs a fresh evh process, several seconds of CPU; a deployment that limits the server's CPU
    pays it in tens of seconds (the evals server's first goals after each start, and after each hour, took 33 s at
    half a core). --write-answers writes the answer of every tool that takes no argument and does not depend on the
    date (not overview or activity); a public server started with --answers serves them without running the
    command, for as long as it runs. A line another harness version wrote is not read.

Fixed

  • A view and a catalog build read each arm's scores once. evh goals read 1,054 arms' scores for 686 arms; that
    reading was 79% of its CPU. The answers are byte for byte the same; goals, arms and models take about a
    quarter to a third less CPU.

Install

pip install "evaluation-harness @ git+https://github.com/chipi/eval-harness@v0.6.2#subdirectory=harness"