Skip to content

eval-harness v0.5.0

Latest

Choose a tag to compare

@hoainho hoainho released this 29 Sep 20:18
· 11 commits to main since this release
38635c4

v0.5.0

  • Adds typed capability, regression, and product evaluations; portable manifests; strict grading; promotion readiness; history, JUnit, and SARIF reporting; and the composite action.
  • Documented 0.x breaking change: default shell checks now allow only constrained jq/printf/wc -l pipelines. Migrate other checks to typed graders or use unsafe_shell: true / EVAL_ALLOW_UNSAFE_SHELL=1 only with trusted case content; this is not an OS sandbox.
  • Adds the Janus 0.1.10 offline benchmark. The matched file-output case did not invoke Janus's candidate executor, while checks: [] was accepted as PASS with zero checks; native adapter recommendation remains no-go until both behaviors are fixed.
  • Benchmark used deterministic stubs. No live model quality, cost, latency, or reliability measurements were made.