Skip to content

v0.5.0

Choose a tag to compare

@github-actions github-actions released this 15 Sep 06:07
· 210 commits to main since this release

0.5.0 - 2026-09-15

Added

  • Four new deterministic scorers.text checks: regex_match (with
    full_match/case_insensitive options), starts_with, ends_with,
    numeric_range (one- or two-sided min/max). All registered under
    default_registry() like the existing text scorers.
  • Five new scorers.rubric templates, autoevals-style: TRANSLATION,
    SECURITY (code/output vulnerability review), SQL_CORRECTNESS
    (semantic equivalence against a reference query, not textual
    similarity), POSSIBLE (fabricate-vs-refuse on unanswerable requests),
    PII_LEAKAGE.
  • Two new zero-setup scorers for the Eval() API: matches (regex
    search) and numeric_close (1% relative tolerance).
  • Thirteen new methodology skills, bringing agentic_evals/skills/ from 4
    to 17 and covering the full eval lifecycle end to end (Frame/Build data/
    Score/Run/Experiment/Investigate/Operate), scoped to this package's own
    API: define-an-eval-objective,
    elicit-eval-criteria, build-an-eval-dataset, choose-a-rubric-template,
    validate-a-scorer, run-a-live-suite, design-an-eval-experiment,
    analyze-an-eval-experiment, discover-failure-modes,
    red-team-an-agent-suite, debug-a-flaky-llm-judge,
    report-eval-results, monitor-evals-in-production.