Skip to content

v0.2.0

Choose a tag to compare

@github-actions github-actions released this 13 Sep 17:52
· 216 commits to main since this release

0.2.0 - 2026-09-13

Added

  • agentic_evals.scorers: a built-in scorer library.
    • scorers.text (deterministic, no LLM): exact_match, contains_all,
      contains_any, levenshtein_similarity, embedding_similarity
      (provider-neutral -- pass an embed_fn, this package makes no
      embedding calls itself), valid_json, json_diff, numeric_diff.
    • scorers.rubric (LLM-graded, provider-neutral): RubricTemplate and
      LLMRubricEvaluator (pass a complete_fn; this package never calls a
      model itself). Ships FACTUALITY, CLOSED_QA, SUMMARY_QUALITY,
      BATTLE (pairwise A/B), and MODERATION templates.
    • scorers.trajectory (reads EvalTrace/EvalSpan directly):
      tool_call_precision, tool_call_recall, no_redundant_tool_calls,
      trajectory_efficiency.
    • default_registry() returns an EvaluatorRegistry pre-populated with
      every text/trajectory scorer under a stable name, ready to pass to
      evaluate_suite(suite, samples, registry=default_registry()).
    • All of the above are re-exported from the package root.