You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Four new deterministic scorers.text checks: regex_match (with full_match/case_insensitive options), starts_with, ends_with, numeric_range (one- or two-sided min/max). All registered under default_registry() like the existing text scorers.
Five new scorers.rubric templates, autoevals-style: TRANSLATION, SECURITY (code/output vulnerability review), SQL_CORRECTNESS
(semantic equivalence against a reference query, not textual
similarity), POSSIBLE (fabricate-vs-refuse on unanswerable requests), PII_LEAKAGE.
Two new zero-setup scorers for the Eval() API: matches (regex
search) and numeric_close (1% relative tolerance).
Thirteen new methodology skills, bringing agentic_evals/skills/ from 4
to 17 and covering the full eval lifecycle end to end (Frame/Build data/
Score/Run/Experiment/Investigate/Operate), scoped to this package's own
API: define-an-eval-objective, elicit-eval-criteria, build-an-eval-dataset, choose-a-rubric-template, validate-a-scorer, run-a-live-suite, design-an-eval-experiment, analyze-an-eval-experiment, discover-failure-modes, red-team-an-agent-suite, debug-a-flaky-llm-judge, report-eval-results, monitor-evals-in-production.