You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Added
The rubric as data.locus.evaluation.Rubric names the dimensions a
judge scores 0-100, each with a weight and a line of guidance, plus
optional letter bands (Rubric.default_grades() offers A 90 / B 75 / C 60
/ D 45 / F 0) and a pass_score. The weighted composite, the letter and
the pass are computed by code; missing dimensions count as zero rather
than vanishing from the average. Rubric.revision() hashes the rubric
and the judge prompt so two reports are comparable only when it matches.
Rubrics load from JSON or YAML (Rubric.load). An EvalCase.rubric may
be a string, as before, or a Rubric.
Typed verdicts with provenance.Verdict carries grade, dimensions, issues, the judge's advisory judge_reported_passed / judge_reported_score, and a JudgeProvenance (judge model id, rubric
revision, prompt hash, temperature, seed, channel, token usage, self_graded). EvalResult exposes grade and the full verdict; EvalReport adds grade_distribution.
A structured judge channel with pinned sampling.LLMJudge sends temperature=0.0 by default, takes an optional seed and timeout_s,
and receives the verdict through a submit_verdict tool call where the
model takes tools (channel="auto"), falling back to a plain-JSON reply; channel="text" never offers the tool. The response under grading is
delimited and declared as data in the prompt.
The run's own evidence as checks.EvidenceLedger lays the run's tool
executions out as rows with stable ids (E0, E1, ..., P for the
prompt). EvalCase.require_receipts fails when a figure, identifier, date
or quoted string in the answer appears in no row; EvalCase.claims are
confirmed by code when the evidence carries their figures and settled by
the judge otherwise, through a submit_claim_checks tool call whose
citations are verified against the rows (a quote not in the cited row
downgrades the check to unverifiable); EvalCase.grounding runs the
GSAR audit over the answer and the ledger with the judge model. Results
carry receipts, claim_checks and grounding_score.
Control-layer assertions.EvalCase.forbidden_tools passes when the
tool never executed, where a call the gate refused did not execute; EvalCase.expected_refusals passes when the gate answered denied or held_for_approval at least once. EvalCase.expected_tool_calls matches
a call on its arguments (ToolCallExpectation).
Budgets.EvalCase.max_total_tokens and max_cost_usd read the run's
metrics; a target that reports none fails the check.
Rubric caps.Rubric.caps drop the grade to a ceiling when a
contradicted claim, a missing receipt or an ungrounded answer is found;
the runner applies them after the verdict and records caps_applied.
Repeats.EvalRunner(repeats=) or arun(cases, repeats=) reads every
case several times. A case passes only when every graded reading passed
(pass^k), the reading shown is the most pessimistic, EvalResult.samples
keeps them all, and EvalResult.consistency flags a case whose readings
disagreed by more than the gate; the report counts unstable cases and
the summary prints pass k/n.
Judge panels and second opinions.EvalRunner(judges=[...]) grades
every judged answer with each judge; the first scores the case, EvalResult.panel records the panel's agreement and EvalResult.trust
the decision between the first two judges (trusted, calibration_required, rejected), which the summary prints.
Pairwise comparison in both orders.LLMJudge.compare asks which of
two answers better satisfies the rubric, once in each order; a preference
that flips with the position is a tie with consistent=False.
locus.evaluation.reliability. Pure functions over verdicts and grade
pairs: consistency, panel_agreement, trust_decision, cohen_kappa, calibration_drift, wilson_interval, bootstrap_ci and publication_gates.
Comparable reports. Every EvalReport carries a fingerprint of the
cases it measured (suite_fingerprint), by_tag counts, check_failures,
the judge models and rubric revisions seen, and save / load / to_markdown. diff_reports compares two reports case by case (regressed,
improved, added, removed, score and dimension deltas, newly failing
checks) and refuses two different suites unless forced. EvalGate is the
rule a build applies (pass rate, average, not graded, unstable, and
against a baseline no regressions or a bounded average drop), reporting
every reason it failed.
Cases as data.load_cases reads a JSON, JSONL or YAML file of cases,
with a top-level rubric applied to cases that name none; dump_cases
writes one back.
locus eval.run loads cases and an agent (and optionally a judge
and a rubric), prints the report, saves it and exits with the gate's
verdict; diff, gate and summary work on saved reports, with --markdown tables for pull-request comments. How-to: Evaluate an agent
in CI.
Notebook 56 now covers the rubric with a judge, receipts, repeats and
a saved, diffed, gated pair of reports.
Changed
A judge that returns no verdict leaves the case not graded. An empty
reply, prose without JSON, a call to another tool or no scored dimension
is Verdict(abstained=True). The result carries not_graded_reason, the
report counts it under EvalReport.not_graded and leaves it out of failed and avg_score, and the summary prints [NOT GRADED] with the
reason. The judge_unparseable:... check is still recorded, so nothing
that used to be visible is hidden.
Self-graded suites are named. When the judge model id equals the
model under test, provenance.self_graded is stamped and the report
summary ends with a warning. The case is not failed for it.