Skip to content

v5.4.0

Choose a tag to compare

@fede-kamel fede-kamel released this 10 Nov 22:10
18d6724

Added

  • The rubric as data. locus.evaluation.Rubric names the dimensions a
    judge scores 0-100, each with a weight and a line of guidance, plus
    optional letter bands (Rubric.default_grades() offers A 90 / B 75 / C 60
    / D 45 / F 0) and a pass_score. The weighted composite, the letter and
    the pass are computed by code; missing dimensions count as zero rather
    than vanishing from the average. Rubric.revision() hashes the rubric
    and the judge prompt so two reports are comparable only when it matches.
    Rubrics load from JSON or YAML (Rubric.load). An EvalCase.rubric may
    be a string, as before, or a Rubric.
  • Typed verdicts with provenance. Verdict carries grade,
    dimensions, issues, the judge's advisory judge_reported_passed /
    judge_reported_score, and a JudgeProvenance (judge model id, rubric
    revision, prompt hash, temperature, seed, channel, token usage,
    self_graded). EvalResult exposes grade and the full verdict;
    EvalReport adds grade_distribution.
  • A structured judge channel with pinned sampling. LLMJudge sends
    temperature=0.0 by default, takes an optional seed and timeout_s,
    and receives the verdict through a submit_verdict tool call where the
    model takes tools (channel="auto"), falling back to a plain-JSON reply;
    channel="text" never offers the tool. The response under grading is
    delimited and declared as data in the prompt.
  • The run's own evidence as checks. EvidenceLedger lays the run's tool
    executions out as rows with stable ids (E0, E1, ..., P for the
    prompt). EvalCase.require_receipts fails when a figure, identifier, date
    or quoted string in the answer appears in no row; EvalCase.claims are
    confirmed by code when the evidence carries their figures and settled by
    the judge otherwise, through a submit_claim_checks tool call whose
    citations are verified against the rows (a quote not in the cited row
    downgrades the check to unverifiable); EvalCase.grounding runs the
    GSAR audit over the answer and the ledger with the judge model. Results
    carry receipts, claim_checks and grounding_score.
  • Control-layer assertions. EvalCase.forbidden_tools passes when the
    tool never executed, where a call the gate refused did not execute;
    EvalCase.expected_refusals passes when the gate answered denied or
    held_for_approval at least once. EvalCase.expected_tool_calls matches
    a call on its arguments (ToolCallExpectation).
  • Budgets. EvalCase.max_total_tokens and max_cost_usd read the run's
    metrics; a target that reports none fails the check.
  • Rubric caps. Rubric.caps drop the grade to a ceiling when a
    contradicted claim, a missing receipt or an ungrounded answer is found;
    the runner applies them after the verdict and records caps_applied.
  • Repeats. EvalRunner(repeats=) or arun(cases, repeats=) reads every
    case several times. A case passes only when every graded reading passed
    (pass^k), the reading shown is the most pessimistic, EvalResult.samples
    keeps them all, and EvalResult.consistency flags a case whose readings
    disagreed by more than the gate; the report counts unstable cases and
    the summary prints pass k/n.
  • Judge panels and second opinions. EvalRunner(judges=[...]) grades
    every judged answer with each judge; the first scores the case,
    EvalResult.panel records the panel's agreement and EvalResult.trust
    the decision between the first two judges (trusted,
    calibration_required, rejected), which the summary prints.
  • Pairwise comparison in both orders. LLMJudge.compare asks which of
    two answers better satisfies the rubric, once in each order; a preference
    that flips with the position is a tie with consistent=False.
  • locus.evaluation.reliability. Pure functions over verdicts and grade
    pairs: consistency, panel_agreement, trust_decision, cohen_kappa,
    calibration_drift, wilson_interval, bootstrap_ci and
    publication_gates.
  • Comparable reports. Every EvalReport carries a fingerprint of the
    cases it measured (suite_fingerprint), by_tag counts, check_failures,
    the judge models and rubric revisions seen, and save / load /
    to_markdown. diff_reports compares two reports case by case (regressed,
    improved, added, removed, score and dimension deltas, newly failing
    checks) and refuses two different suites unless forced. EvalGate is the
    rule a build applies (pass rate, average, not graded, unstable, and
    against a baseline no regressions or a bounded average drop), reporting
    every reason it failed.
  • Cases as data. load_cases reads a JSON, JSONL or YAML file of cases,
    with a top-level rubric applied to cases that name none; dump_cases
    writes one back.
  • locus eval. run loads cases and an agent (and optionally a judge
    and a rubric), prints the report, saves it and exits with the gate's
    verdict; diff, gate and summary work on saved reports, with
    --markdown tables for pull-request comments. How-to: Evaluate an agent
    in CI
    .
  • Notebook 56 now covers the rubric with a judge, receipts, repeats and
    a saved, diffed, gated pair of reports.

Changed

  • A judge that returns no verdict leaves the case not graded. An empty
    reply, prose without JSON, a call to another tool or no scored dimension
    is Verdict(abstained=True). The result carries not_graded_reason, the
    report counts it under EvalReport.not_graded and leaves it out of
    failed and avg_score, and the summary prints [NOT GRADED] with the
    reason. The judge_unparseable:... check is still recorded, so nothing
    that used to be visible is hidden.
  • Self-graded suites are named. When the judge model id equals the
    model under test, provenance.self_graded is stamped and the report
    summary ends with a warning. The case is not failed for it.