Repository navigation
v3.7.2 — eval integrity, offline grading, and CI smoke
This release tightens saved evaluation validation and adds an opt-in way to reduce grading calls while preserving paired eval coverage. It also consolidates the verification and failure-handling notes from the older v3.6.0 draft.
New in v3.7.2
- Offline grading for paired evals. Preflight checks fixtures, baseline and planned cases before dispatch. Reviewed JSON Pointer and CSV row-count assertions can be graded offline. Unresolved assertions go to the independent grader. Each candidate and baseline still runs fresh for every selected case and model. Grades are bound to the prepared run, and the benchmark checks pairing, model identity and rubric completeness. Keep full grading as the default and replay each verifier against saved independent grades before enabling it. Synthetic replay confirmed the mechanics. Historical cost savings have yet to be measured.
- CI smoke and release drift. Automatic PR triage makes zero model calls. An authorized maintainer can request a five-call Haiku smoke test for the exact head of an eligible same-repository PR once the protected environment is configured. The workflow verifies the pinned CLI, rechecks eligibility before launch and reserves one paid attempt per PR and commit. Release reporting compares trigger inputs and compatible retained canary evidence offline. The scheduled and release canary still checks Haiku and Sonnet.
- Saved evaluation integrity. Comparisons reject incomplete runs, contradictory counts or pass flags, out-of-range metrics and conflicting result payloads. Matrix comparisons and live baseline preflight require explicit model identities. Legacy raw single-pair comparisons can still use unknown identities.
- Python package metadata declares Apache-2.0, and bundled release history points to the canonical changelog.
Verification and failure handling carried forward
- Skill validation rejects references that escape the skill directory. Documentation checks verify review commands against the CLI, and claim checks reject missing sources and obviously hollow tests. The shell validator runs repository-level source, documentation and claim checks when validating the toolkit.
- Transcript grading writes separate atomic reports per transcript and preserves original transcript bytes on Windows. The optimization loop retains completed iterations if the improvement step fails. Malformed lifecycle or test paths produce validation errors.
- Auto-repair preserves unmodelled frontmatter keys. Packaging excludes additional credential and key file patterns and handles case variants of the eval directory consistently.
- The eval viewer drains early 403 and 415 request bodies within a bounded deadline. Other early errors and large bodies can still race. Hollow-test detection is heuristic, and abrupt termination can leave a temporary file beside a report.