Skip to content

v1.13.1 — Pierre pass/not-applicable fix, eval-verified; three-model scorecard

Choose a tag to compare

View all tags
@berkayturk berkayturk released this 12 Jul 01:12
· 48 commits to main since this release
v1.13.1
93eccd2

Fixed

Pierre's pass-vs-not-applicable boundary. The deep-review output format now states explicitly: "not applicable" is only for checks whose subject matter is entirely absent from the project; material that exists and is clean gets a plain REVIEW-PASS with evidence.

This is the eval harness (v1.13.0) paying for itself on day one: the consistency metric caught 3 of 63 clean-case verdicts drifting to "not applicable" across Opus 4.8 and Fable 5 (binary verdicts were never wrong — only the wording boundary). After the one-paragraph clarification, full re-runs of all three models show consistency 1.00 across the board with no regression on genuine not-applicable cases. Measure → fix → re-measure, closed loop.

Added (eval)

  • Multi-model scorecard: docs/llm-scorecard.md opens with a comparison table and renders every committed baseline — currently Claude Sonnet 5, Opus 4.8, and Fable 5, before and after the prompt clarification. The CI Tier-A F1 ≥ 0.80 floor gates each baseline separately.
  • Fable/Mythos-tier support: eval/run.sh --model claude-fable-5 works (thinking field omitted, max_tokens headroom, both recorded in the manifest).
  • Prompt fingerprinting + stricter resume: every manifest records a prompt_sha256 of the Pierre prompt; the resume guard refuses to mix models or prompt versions in one cache dir.

Full changelog: https://github.com/berkayturk/appstore-precheck/blob/main/CHANGELOG.md

Assets 2
Loading