Expand eval corpus to 20 cases with stratified stats - #11
Open
jmjava wants to merge 2 commits into
Open
Conversation
Add a third wave of bounded extract/rename/split/implement/test/prose cases and report pass@1, pass@end, and first-failure rates by tool and category. Categories label the committed corpus; they are not a task classifier. Co-authored-by: jmjava <jmjava@gmail.com>
Dated Cloud Agent note for the expanded corpus. Stub observed campaign is pass@1 0.00 / pass@end 1.00 with layer mix; not live GPU timings. Co-authored-by: jmjava <jmjava@gmail.com>
jmjava
marked this pull request as ready for review
September 7, 2026 15:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The committed fixture corpus is now 20 cases (was 10) so stub/live harness runs can report more detailed rates without dumping transcripts.
Corpus
Ten additional bounded cases in
src/local_coding_slm/eval/cases_more.py:Persondataclass →person_model.py(precise + vague prompt, same checker)fetch→loadacross three filessettings.pyslugify, odd-lengthmedianmean; review missing zero-denominator check41 fixtures: every golden still passes all four layers; known-fail fixtures stop at format, structure, or behavior as labeled.
Stats
scripts/run_harness.py --outsummaries now includeby_toolandby_category(pass@1,pass@end,escalated,first_failure). Categories are labels on the committed corpus, not a task classifier (spec §8).Stub
observedcovers a mix of first-attempt format/structure/behavior fails plus two persistent-fast cases that must escalate (extract_dataclass_vague,test_clamp_execute).Cloud Agent stub run (no GPU): golden pass@1 = 1.00 (20/20). Observed 20 cases / 42 attempts: pass@1 = 0.00, pass@end = 1.00, escalated = 0.40; first failures format 6 / structure 10 / behavior 6. Dated note:
docs/cloud-corpus-stats-2026-09-07.md.Local checks on this branch: 109 tests passed, plus
run_eval.py,run_orchestration.py, and stub golden / observed / orchestrate harness.Not in this PR