feat(eval): eval harness + first failure-mode fixtures#3
Merged
officialCodeWork merged 1 commit intoJul 12, 2026
Merged
Conversation
Step 0.2 (TRACKER). The measurement loop every later phase is gated by: - @coderadar/eval: fixture discovery, golden.json diffing, scorecard.json, history.jsonl (--record), thresholds gate with non-zero exit. - Per-check expectedFail (xfail semantics): failing marked checks report xfail and don't gate; passing marked checks report unexpected-pass and DO gate, so capability arrival is an explicit reviewed event. Forbidden (poison) checks never xfail. - Fixtures: c1-shared-datatable (headline case — per-instance attributions xfail'd until step 2.2, poison assertions active now), a4-generic-text (ambiguity honesty on 'Save' collisions), demo-app (baseline, app shared with examples/). - Current scorecard: 25 pass / 0 fail / 2 xfail / 0 unexpected-pass; lineage precision 1.000, recall 0.714 (xfail gap), match accuracy 1.000. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TRACKER Step 0.2 — the measurement loop. Every later phase's "are we going the right way" question is now answered by
pnpm eval: fixtures encode failure modes, goldens encode both the right answer and the specific poison answer, and thresholds gate CI.What's in it
@coderadar/eval— scans everyeval/fixtures/*/, diffs againstgolden.json, prints a per-failure-mode scorecard, writeseval/scorecard.json, appends toeval/history.jsonlwith--record, exits non-zero oneval/thresholds.jsonviolations.expectedFailmarkers let us assert the future behavior today: the C1 per-instance attributions are red-but-expected until prop-flow (2.2); the moment they start passing, the run fails withunexpected-passuntil the stale marker is removed — capability arrival is always an explicit, reviewed event. Forbidden (poison) checks never xfail.c1-shared-datatable— the headline case: sharedDataTable, two pages, two APIs. Poison assertions (cross-page endpoint bleed) are active now.a4-generic-text— three components sharing "Save": lone generic term must yieldambiguous, one distinctive term must resolve it.demo-app— baseline, sharing the app source withexamples/(no duplicate copy).Current scorecard
(recall 0.714 is the xfail gap — the two C1 instance attributions — and is exactly the number that jumps when Phase 2.2 lands)
Worth noting: the harness caught its first design bug during this PR — the original per-kind expectedFail marker flagged the already-passing definition-level attributions as unexpected-pass, forcing the switch to per-check markers. The gate works.
Acceptance criteria (from TRACKER 0.2)
pnpm evalgreen (C1 attribution assertions xfail as planned)Documentation
docs/testing-strategy.md— golden format updated to the implemented contract (per-check expectedFail, exact-set endpoints); history recording clarifiedTRACKER.md— 0.2 done, Status → 0.3🤖 Generated with Claude Code