Skip to content

feat(eval): eval harness + first failure-mode fixtures#3

Merged
officialCodeWork merged 1 commit into
developmentfrom
build/phase-0/step-0.2-eval-harness
Jul 12, 2026
Merged

feat(eval): eval harness + first failure-mode fixtures#3
officialCodeWork merged 1 commit into
developmentfrom
build/phase-0/step-0.2-eval-harness

Conversation

@officialCodeWork

Copy link
Copy Markdown
Owner

Summary

TRACKER Step 0.2 — the measurement loop. Every later phase's "are we going the right way" question is now answered by pnpm eval: fixtures encode failure modes, goldens encode both the right answer and the specific poison answer, and thresholds gate CI.

What's in it

  • @coderadar/eval — scans every eval/fixtures/*/, diffs against golden.json, prints a per-failure-mode scorecard, writes eval/scorecard.json, appends to eval/history.jsonl with --record, exits non-zero on eval/thresholds.json violations.
  • xfail semantics (per check)expectedFail markers let us assert the future behavior today: the C1 per-instance attributions are red-but-expected until prop-flow (2.2); the moment they start passing, the run fails with unexpected-pass until the stale marker is removed — capability arrival is always an explicit, reviewed event. Forbidden (poison) checks never xfail.
  • Fixtures:
    • c1-shared-datatable — the headline case: shared DataTable, two pages, two APIs. Poison assertions (cross-page endpoint bleed) are active now.
    • a4-generic-text — three components sharing "Save": lone generic term must yield ambiguous, one distinctive term must resolve it.
    • demo-app — baseline, sharing the app source with examples/ (no duplicate copy).

Current scorecard

25 pass · 0 fail · 2 xfail · 0 unexpected-pass
lineage precision=1.000 recall=0.714 · match accuracy=1.000
eval gate: OK

(recall 0.714 is the xfail gap — the two C1 instance attributions — and is exactly the number that jumps when Phase 2.2 lands)

Worth noting: the harness caught its first design bug during this PR — the original per-kind expectedFail marker flagged the already-passing definition-level attributions as unexpected-pass, forcing the switch to per-check markers. The gate works.

Acceptance criteria (from TRACKER 0.2)

  • Runner with golden diffing, forbidden entries, scorecard.json, thresholds gate
  • expected-fail support in the runner
  • ≥ 3 fixtures including demo-app
  • pnpm eval green (C1 attribution assertions xfail as planned)

Documentation

  • docs/testing-strategy.md — golden format updated to the implemented contract (per-check expectedFail, exact-set endpoints); history recording clarified
  • TRACKER.md — 0.2 done, Status → 0.3

🤖 Generated with Claude Code

Step 0.2 (TRACKER). The measurement loop every later phase is gated by:

- @coderadar/eval: fixture discovery, golden.json diffing, scorecard.json,
  history.jsonl (--record), thresholds gate with non-zero exit.
- Per-check expectedFail (xfail semantics): failing marked checks report
  xfail and don't gate; passing marked checks report unexpected-pass and DO
  gate, so capability arrival is an explicit reviewed event. Forbidden
  (poison) checks never xfail.
- Fixtures: c1-shared-datatable (headline case — per-instance attributions
  xfail'd until step 2.2, poison assertions active now), a4-generic-text
  (ambiguity honesty on 'Save' collisions), demo-app (baseline, app shared
  with examples/).
- Current scorecard: 25 pass / 0 fail / 2 xfail / 0 unexpected-pass;
  lineage precision 1.000, recall 0.714 (xfail gap), match accuracy 1.000.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@officialCodeWork
officialCodeWork merged commit 9a075fc into development Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants