test(evaluation-system): add harness prompt eval cases and fixtures - #37
Merged
cmschuetz merged 3 commits intoJul 28, 2026
Merged
Conversation
…ponse cache Foundational layer of the prompt-eval suite (D1). Adds src/eval/ with a single-shot Claude Code CLI (`claude -p`) substrate — never the Anthropic SDK: - types.ts: EvalRequest/EvalResult/Envelope + injectable Spawn and Cache seams. - engine.ts: buildArgs (pure print-mode argv: --print --output-format json --json-schema --model --strict-mcp-config --settings + conditional system-prompt/tool-policy flags; prompt on stdin), parseEnvelope (graded, never throws — prefers structured_output, falls back to result, shallow schema check), runClaude (cache-first, spawn, cache-on-success), isAvailable and a skip signal distinct from graded failure for an absent CLI. - cache.ts: content-addressed ResponseCache keyed on a canonical SHA-256 over every output-affecting field. - tests/helpers/fake-claude.ts + tests/eval/*: hermetic engine/cache coverage via the injected Spawn stub, no real claude. Not a plugin deployable — no build.ts entry, no version bump.
…x runner, report, CLI Grading + orchestration layer on top of the D1 `claude -p` engine (D2). - src/eval/case.ts: EvalCase / Variant types, defineCase, toRequest lowering. - src/eval/grade.ts: Grader/GradeResult, built-in schemaConforms/assert/judge (judge routes through the D1 engine, cached/stubbable), weighted gradeCase. - src/eval/runner.ts: runCase, bounded-parallel runSuite (mapPool), runAB delta record (Δcorrectness/Δcost/Δlatency/Δturns), runMatrix per-model grid; skips with notice when the CLI is unavailable. - src/eval/report.ts: aligned text table + JSON over all three axes, optional clearly-labelled projected-cost column from a static pricing table. - src/eval/cli.ts: `bun src/eval/cli.ts` — suite loading, --ab/--matrix/--filter/ --models/--json/--tolerance; report-only exit 0 except an A/B correctness regression past tolerance → exit 1; absent claude → notice + exit 0. - package.json: `eval` script (kept OUT of `test`; test stays hermetic/offline). - CLAUDE.md + conventions.md: run the eval suite alongside `bun test` when changing harness prompts (docs-only, no generated-artifact churn, no bump). - tests/eval/framework.test.ts: graders, gradeCase, runAB/runMatrix deltas, bounded pool, report text/JSON, CLI exit-code semantics — all via fakeSpawn.
Modular eval cases exercising the harness's real agent prompts against their real forced schemas from schemas.generated.ts: planner, reviewer, refuter, and implementer. Each case carries a baseline prompt snapshot copied verbatim from its live stage (holes filled with inline fixtures), imports the matching schema constant, and grades correctness with pure assert/schemaConforms predicates. Fixtures embed a compact source-plan ask, a seeded-gap plan (an AC with no covering test + a dropped docs requirement), and an obviously-weak finding. index.ts aggregates the cases as the CLI --suite export; a hermetic smoke test (tests/eval/harness-suite.test.ts) proves every case is well-formed, ids/tags are unique, the suite loads through the D2 loader, and each case's graders discriminate a good canned structured_output from a bad one via fakeSpawn — no real claude. Entirely under src/eval/ + tests/, no plugin deployable churn. Implements D3 of the evaluation-system run.
This was referenced Jul 15, 2026
cmschuetz
commented
Jul 15, 2026
| const what = typeof f.what === 'string' ? f.what.toLowerCase() : '' | ||
| return f.severity !== 'suggestion' && (what.includes('test') || what.includes('json') || what.includes('doc')) | ||
| }) | ||
| const gapInAc = ac.some(a => a.verdict === 'violation') |
Owner
Author
There was a problem hiding this comment.
gapInAc passes on any ac_checklist entry with verdict violation, not specifically the seeded gap (AC2, the untested --json mode). A live reviewer that flags an unrelated AC would satisfy the discriminator without detecting the seeded flaw — the eval can't tell a good reviewer from a lucky one. Tighten to the seeded AC id. (D3 review finding — the "eval too easy" risk.)
| schemaConforms(), | ||
| // Discriminator: the ask has independently-testable pieces (a pure resolver, | ||
| // the flag, the JSON mode, docs) → a real planner returns more than one. | ||
| assert('at-least-two-deliverables', o => { |
Owner
Author
There was a problem hiding this comment.
at-least-two-deliverables is a weak discriminator: the planner prompt it snapshots explicitly biases toward consolidation ("split only when >~1000 lines"), so a correct single-deliverable plan would fail this and a bad over-fragmented plan passes. Grade on something the seeded ask actually forces (e.g. the independently-testable pieces landing as distinct deps), not raw count. (D3 plan-review F1.)
| @@ -0,0 +1,158 @@ | |||
| // Harness eval SUITE smoke test — fully hermetic (no real `claude`). | |||
Owner
Author
There was a problem hiding this comment.
Drift-guard gap. The cases carry verbatim snapshots of the live stage prompts, but nothing asserts a snapshot still equals its live builder text. The moment someone edits a real prompt, the suite silently tests stale text — D4 already had to hand-sync these snapshots when it compacted the prompts. Add a hermetic test here that renders each live stage prompt and asserts it byte-matches the case snapshot (the guarantee the original golden-test design gave). Seeds the review loop.
Base automatically changed from
strapped/evaluation-system/D2-eval-framework
to
main
July 28, 2026 22:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add the actual eval content: modular cases that exercise the harness's real
agent prompts and schemas, turning the suite into the heavy test layer for
prompt changes and the substrate for D4's A/Bs. Each case imports a real schema
from
schemas.generated.tsand carries a baseline prompt snapshot copiedverbatim from its stage (stages are not refactored to export prompts). Cases
are one file each so adding coverage for a new harness feature is trivial.
Summary
New
src/eval/suites/harness/:planner.case.ts,reviewer.case.ts(fixtureplan with a seeded gap),
refuter.case.ts(obviously-weak finding →refuted),implementer.case.ts, sharedfixtures/, andindex.tsexportingthe tag-labelled case array the CLI
--suiteloads. A hermetictests/eval/harness-suite.test.tsproves every case is well-formed and itsgraders return the expected verdicts against canned
structured_outputviafakeSpawn— no realclaude.Acceptance criteria
refuted).--suite src/eval/suites/harness), filters by tag, and runs in parallel.tests/eval/harness-suite.test.tsproves every case is well-formed and graders return expected verdicts against canned outputs — fully hermetic.bun run typecheck && bun run lint && bun testgreen.Stack
All deliverables target the
strappedrepo (linear stack, each PR based on its parent's branch).strapped/evaluation-system/D1-eval-enginestrapped/evaluation-system/D2-eval-frameworkstrapped/evaluation-system/D3-harness-eval-suitestrapped/evaluation-system/D4-verified-optimizationsDepends on #36