Record what each published row was measured under - #65
Merged
Conversation
`export-results --merge` carried forward every cell a run did not re-execute, with no notion of a row going out of date. The 25 August snapshot held 114 rows across six execution dates spanning thirteen days — 22 of them from 13 August, measured before the CLI moved to 2.5.0, before fixed sleeps became polling, before several scorer corrections and before the base prompt gained "Do not ask clarifying questions". Nothing in the file said so, and the `ranAt` comment already described the problem without acting on it. Each row now carries two fingerprints, following vercel-labs/agent-eval: `harness` over the base prompt and the CLI and skills pins, `scenario` over the scenario's own files. Two rather than one because "this scenario was rewritten" and "the harness moved under everything at once" want different responses and a single hash cannot tell them apart. The merge reports what it is carrying, by date, and which rows were measured under something else. It still publishes them: `--drop-stale` is opt-in, because a snapshot with holes and a snapshot with silent thirteen-day-old rows are both wrong and which is less wrong depends on what the snapshot is for. The defect in #60 was that nothing was ever *said*, not that stale rows were carried. Also: apps/framework had test files that no command ran. `redact.test.ts` guards the credential leak that reached a public artifact on the first real run, and has never been executed by CI. Added a `test` script, so `pnpm -r test` now covers it and the new provenance tests — 13 tests that previously ran nowhere. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
This was referenced Aug 28, 2026
leggetter
added a commit
that referenced
this pull request
Aug 28, 2026
…picked The ticket said "so they receive order events at their endpoint" and named no endpoint, so any reachable URL satisfied it. Across the six stored cells of this scenario the agents picked six different receivers: three Event Gateway sources on hkdk.events, a webhook.site inbox, a mock.hookdeck.com path, and a localtunnel inside the agent's own sandbox. Five passed. The sixth is the one worth reading. It failed, and not because the customer was not subscribed — the agent built the tenant, the destination and the topics, and reported delivery confirmed. Its tunnel died with the container before scoring. So the check named "an order event reaches the customer" was discriminating on whether an improvised receiver outlived the run. EVAL.ts already carried this as a known hole and deferred it, because fixing it changes a published scenario. That is the right call to revisit now rather than later: publishing is held under #66, the scenario fingerprint from #65 records the change for anyone comparing rows across it, and the next matrix has not been paid for yet. Doing it after that run would mean measuring this scenario twice. The ticket now names mock.hookdeck.com/api/v1/acme/orders — the URL SOLUTION.ts was already delivering to, which is how the mismatch surfaced — and the scorer counts only destinations pointed there. A customer's endpoint is not the integrator's to choose. Verified with score-only rather than an agent run: the solution path passes 2/2, the unsolved path fails, and a solution edited to deliver to a receiver of its own choosing now fails where it would previously have passed. Refs #47 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The stale-rows fix for #60.
The problem
export-results --mergecarries forward every cell a run did not re-execute. That earns its place — the-no-skillstwins refresh monthly, so without it a weekly snapshot would have holes. What it lacked was any notion of a row going out of date.The 25 August snapshot holds 114 rows across six execution dates spanning thirteen days, 22 of them from 13 August — measured before the sandbox CLI moved to 2.5.0, before fixed sleeps became polling, before several scorer corrections, and before the base prompt gained "Do not ask clarifying questions" yesterday. Nothing in the file said so.
The
ranAtfield's own comment already described this ("the published grid is not one snapshot… two cells side by side can be a month apart"). It just never acted on it.The change
Each row now carries two fingerprints, following vercel-labs/agent-eval:
harness— the base prompt, the Hookdeck CLI pin, the skills CLI pin, and the skills submodule commit. What every scenario shares.scenario— a walk of the scenario's own files: prompt, scorer, seed, solution, fixtures.Two rather than one because "this scenario was rewritten" and "the harness moved under everything at once" want different responses, and a single hash cannot distinguish them.
--mergenow reports what it is carrying, by execution date, and which rows were measured under something else:It still publishes them.
--drop-staleis opt-in, because a snapshot with holes and a snapshot with silent thirteen-day-old rows are both wrong, and which is less wrong depends on what the snapshot is for. A release cut to report a measured change wants it on; a weekly refresh keeping the page populated probably does not. The defect in #60 was that nothing was ever said — not that stale rows were carried.Rows predating the field report as "no provenance recorded" and are never counted as current. Not knowing what a row was measured under is the condition this exists to surface.
A gap found while doing it
apps/frameworkhad test files that no command ran. It has notestscript, sopnpm -r testskipped it entirely — includingredact.test.ts, which guards the credential leak that reached a public artifact on the first real run. That test has never been executed by CI.Added the script.
pnpm -r testnow covers 13 framework tests that previously ran nowhere, and frameworkcheckruns them too.Verification
pnpm typecheckclean,pnpm -r testgreen (267 tests, up from 254),pnpm -r buildclean. Five new tests cover the scenario fingerprint: stable for identical content, changes on scorer and prompt edits, covers files added later without being told about them, ignores dotfiles andnode_modules.Not in this PR
Re-scoring stored trajectories — τ³-bench's approach, and the clean fix — needs #21 (durable transcripts) first. This makes staleness visible; that would make it repairable without re-running an agent.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA