Skip to content

fix(prose-tests): the asserter is told which substitutions were armed#554

Open
leeovery wants to merge 1 commit into
fix/environment-check-gathers-onlyfrom
prose-tests/asserter-knows-stubs
Open

fix(prose-tests): the asserter is told which substitutions were armed#554
leeovery wants to merge 1 commit into
fix/environment-check-gathers-onlyfrom
prose-tests/asserter-knows-stubs

Conversation

@leeovery

@leeovery leeovery commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • The gap the recording created. The action log shows a stub as the walker writing a report an agent would have produced. An asserter that has never heard of the stub reads that as a missing agent dispatch — root-cause-validation-clean-verdict failed 2/5 on precisely that, having behaved correctly throughout. Armed stubs now reach the asserter with their triggers and descriptions, as a distinct prompt section (P4b).
  • An authoring rule from the same round (P4c). A claim about something displayed — "offers the choice and waits" — can never be evidenced by the action log, because emitting text is not a tool call. It rests on the walker's narrative, the one piece of evidence it authors itself. Most display claims have a consequence that can be asserted: the answer consumed, the arm taken, the state produced. Prefer the consequence.
  • Deletes smoke-boot-in-world. It absorbed three authoring fixes in one day — a contradictory stop point, then claims about renderings after that stop, then a scope anchor (#Boot) fighting its own claims — for a framework smoke test whose job start-lists-active-work already does properly and passes 5/5.

Test plan

  • npm test green: 1698 tests, 0 fail (one fewer suite entry: the deleted case's snapshot check).
  • Corpus revalidates: 12 cases, 1 stub.
  • root-cause-validation-clean-verdict should now be judged correctly rather than failing on its own substitution — worth a re-run once this lands.

🤖 Generated with Claude Code

Stack

  1. docs(design): prose-tests programme design log #544
  2. feat(prose-tests): the framework — cases, worlds, runner, skill #545
  3. test(prose): feature happy-path corpus — five worlds, seven cases #546
  4. test(prose): bugfix corpus — the investigation-centric surfaces #548
  5. test: retry recursive teardown removals — kill a class of phantom failures #549
  6. fix(entry-skills): close the handoff fences — six files render their arms wrong #550
  7. docs: a contributing page for working on the system #551
  8. fix(entry-skills): every handoff arm says to invoke the skill #552
  9. fix(implementation): environment setup belongs to the setup reference alone #553
  10. fix(prose-tests): the asserter is told which substitutions were armed #554 👈 current
  11. feat(prose-tests): the mid-flow substitution, and a world only prose can describe #555
  12. test(prose): claims assert consequences, not what was displayed #556
  13. feat(prose-tests): record everything the agents do, results included #557
  14. fix(discussion-entry): the handoff reports the source it actually had #558
  15. fix(prose-tests): the stop hook records, and names the model that walked #559
  16. fix(prose-tests): command output was never actually recorded #560
  17. feat(prose-tests): judge the walk as told, not the summary returned #561
  18. feat(prose-tests): decide in code what an agent should not be deciding #562
  19. test(prose): a case starts where a session starts #563
  20. feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure #564
  21. test(prose): give the eight read-only cases something that can fail #565
  22. test(prose): only walks that can be observed, and checks that survive the trip #566
  23. fix(prose-tests): the verdict names only the model the record names #567
  24. test(prose): discovery, walked to the point where work first exists #568
  25. fix(prose-tests): the asserter judges which of prose or walker was at fault #569
  26. docs(conventions): a step whose reference routes every exit still signposts #570
  27. test(prose): discovery's epic arm, to the same durability boundary #571

@leeovery
leeovery force-pushed the fix/environment-check-gathers-only branch from f4c27d7 to 42108f0 Compare July 26, 2026 12:52
@leeovery
leeovery force-pushed the prose-tests/asserter-knows-stubs branch from 872d47c to db46617 Compare July 26, 2026 12:53
@leeovery
leeovery force-pushed the fix/environment-check-gathers-only branch from 42108f0 to 4c916fa Compare July 26, 2026 13:37
@leeovery
leeovery force-pushed the prose-tests/asserter-knows-stubs branch from db46617 to e81714b Compare July 26, 2026 13:37
@leeovery
leeovery force-pushed the fix/environment-check-gathers-only branch from 4c916fa to 1976e37 Compare July 26, 2026 13:43
First round with recording live exposed the gap: the action log shows a
stub as the walker writing a report an agent would have produced, and an
asserter that has never heard of the stub reads that as a missing
dispatch. root-cause-validation failed 2/5 on exactly that, having
behaved correctly throughout. Armed stubs now reach the asserter with
their triggers and descriptions.

Also from the same round, as an authoring rule (P4c): a claim about
something *displayed* can never be evidenced by the action log, because
emitting text is not a tool call — it rests on the walker's narrative,
the one piece of evidence it authors itself. Most display claims have a
consequence that can be asserted instead: the answer consumed, the arm
taken, the state produced.

And smoke-boot-in-world is deleted. It absorbed three authoring fixes —
a contradictory stop point, claims about post-stop renderings, then a
scope anchor fighting its own claims — for a framework smoke test whose
job start-lists-active-work already does properly, and passes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@leeovery
leeovery force-pushed the prose-tests/asserter-knows-stubs branch from e81714b to f4cfb5a Compare July 26, 2026 13:43
This was referenced Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant