test(prose): only walks that can be observed, and checks that survive the trip

[leeovery]

Summary

Two findings from running all nine cases. Eight passed on Sonnet, and every invariant derived from prose in #565 held up against a live walk — that PR's open caveat is now closed.

The ninth came back INVALID, twice. smoke-start-boot-structure was the one structure-only case — read the prose, trace the logic, execute nothing — so it built no world. No world means the hook records no actions and there's no delta, so the asserter received an expected path and no evidence of any kind. It refused to judge, correctly: nothing about that walk was observable, so nothing it claimed could be answered.

It was scaffolding from before real cases existed, and its one claim (casing conventions load before the boot pipeline) is covered by start-lists-active-work, which walks the same skill against a world. So it goes, and a world becomes required — removing the category rather than leaving a trap for the next author. The branches serving it go too: the structural walker prompt, the world: null response, the no-world arms in prompt / diff / assert.

The second is subtler. The orchestrator's verdict block had no line for the deterministic checks, so the asserter reported them and the orchestrator dropped them. Nine PASSes came back and the only reason to believe the checks passed was the rule that a failing one forces FAIL — an inference, about the one part of the result that exists precisely so nothing has to infer anything. They now travel through to the block and into the skill's collation table, unsummarised.

Test plan

• Prose suites 75/75; corpus valid at 8 cases
• The live run that produced both findings: 8/8 PASS on claude-sonnet-5, all worlds destroyed, tree clean

🤖 Generated with Claude Code

Stack

1. docs(design): prose-tests programme design log #544
2. feat(prose-tests): the framework — cases, worlds, runner, skill #545
3. test(prose): feature happy-path corpus — five worlds, seven cases #546
4. test(prose): bugfix corpus — the investigation-centric surfaces #548
5. test: retry recursive teardown removals — kill a class of phantom failures #549
6. fix(entry-skills): close the handoff fences — six files render their arms wrong #550
7. docs: a contributing page for working on the system #551
8. fix(entry-skills): every handoff arm says to invoke the skill #552
9. fix(implementation): environment setup belongs to the setup reference alone #553
10. fix(prose-tests): the asserter is told which substitutions were armed #554
11. feat(prose-tests): the mid-flow substitution, and a world only prose can describe #555
12. test(prose): claims assert consequences, not what was displayed #556
13. feat(prose-tests): record everything the agents do, results included #557
14. fix(discussion-entry): the handoff reports the source it actually had #558
15. fix(prose-tests): the stop hook records, and names the model that walked #559
16. fix(prose-tests): command output was never actually recorded #560
17. feat(prose-tests): judge the walk as told, not the summary returned #561
18. feat(prose-tests): decide in code what an agent should not be deciding #562
19. test(prose): a case starts where a session starts #563
20. feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure #564
21. test(prose): give the eight read-only cases something that can fail #565
22. test(prose): only walks that can be observed, and checks that survive the trip #566 👈 current
23. fix(prose-tests): the verdict names only the model the record names #567
24. test(prose): discovery, walked to the point where work first exists #568
25. fix(prose-tests): the asserter judges which of prose or walker was at fault #569
26. docs(conventions): a step whose reference routes every exit still signposts #570
27. test(prose): discovery's epic arm, to the same durability boundary #571