story-016: Assert the coordinator's output shapes directly instead of against a historical implementation

[jerodw]

Summary

Six tests in tests/test_story_011_validation.py loaded the pre-story-011 coordinator out of git history, ran it beside the current one, and asserted the two produced identical output — same events.log, same state.json, same escalation summary, same run-directory contents.

That was the right instrument for story-011, whose entire constraint was that adding execution-history.json changed nothing else. Once merged it said something different: that the coordinator's output may never differ from what one implementation produced on one day, measured against whatever workflow ships today. Nobody holds that requirement. story-012 adds retry-history.json; story-014 adds clean-clone-result.json. Each legitimately adds an artifact and an event, and story-014 escalated on nine failures — every one of them this comparison, in a file inside that story's do_not_modify, so no bounded retry could reach it.

The instrument also had a shelf life independent of that. The historical coordinator is run against today's workflow, schemas and config, a pairing that grows more artificial each story until the old code cannot run at all — not because anything broke, but because it is old.

This story replaces equality-with-the-past with direct statements of the shapes the harness wants now. A shape stated outright can be read, argued with, and deliberately changed by a later story that means to change it. Equality with a frozen implementation can only be broken.

Changes

• tests/test_coordinator_contract.py (new) — the standing home for the coordinator's output contract, deliberately not named for a story and not one story's evidence:
  • state.json's exact field set, read from dataclasses.fields(RunState) rather than typed out, so the assertion cannot drift from the definition it describes; each field's type; and the statuses a run can end in, exercised by runs that actually reach them.
  • The events.log line pattern — timestamp shape, brackets, single separating space.
  • The escalation summary's five parts, located in the text of a real escalated run.
  • The run directory's required subset of artifacts, derived from the loaded workflow's outputs, permitting others. This is the assertion that removes the friction: a later story adding an artifact no longer fails a test about something else.
• tests/test_story_011_validation.py — the six comparisons removed, along with both_implementations, legacy_coordinator, clone_target, normalize_timestamps and SHAPES, which served only them. Net 32 insertions / 138 deletions. Everything that never depended on the historical coordinator survives untouched: the log-line-to-history correspondence, the ordering and retry-stream checks, the schema conformance of a run's history, and the prompt-scope assertion with its baseline resolution intact. No module under tests/ loads a coordinator implementation out of git history any longer.
• tests/test_story_016_validation.py (new, tester stage) — independent validation: that each new contract assertion can be made to fail, that the removals took only unreachable code, and that a run gaining an artifact and an event no longer fails anything.
• .harness/docs/ARCHITECTURE.md — records where the output contract now lives, the rule that a differential test against a frozen implementation should be retired once the constraint it was built for has landed, the corollary that the restatement must land before the removal with each new assertion demonstrated red, and the standing rule not to assert a run directory as an exact set (with the schemas inventory named as the deliberate exception, since there the point is that a new shape cannot appear unnoticed).

Testing

479 passed on this branch.

The verifier did not accept the new assertions on description. It re-ran a mutation probe against the contract file and recorded each result: log format without brackets caught, separator drift caught, timestamp drift caught, state.json gaining a field RunState does not declare caught, retry_count written as a string caught, escalation summary losing Stage: caught, completion-report.md missing caught by the required-subset check, unmutated control clean.

It also confirmed the surviving prompt-scope assertion still bites rather than passing unconditionally: the real range over prompts/, workflows/, rules/ is clean, while a widened range containing prompt edits returns 548 diff lines.

Notes for review

• This story retried once, and the harness caught it rather than a human. Attempt 1's tests/test_story_016_validation.py resolved its "before" copy of the story-011 file as git show HEAD:.... In the working tree that diff was real; in a clean clone with the story committed, the baseline was the story's own file, so the removal assertion compared the file against itself — one test failed and two neighbouring diff assertions went silently green, the more dangerous outcome. This is the third appearance of the HEAD-baseline trap, and the first time it was caught automatically. Attempt 2 resolves the baseline by walking git log -- tests/test_story_011_validation.py newest-first for the first blob still carrying all six removed tests, raising loudly when none exists, with three tests driving that resolution against synthetic histories (post-story newest, further commits stacked on top, and the refusal case). ARCHITECTURE.md records that the trap applies to any test resolving a baseline out of git, not only to differential tests of orchestration code.
• The branch was rebuilt, and the verifier's numbers reflect the tree it ran in. story-016 executed in a working tree that still carried story-014's uncommitted implementation — story-014 had escalated without committing — and _complete's git add -A swept all of it into the story's commit. That is exactly the defect the commit-only-what-the-run-produced request describes. This branch was reconstructed by cherry-picking onto main with story-014's files reverted, so it contains only story-016's five files. Consequence for reviewers: the verifier's evidence cites 530 passed because story-014's 826-line test file was present; this branch gives 479 passed. story-016's own tests never depended on story-014's code — the only contact is a synthetic clean-clone-result.json written into a fixture to prove the required-subset assertion tolerates an artifact it does not name.
• story-014 is unaffected and will be re-run after this lands. Its escalation was caused entirely by the six comparisons this story removes.
• Two items the verifier recorded rather than raised: .harness/docs/ARCHITECTURE.md was still unwritten at verification time (documenter stage, not an acceptance criterion), and the implementation summary names the new guard test test_the_baseline_is_not_this_storys_own_file while the file defines test_the_baseline_walks_past_this_storys_own_commit — the test exists and does what the summary describes; only the prose name is wrong.

🤖 Generated with Claude Code