Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,7 @@ Add or update a test alongside any change to engine scripts, adapters, migration

## Prose Tests

`tests/prose/` — end-to-end tests for the prose logic (design: `design/prose-tests.md`). A case is a directory under `tests/prose/cases/{case-id}/`: `case.json` (values code branches on), `fixture.md` + `fixture-state.cjs` (the starting world), `act.md` (the coarse instruction a walker agent follows through the real skills), `assert.md` (the expected path, seen only by the asserting agent), optional `assertion-state.cjs` (the world the walk should produce; absent = unchanged), and the two generated snapshots. Nothing is parsed and prose never shares a file with code. Shared pipeline stages live in `tests/prose/mainlines/`. Walks cost tokens — run on command via the `/prose-test` dev skill, never as part of a routine gate. The deterministic perimeter runs under `npm test`: corpus validation plus snapshot goldens — every snapshot must rebuild byte-identical from its recipe under the frozen clock, skipped when nothing feeding it changed. When an engine change moves a world, regenerate with `node tests/prose/run.cjs snap {case-id}` and land the snapshot diff in the same PR; never hand-edit a snapshot. Every review finding lands twice: the fix, and the case that would have caught it. At the end of a PR that touches skill prose, run `node tests/prose/run.cjs select --diff main` and suggest running the intersecting cases.
`tests/prose/` — end-to-end tests for the prose logic (design: `design/prose-tests.md`). A case is a directory under `tests/prose/cases/{case-id}/`: `case.json` (values code branches on, including the required `entry` and any deterministic `invariants`), `fixture.md` + `fixture-state.cjs` (the starting world), `act.md` (the coarse instruction a walker agent follows through the real skills), `assert.md` (the expected path, seen only by the asserting agent), optional `assertion-state.cjs` (the world the walk should produce; absent = unchanged), and the two generated snapshots. Nothing is parsed and prose never shares a file with code. **A case starts where a session starts and runs until state lands**: `entry` must name `workflow-start`, a `workflow-*-entry` skill, or `workflow-discovery` — never a navigation or processing skill, never a reference — because a walk begun mid-session lacks the context the prose was written for, and one that stops at a gate leaves an empty delta and nothing but display claims to judge. Claims that code can settle belong in `invariants` (`engine_before_write`, `calls_include`, `calls_exclude`), which run before the asserter and whose verdicts it cannot overturn. Shared pipeline stages live in `tests/prose/mainlines/`. Walks cost tokens — run on command via the `/prose-test` dev skill, never as part of a routine gate. The deterministic perimeter runs under `npm test`: corpus validation plus snapshot goldens — every snapshot must rebuild byte-identical from its recipe under the frozen clock, skipped when nothing feeding it changed. When an engine change moves a world, regenerate with `node tests/prose/run.cjs snap {case-id}` and land the snapshot diff in the same PR; never hand-edit a snapshot. Every review finding lands twice: the fix, and the case that would have caught it. At the end of a PR that touches skill prose, run `node tests/prose/run.cjs select --diff main` and suggest running the intersecting cases.

## Knowledge Base Subsystem

Expand Down
47 changes: 47 additions & 0 deletions design/prose-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,30 @@ testing.
A failure is a finished result; diagnosing it is a separate, human-led
act. This is stated in prose in each definition rather than relied on
from tool restrictions, which an agent may hold regardless.
- **P9 — a case starts where a session starts, and runs until state
lands.** The only real openings are `workflow-start` (the user's), a
`workflow-*-entry` skill (what a bridge plan file invokes after a
context clear), and `workflow-discovery` (epic continuation). Navigation
and processing skills are always reached mid-session; a reference is
never entered directly. A walk carries only the context it accumulates,
so one begun in the middle is judged under conditions the prose was
never written for, and its verdict means nothing in either direction.
Enforced in `lib/cases.cjs` by the required `entry` field, not left to
authoring discipline. The same rule ends a case: stop at a gate and the
delta is empty, the claims turn into descriptions of a display, and the
asserter is left comparing the walker's account of what it showed
against the case's account of what it should have shown — with no
independent ground truth anywhere in the loop.

- **P10 — what code can decide, code decides.** A case declares
invariants over the recorded actions (`lib/invariants.cjs`) that run
before the asserter sees anything, and whose verdicts it may not
overturn. They exist because the world delta proves the outcome but not
that it was earned — a walk that ignored every instruction and wrote
the expected files lands the same state. Not a pinned call sequence:
recording and replaying one would freeze whatever the walker happened
to do, certifying broken prose instead of catching it.

- **P7 — a failing case is a finding either way.** Either the prose
broke, or the world/design moved and the case is stale. The
adjudication is the point; only the typo-class staleness is
Expand Down Expand Up @@ -218,6 +242,29 @@ testing.

## Log

- 2026-07-26 — Two findings that dissolved a long-running mystery, and
the rules that follow from them. First: we were judging a walk by the
single message an agent returns, while the walk itself happens across
dozens of turns. Every step narrated at the time and compressed out of
the closing summary read as a step never taken — the "under-reporting"
blamed first on the walker, then on the model, was ours. A Sonnet walk
of continue-feature was marked down for skipping the selection menu;
its transcript shows it emitted the display, emitted the menu verbatim,
stopped, consumed the scripted answer and validated the selection. An
Opus walk of the same case made byte-identical tool calls. The runtime
writes every turn already and the stop payload names the file, so the
hook now lifts them into the world (P6e). Second, and underneath it:
four cases started where no session ever starts — two at navigation
skills, one at a processing skill, one inside a reference. They are
retired, and the rule is enforced rather than remembered (P9). The
residue is now decided in code where it can be (P10). Along the way the
recorder was found to have been reading `tool_output` when the field is
`tool_response`, so every successful call had recorded its status and
nothing else, and the stop hook had never once fired — its world is
resolved from the transcript now, which also stamps the model on every
verdict, since an unreloaded definition can otherwise run a walk on a
model nobody chose.

- 2026-07-26 — The first round that found more prose than instrument.
Recording finally live (the blocker was `hasTrustDialogAccepted`, which
gates frontmatter hooks in project subagents and fails silently), five
Expand Down

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

13 changes: 0 additions & 13 deletions tests/prose/cases/continue-feature-routes-to-discussion/assert.md

This file was deleted.

11 changes: 0 additions & 11 deletions tests/prose/cases/continue-feature-routes-to-discussion/case.json

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

Loading