Skip to content

docs: case study — substantive blind contract testing#2

Merged
dscherm merged 1 commit into
masterfrom
docs/blind-contract-case-study
Jul 7, 2026
Merged

docs: case study — substantive blind contract testing#2
dscherm merged 1 commit into
masterfrom
docs/blind-contract-case-study

Conversation

@dscherm

@dscherm dscherm commented Jul 6, 2026

Copy link
Copy Markdown
Owner

Summary

Adds docs/case-study-blind-contract-testing.md — a worked example of a substantive blind contract-test pass, framed in Themis's own terms as both a template (what such a pass should look like) and a motivation (why doing it on trust isn't enough).

Origin

While reviewing a ~1,000-line feature in a sibling project (an optional beads task backend in ralph-universal), we ran a second, independent verification pass in the Themis method, by hand: retroactive acceptance criteria → a fresh writer blind to the implementation → black-box contract tests → 10/10 against the live system, independently re-run to confirm.

What the doc contributes

  • The template half — the properties that make a blind pass substantive rather than a rubber stamp: derived from the spec and blind to the code, black-box only, two-sided assertions (not-ready-before and ready-after), real concurrency for the atomic-claim / no-clobber criteria, and the writer independently reconstructing a failure mode instead of reading about it.
  • The motivation half — the honest gap: every good property rested on an instruction ("don't read src"), executed through an in-conversation Agent-tool subagent — the exact spawn path this repo's own impossible-ac-results.md §4 measures as ~10× leakier than claude -p. It held, but "it held once" is an anecdote, not a guarantee. A mapping table then shows how Themis makes each property structural (read-deny hook, SHA-256 seal, separate runner, arbiter).
  • Closes with the independence ≠ strength caveat, consistent with the README's "necessary, not sufficient" framing.

Notes

  • Docs-only; no code or behavior change.
  • The README is intentionally left unedited (tightly authored, no examples index). Happy to add a one-line pointer near the limitations / related-work discussion if wanted.

🤖 Generated with Claude Code

…gap it motivates

A worked example (from a ~1000-line beads backend in ralph-universal) of what
a substantive blind contract-test pass looks like: retroactive AC -> a fresh
writer blind to the implementation -> black-box tests -> 10/10 against the
live system, independently re-verified.

Framed honestly in Themis's own terms: the blindness here was
INSTRUCTION-enforced, run through an in-conversation Agent-tool subagent —
the exact highest-leakage spawn path impossible-ac-results.md §4 measures. It
held, but 'it held once' is an anecdote, not a guarantee. The doc maps each
good property (writer can't see src, tests unedited, fresh runner, no silent
delete) to how Themis makes it structural (read-deny hook, SHA-256 seal,
separate runner, arbiter), so the example doubles as a template and as
motivation for the gate. Includes the independence-not-strength caveat.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dscherm
dscherm merged commit e2d22a7 into master Jul 7, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant