[docs] record the adversarial review and what it is not - #124
Merged
Conversation
Tests answer "does the thing I thought of still work". They cannot answer "what didn't I think of", which on a tool that drives production daemons is the expensive question — so the tree is swept by lane-separated reviewers on Claude Fable 5, each told to refute a finding before reporting it and that an empty report is a fine outcome. Written down with its limits attached: it is not a tier, it is not reproducible, it guards nothing, and machine reviewers file confident findings about code that does not exist. A finding is handled only when it lands as a fix plus a test that fails without it.
Contributor
There was a problem hiding this comment.
Pull request overview
Documents the repository’s “adversarial review” practice (and its limits) across the main reader-facing entry points, emphasizing that it is not a test tier and does not provide ongoing regression protection.
Changes:
- Adds an “Adversarial review — and why it is deliberately not a tier” section to
docs/testing.md, including lane definitions and prompt discipline. - References this practice from
CONTRIBUTING.mdandREADME.md. - Adds a “what review this code has had — and what it hasn’t” section to
SECURITY.md, explicitly stating non-audit status and limitations.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| SECURITY.md | Adds a section describing review practices and explicitly stating what they do not imply. |
| README.md | Adds a short “adversarial review” note alongside the testing overview. |
| docs/testing.md | Adds the main documentation defining adversarial review lanes and why it isn’t a test tier. |
| CONTRIBUTING.md | Adds a pointer to the adversarial review write-up in the review discipline section. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| untrusted input, backend correctness, frontend), each instructed to refute a | ||
| finding before reporting it. A finding is handled the same way a report from you | ||
| would be: a fix, plus a regression test that fails without it. The method and its | ||
| limits are written up in [docs/testing.md](docs/testing.md#adversarial-review--and-why-it-is-deliberately-not-a-tier). |
| to refute a finding before reporting it. It is not a smoke test and it is not a | ||
| tier: it finds what to test, and a finding is only handled once it lands as a fix | ||
| plus a test that fails without it. See | ||
| [docs/testing.md](docs/testing.md#adversarial-review--and-why-it-is-deliberately-not-a-tier). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Writes down the review practice that sits on top of the test tiers, in the three
places that have different readers — and, in each, attaches its limits.
docs/testing.md— new section "Adversarial review — and why it isdeliberately not a tier": lane-separated reviewers (auth & crypto,
authorization, untrusted input, backend correctness, frontend) running on
Claude Fable 5, with the prompt discipline that makes it worth anything —
read the code rather than infer it, refute a candidate finding before
reporting it, say whether a test already covers the guard, and treat an empty
report as a respectable outcome.
CONTRIBUTING.md— a paragraph in Review discipline pointing at it.README.md— one sentence in How it's tested.SECURITY.md— a new section, written the other way round: what this reviewis not. No third-party audit has been done; a lane coming back clean is
evidence about that lane on that day; findings are fixed before they are
described publicly, so an older release is not the same as the current one.
The through-line, stated in every one of them: a review is not a tier. It is
not reproducible, does not run in CI, and guards nothing by itself — the next
regression walks straight past it. It finds what to test. A finding counts as
handled only when it lands as a fix plus a test that fails without the fix, and
where the finding is an instance of a class, plus the sweep that enumerates the
class.
It also records that machine reviewers file confident findings about code that
does not exist — with the concrete example from this repo, where an automated
review on a recent PR raised imports the file never had and a missing
Bearerprefix that was already there. Without that caveat the section would read as
marketing.
Type of change
Checklist
go test -short ./...andgo vet ./...pass — N/A (no Go changed)gofmtgate is clean — N/A (no Go changed)web/srcchange)web/dist— N/A (noweb/srcchange)docs/and added aCHANGELOG.mdentry — docs only, no app behaviour changed, so no CHANGELOG entryNotes for reviewers
Deliberately not included: anything specific about what the most recent sweep
turned up. The repository is public, so the section describes the method and not
current findings.