Skip to content

[docs] record the adversarial review and what it is not - #124

Merged
malickyeu merged 1 commit into
mainfrom
docs/adversarial-review
Aug 3, 2026
Merged

[docs] record the adversarial review and what it is not#124
malickyeu merged 1 commit into
mainfrom
docs/adversarial-review

Conversation

@malickyeu

Copy link
Copy Markdown
Contributor

Summary

Writes down the review practice that sits on top of the test tiers, in the three
places that have different readers — and, in each, attaches its limits.

  • docs/testing.md — new section "Adversarial review — and why it is
    deliberately not a tier"
    : lane-separated reviewers (auth & crypto,
    authorization, untrusted input, backend correctness, frontend) running on
    Claude Fable 5, with the prompt discipline that makes it worth anything —
    read the code rather than infer it, refute a candidate finding before
    reporting it
    , say whether a test already covers the guard, and treat an empty
    report as a respectable outcome.
  • CONTRIBUTING.md — a paragraph in Review discipline pointing at it.
  • README.md — one sentence in How it's tested.
  • SECURITY.md — a new section, written the other way round: what this review
    is not. No third-party audit has been done; a lane coming back clean is
    evidence about that lane on that day; findings are fixed before they are
    described publicly, so an older release is not the same as the current one.

The through-line, stated in every one of them: a review is not a tier. It is
not reproducible, does not run in CI, and guards nothing by itself — the next
regression walks straight past it. It finds what to test. A finding counts as
handled only when it lands as a fix plus a test that fails without the fix, and
where the finding is an instance of a class, plus the sweep that enumerates the
class.

It also records that machine reviewers file confident findings about code that
does not exist — with the concrete example from this repo, where an automated
review on a recent PR raised imports the file never had and a missing Bearer
prefix that was already there. Without that caveat the section would read as
marketing.

Type of change

  • Bug fix
  • New feature
  • Docs only
  • Refactor / chore

Checklist

  • go test -short ./... and go vet ./... pass — N/A (no Go changed)
  • gofmt gate is clean — N/A (no Go changed)
  • Frontend type-checks — N/A (no web/src change)
  • Rebuilt and committed web/dist — N/A (no web/src change)
  • Added/updated tests for the change — N/A (documentation only)
  • Updated docs/ and added a CHANGELOG.md entry — docs only, no app behaviour changed, so no CHANGELOG entry

Notes for reviewers

Deliberately not included: anything specific about what the most recent sweep
turned up. The repository is public, so the section describes the method and not
current findings.

Tests answer "does the thing I thought of still work". They cannot answer
"what didn't I think of", which on a tool that drives production daemons is
the expensive question — so the tree is swept by lane-separated reviewers
on Claude Fable 5, each told to refute a finding before reporting it and
that an empty report is a fine outcome.

Written down with its limits attached: it is not a tier, it is not
reproducible, it guards nothing, and machine reviewers file confident
findings about code that does not exist. A finding is handled only when it
lands as a fix plus a test that fails without it.
Copilot AI review requested due to automatic review settings August 3, 2026 19:56
@malickyeu
malickyeu merged commit 615366c into main Aug 3, 2026
4 checks passed
@malickyeu
malickyeu deleted the docs/adversarial-review branch August 3, 2026 19:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Documents the repository’s “adversarial review” practice (and its limits) across the main reader-facing entry points, emphasizing that it is not a test tier and does not provide ongoing regression protection.

Changes:

  • Adds an “Adversarial review — and why it is deliberately not a tier” section to docs/testing.md, including lane definitions and prompt discipline.
  • References this practice from CONTRIBUTING.md and README.md.
  • Adds a “what review this code has had — and what it hasn’t” section to SECURITY.md, explicitly stating non-audit status and limitations.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
SECURITY.md Adds a section describing review practices and explicitly stating what they do not imply.
README.md Adds a short “adversarial review” note alongside the testing overview.
docs/testing.md Adds the main documentation defining adversarial review lanes and why it isn’t a test tier.
CONTRIBUTING.md Adds a pointer to the adversarial review write-up in the review discipline section.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread SECURITY.md
untrusted input, backend correctness, frontend), each instructed to refute a
finding before reporting it. A finding is handled the same way a report from you
would be: a fix, plus a regression test that fails without it. The method and its
limits are written up in [docs/testing.md](docs/testing.md#adversarial-review--and-why-it-is-deliberately-not-a-tier).
Comment thread CONTRIBUTING.md
to refute a finding before reporting it. It is not a smoke test and it is not a
tier: it finds what to test, and a finding is only handled once it lands as a fix
plus a test that fails without it. See
[docs/testing.md](docs/testing.md#adversarial-review--and-why-it-is-deliberately-not-a-tier).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants