Testing Methodology

Testing Methodology

This page describes the actual pipeline /16-eyes audit and /16-eyes audit-diff run — both use the identical engine; the only difference is scope (whole repo vs. a diff). The logic described here is plain text, not a black box — see skills/16-eyes/references/audit-flow.md and audit-diff-flow.md in the repo for the exact prompts and code.

1. Profile & lens design (/16-eyes init, once)

Before any audit runs, /16-eyes init does two agent calls:

1. Profile — explores the repo's manifests, lockfiles, directory layout, README, CI config, and entry points to identify languages/frameworks, a domain summary, an architecture summary, and a list of risk_relevant_subsystems — concrete things this repo has that matter for security (e.g. "handles payment/money movement", "has public webhooks", "calls an LLM with user-controlled input", "parses uploaded files", "runs SQL built from user input somewhere").
2. Lens design — given that profile, designs a list of investigation lenses: each one a specific, non-overlapping area an independent subagent will investigate, complete with the exact instructions that subagent will receive. Categories that don't apply to the repo are skipped; repo-specific ones that do apply are added. A small single-purpose service might get 6–8 lenses; a large multi-domain backend might get 18–20+.

The result — { profile, lenses } — is persisted to .16-eyes/lenses.json. Every later audit or audit-diff run reuses this set instead of re-profiling and re-designing from scratch. This is a deliberate choice: it keeps every run consistent and comparable, and it means audit-diff (which only sees a diff, not the whole repo) can still run repo-aware lenses instead of generic ones.

If .16-eyes/lenses.json doesn't exist when audit/audit-diff is invoked, this same profile+lens-design step runs automatically first (non-interactively, with sane defaults) — safe inside a headless CI job with no human present. init can always be re-run afterward to customize exclude patterns, depth, output location, or language.

2. Lenses → verification (every run)

Each persisted lens investigates independently — across the whole repository for audit, or scoped to a diff's changed hunks for audit-diff (the lens's own prompt is wrapped with the diff content and an instruction to stay inside it). A lens returns a list of candidate findings: title, file, line, description, plus an initial impact/probability guess.

Every candidate finding then gets an independent skeptical re-check — a second agent call that re-reads the actual code at that location (not just the first pass's description) and decides:

• is_real — is this a genuine issue in the code as it exists today, not hypothetical, not already mitigated elsewhere, not dead code?
• impact / probability — high/medium/low, based on real callers and current config, not "possible in theory".
• fix_type — safe (a purely mechanical fix, no behavior change for any legitimate flow) or risky (fixing it changes behavior, touches money/auth, or needs a product decision).
• exploit_scenario, why, suggested_fix.

The verifier prompt explicitly instructs: "Be skeptical. A finding that sounds scary in the abstract but isn't actually reachable, or is already handled by a check elsewhere, is NOT real — say so."

Dedup: findings are deduplicated by file:line as lenses report them, so two lenses flagging the same location don't get verified (and counted) twice.

3. Adversarial review (high-impact findings only)

Findings the verifier classified as impact: high go through one more round: several independent reviewer agents (3 by default, configurable via adversarial.votesPerFinding, 1–5) each try hard to refute the finding — read the actual code, and look for a guard elsewhere, a precondition that can't occur, dead code, a framework default that already prevents it, or a misread. A finding is dropped ("refuted") only if a majority of reviewers fail to find it real; it survives otherwise. If a reviewer is uncertain, the prompt tells it to default to refuted: true — a finding has to survive skepticism, not just avoid being disproven.

Refuted high-impact findings aren't silently deleted — they're listed in the report's appendix, so nothing about the audit's decisions is hidden.

4. Guards against a bad model response

Two specific failure modes are handled explicitly in code, not left to hope:

• Corrupted/placeholder responses. If every string field in a verifier's or refuter's structured response is literally the word "test" (a known failure pattern for a corrupted or placeholder model output), that response is discarded and the finding is routed to a "corrupted — needs manual review" bucket instead of being silently trusted either way.
• "Zero valid votes ≠ survived." If every refuter in an adversarial round returns a corrupted response (leaving zero valid votes), the finding is not automatically treated as having survived adversarial review — a bug pattern that's easy to introduce (e.g. refuted_count < total / 2 is true when both are 0) and is guarded against explicitly: survival requires at least one valid vote.

5. Classification & report

Findings that survive all of the above are split into two buckets:

• SAFE — mechanical fix, no behavior change. /16-eyes fix can apply these directly.
• RISKY — needs a human decision before fixing (touches money/auth, changes behavior a real flow might depend on, or needs a product tradeoff). /16-eyes fix presents these one at a time and requires an explicit yes before writing anything.

The report is written as both a Markdown file (human-readable, with an executive summary, a risk matrix, and an appendix of everything discarded and why) and a JSON companion (machine-readable, consumed by /16-eyes fix).

Cost note

Every agent call in this pipeline — profile, lens design, lens investigation, verification, adversarial refutation, synthesis — is explicitly pinned to run on Sonnet, regardless of which model the invoking Claude Code session itself is using. A full audit can be dozens of subagent calls; letting that fan-out silently inherit a more expensive model tier would make cost unpredictable.