Limitations and Honest Caveats

Limitations & Honest Caveats

This page states plainly what 16 Eyes doesn't do and where its guarantees are weaker, alongside the methodology pages. The full formal versions of some of this live in the repo at docs/security-and-safety.md, docs/privacy-policy.md, and docs/terms-of-service.md — this is the short version.

What it is (and isn't)

16 Eyes is a developer-assistance tool: it reads a repository or a diff, produces a report, and — only on a separate, explicit invocation (/16-eyes fix) — proposes code edits.

• It never automatically merges, commits, or pushes code.
• /16-eyes fix applies safe findings directly (mechanical, no behavior change), but presents every risky finding one at a time and requires an explicit yes before writing anything.
• It never takes autonomous action on a live system — no deploys, no calls to external APIs on your behalf beyond the AI model itself, and no decisions about people (no biometric identification, scoring, or employment/credit/access decisions).
• Its output is a report a human reads and acts on — the same shape as a human contractor's audit findings, not an autonomous agent making consequential decisions unsupervised.

It is not a substitute for professional security testing

An LLM-driven audit — however many independent passes verify each finding — is not a replacement for a manual penetration test, a paid third-party security audit, or a dedicated SAST/DAST tool with its own track record. Treat it as one more layer: fast, repo-aware, and free of the "single pass over a diff" limitation that most CI-wired scanners have — not a certification that a codebase is secure.

No compliance certification is claimed

No SOC 2, ISO 27001, or similar third-party certification exists for this project — none is claimed, because none has been obtained. It's a small open-source project maintained without a formal compliance program.

EU AI Act: 16 Eyes produces a report for human review and never acts autonomously on production systems or on decisions about individuals — on that basis, it does not appear to fall under the Annex III high-risk use cases (biometric identification, critical infrastructure, employment, credit scoring, law enforcement, etc.). This is the maintainer's own good-faith reading of the Act's scope, not a formal legal conformity assessment. If your organization has its own EU AI Act obligations, get your own counsel to confirm classification before relying on this in a context where that matters.

Weaker guarantees outside Claude Code

Claude Code's Workflow tool runs a script-authored pipeline with JSON-schema-enforced agent calls — the fan-out, verify, and adversarial-review stages in the Testing Methodology page are hard-enforced by the tool itself, not just prompted.

The Gemini CLI, Cursor, and GitHub Copilot adapters (installed via npx 16-eyes install --target <tool>) restate the same methodology using each tool's own native subagent primitives, but none of those tools has an equivalent schema-enforcement mechanism as of this writing. Their adapters ask each delegated agent to return a fenced JSON block and parse it themselves, with the same corruption/refutation guards written as explicit prose instructions instead of enforced code. Same methodology, slightly weaker guarantees. GitHub Copilot's async coding agent specifically has no slash-command equivalent at all — it relies on AGENTS.md for baseline awareness instead of a dedicated adapter.

Data handling, in short

No telemetry, no backend, no data collected by the maintainer. Code/diff content is sent only to the AI provider you've configured (Anthropic, via your own API key), and in CI, to the GitHub API to read a PR's diff and post the results as a comment, using GitHub's own token for the workflow (or one you supply). Full breakdown in docs/privacy-policy.md.

Reporting a security issue in 16 Eyes itself

See SECURITY.md in the repo. A vulnerability found by an audit belongs in that audit's own report, not there — that file is only for issues in 16 Eyes' own code/logic.