RFC: Review infrastructure — industry research, gap analysis, and architectural decisions

[sonupreetam]

Context

This discussion captures the research, industry analysis, and architectural decisions behind PR #229 (finding consolidation, compound severity, constitution Principle V) and the broader review infrastructure improvements. It serves as a decision record for current and future work on the Divisor Council, /review-pr, and /review-council.

Related: #228, #233, PR #229, unbound-force/website#142

---

Part 1: Why Issues #228 and #233 Happened

Both issues exposed structural gaps in how AI review agents handle security findings:

Issue #228 — Finding fragmentation: During a supply chain review (complytime-collector-components#229), the Adversary persona split one compound finding (unsigned binary download + privileged CI container) into three separate findings at LOW/MEDIUM. Five root causes:

1. Adversary checklist missing dependency necessity check
2. No compound severity escalation rule in severity.md
3. No finding consolidation pass in /review-pr
4. CI bot annotations (Scorecard, Trivy, GHAS) dismissed as unrelated
5. No cross-persona finding consolidation in /review-council

Issue #233 — Severity anchoring bias: During complytime/org-infra#285 review, three HIGH findings were classified as MEDIUM because the reviewer anchored on a positive first impression. Three root causes:

1. No severity calibration step forcing re-read of definitions
2. No adversarial input enumeration (category checklist, not per-input)
3. No linked-issue suggestion gap detection

Common theme: Security detection infrastructure is comprehensive, but the synthesis layer is missing.

---

Part 2: How Other Platforms Handle This

Finding Consolidation & Deduplication

System	Approach
Microsoft MDASH	100+ agents; debater agents where disagreement raises posterior credibility
RevHive	9 reviewers + 1 coordinator; semantic deduplication (Jaccard similarity) + LLM conflict resolution
Screvyn	4 parallel agents + severity aggregator that deduplicates and ranks P0/P1/P2
AgenticVM	Integration Agent consolidates Detection/Assessment/Prediction agents; reduced 3,983 detections to 155 CVEs
Agent-Field/sec-af	AI-assisted duplicate checking via LLM call per CWE pair
TitanCA	Multi-module pipeline (Singapore/GovTech); 127K+ repos, 203 zero-days, 118 CVEs

Severity Calibration & Anchoring Bias

• Anchoring bias in LLMs (2026): Chain-of-Thought, Principles, Ignoring Anchor Hints, and Reflection are NOT sufficient to mitigate anchoring. Strong models are easy to influence by expert opinions in prompts.
• Mitigating Prompt-Induced Cognitive Biases in SE (April 2026): Making implicit best practices explicit via axiomatic reasoning cues lowers bias sensitivity by ~50% (up to 73%). This validates our approach — forcing the agent to quote severity.md definitions.
• TitanCA: Logistic confidence mapping from model log-odds, reducing false positives 28% to 20%.
• Multi-model consensus (Mozilla, Milvus): A finding from Claude + GPT-5.5 + Gemini independently is orders of magnitude more reliable.
• CodeRabbit: Anchors in deterministic linter baselines (40+ bundled). Bug catch: ~44%.
• Qodo: Grounds in test execution/behavioral analysis. Bug catch: ~60.1% F1.

False Positive Reduction

• Planner-Critic-Rewriter: Dedicated critic pass scores every finding on severity x confidence before output. Precision: ~30% to 90%+.
• Propel: Learns from dismissed findings; suppresses repeats; classifies nit/concern/must-fix.
• Tzpect-CodeReview: SQLite feedback DB tracks accuracy; learns from false positives.
• mountainowl/ai-code-review: --sync-outcomes grades posted findings against SCM state (resolved/disputed/false_positive/duplicate).
• CodeGenie: Calibrated weighted-fusion with learned coefficients; exponential-smoothing personalization loop adapts to developer patterns.

Supply Chain & Security Beyond Code Review

• Agentic AIBOMs: Extending SBOMs into active provenance artifacts with multi-agent architecture (environment reconstruction + drift monitoring + VEX reasoning).
• Cloudsmith 2026 Guide: Industry has moved from static SBOMs to agentic governance — AI agents detect CVEs, reason about exploitability, and remediate.
• OWASP SBOM-VEX-Taint-Analysis: Multi-agent VEX generation with mandatory HITL for high-severity. Cosign signing. Every agent decision in audit log.

---

Part 3: What We Implemented (PR #229)

Change	File(s)	Addresses
Compound severity escalation	severity.md	#228
Finding consolidation (Step 8f)	review-pr.md	#228
CI bot cross-referencing (Step 8e)	review-pr.md	#228
Cross-persona consolidation	review-council.md	#228
Dependency necessity check	divisor-adversary.md	#228
Severity calibration (Step 8g)	review-pr.md	#233
Adversarial input enumeration	divisor-adversary.md, review-pr.md	#233
Issue suggestion gap detection	review-pr.md	#233
Constitution Principle V	constitution.md	Both

---

Part 4: Gap Analysis — What We Have Not Done Yet

Tier 1: Structural gaps in the review system

Gap	Industry standard	Our status
No critic/verification pass	Planner-Critic-Rewriter (90%+ precision)	Missing — findings go straight to output
No feedback learning loop	Tzpect, Propel, mountainowl all track dismissals	Open as #207
No confidence scoring	TitanCA (log-odds), AgenticVM (ML confidence)	Missing
Same model for all personas	MDASH uses different models per role	All personas use same model
No PR-level risk score	RevHive (0-100 composite)	Missing
Discovery includes non-reviewers	N/A	Scribe/Herald/Envoy match divisor-* pattern but are not reviewers
Iteration re-runs all personas	Spec 005: only failing personas	/review-council re-runs all

Tier 2: Unowned review dimensions

Dimension	Status
CLI UX (help text, flag validation)	Only /review-pr path heuristic
Dependency license compliance	Mentioned in focus heuristic; no persona
Accessibility	speckit.checklist.md asks; no persona reviews
Privacy / data retention	Unowned
API backward compatibility	Partially SRE
OpenSpec artifact scope	Council reviews; personas do not read openspec/

Tier 3: Overlap / ambiguity

Overlap	Personas
Documentation completeness	Guard + Curator (both check CHANGELOG/AGENTS/README)
Gate tampering	Guard (governance) + Adversary (security controls)
Testing conventions	Architect [PACK] + Tester
Supply chain	Adversary (CVEs) + SRE (SBOM, signed releases)
Cross-spec consistency	Guard + Adversary + Architect in spec mode

---

Part 5: Architectural Decision — New Persona vs Enhanced Orchestration

Decision: Enhance the orchestration layer, not a new persona.

Every major system (MDASH, RevHive, Screvyn, CodeGenie) puts synthesis in the orchestration layer, not in a specialized review agent. A new persona would create confusion about whether it reviews code or just processes findings.

/review-council already IS the synthesizer — it discovers agents, delegates, consolidates, and enforces verdicts. The synthesis steps are just thin.

Recommended enhancements to /review-council:

1. Critic pass (Step 3.5) — filter low-confidence findings before output
2. Confidence scoring — each persona assigns confidence; orchestrator uses it
3. PR-level risk score — aggregate from findings
4. Conflict resolution — when personas disagree on severity
5. Discovery exclusion — filter Scribe/Herald/Envoy
6. Iteration scope — re-run only failing personas per Spec 005 FR-005

The one case for a new persona: If we want to cover an unowned review dimension like Compliance (license audit, privacy, accessibility) or API Contracts (backward compatibility, schema evolution). That is domain expertise, not synthesis.

---

Part 6: What to Track

Item	Tracking
Critic pass + confidence scoring	File new issue
Feedback learning loop	Existing #207
Discovery exclusion + iteration fix	File new issue
Guard/Curator overlap de-duplication	File new issue
PR-level risk score	File new issue
Unowned dimensions (CLI UX, license, a11y)	File new issue when prioritized
OpenSpec scope in persona agents	File new issue

CC @jflowers @marcusburghardt

[sonupreetam]

Update: Citation audit and new research (June 2026)

Audited all citations from the original post and searched for newer work. Summary below.

---

Citation Status

All 16 URLs remain live. One note:

Citation	Status	Note
Agentic AIBOMs (dl.acm.org)	403 paywall	Use the open arxiv version: arxiv:2603.10057
MDASH (LetsDataScience)	Live	Official source now available: Microsoft Security Blog
All others	Live	No changes needed

Low-traction repos: RevHive (2 stars), Screvyn (2 stars), Tzpect (2 stars), ChuksForge (1 star), mountainowl (0 stars), OWASP SBOM-VEX-Taint-Analysis (0 stars). These are reference implementations, not widely-adopted tools. Agent-Field/sec-af (91 stars) and MDASH (Microsoft production) remain the strongest references.

---

New Research (not in original post)

1. Confirmation Bias in LLM-Assisted Security Code Review

arxiv:2603.18740 (March 2026) — more specific to our problem than the general anchoring paper we originally cited.

• Adversarial PR metadata (commit messages, PR descriptions framed as "security fix") reduces vulnerability detection by 16-93%
• Claude Code in autonomous mode: 88% attack success when adversary can iteratively refine framing
• Debiasing: metadata redaction + explicit instructions restores 94% detection in autonomous agents, 100% in interactive
• Implication for us: Our severity calibration step (8g) partially addresses this, but we should also consider metadata redaction before the adversary persona runs — strip PR descriptions and commit messages, review code diff only

2. Adversarial Comments vs LLM Code Reviewers

arxiv:2602.16741 (Feb 2026) — large-scale empirical study

• Adversarial code comments have statistically non-significant aggregate effect on LLM reviewers (p>0.21 for all 8 models)
• SAST Cross-Referencing is the feat: complete spec 001 org-constitution implementation #1 defense: 96.9% detection, 47% recovery of baseline misses
• Comment stripping hurts weaker models (DeepSeek dropped from 93% to 89%) — stripping removes helpful context too
• Validates our Step 8e (CI Bot Annotation Cross-referencing). The paper explicitly recommends piping SAST output into the review prompt as "verification targets"

3. AgentFlow: Synthesizing Multi-Agent Harnesses

arxiv:2604.20801 (April 2026) — strongest validation of our "orchestration > persona" decision

• "Changing only the harness can still change success rates by several-fold on public agent benchmarks"
• A mid-tier model (Kimi K2.5) with a good harness found 10 zero-days in Chrome including 2 critical sandbox escapes (CVE-2026-5280, CVE-2026-6297)
• Typed graph DSL covering: agent roles, prompts, tools, communication topology, coordination protocol
• Feedback-driven outer loop diagnoses which part of the harness caused failure
• Implication for us: The harness IS the product. Improving /review-council orchestration is the highest-leverage work, not adding models or personas.

4. Co-RedTeam (ICML 2026)

arxiv:2602.02164 (Feb 2026, accepted ICML 2026)

• Orchestrator-based multi-agent red teaming with execution-grounded feedback
• Validation Agent acts as a safety and consistency gate before execution
• Long-term memory — agents learn from prior attack trajectories
• 60%+ exploitation success, 10%+ improvement over baselines
• Implication for us: The Validation Agent pattern maps directly to a critic pass in /review-council. The memory pattern maps to our Feature: Close learning feedback loop #207 feedback loop.

5. Quorum — Multi-Agent Code Reviewer

skmdroid/quorum (2026)

• Panel of specialist agents (security, performance, correctness, tests, style) running in parallel
• Model tiering: stronger model for security/correctness, faster model for style/nits
• Synthesizer de-duplicates, ranks by severity, produces single verdict
• Ships a detection benchmark with planted defects; measures recall on defects and false alarms on clean code separately
• Graceful degradation when an agent fails (single failure never aborts the review)
• Implication for us: Model tiering is something we have not explored. The detection benchmark methodology could be adapted for evaluating Divisor Council changes.

Also notable: nexusentis/quorum — MCP-based multi-model consensus with blind judge scoring and mechanical deduplication; and Solvely-Colin/Quorum — 7-phase deliberation with debate topologies (mesh, star, tournament) and hash-chained audit ledger.

6. RepoReviewer

arxiv:2603.16107 (March 2026) — pragmatic architecture paper

• Hybrid design: deterministic logic for structural tasks, LLM for semantic reasoning
• PriorityAgent uses largely deterministic dedup/sort (not model-driven)
• Implication for us: Validates using deterministic rules (like our severity.md compound escalation) alongside LLM judgment

7. Code Broker

arxiv:2604.23088 (April 2026) — Google ADK-based

• 5-agent hierarchy: root orchestrator -> pipeline agent -> 3 concurrent specialists -> improvement recommender
• Fuses LLM reasoning with deterministic Pylint signals
• Session memory for retaining prior assessment context across runs
• Implication for us: Another example of SAST + LLM fusion. Validates both our CI cross-referencing step and the feedback loop concept.

8. JetBrains: AI Code Enters Review

JetBrains AI Blog (May 2026)

• AI-generated PRs contain nearly twice the code redundancy but draw fewer negative reactions from reviewers
• 7.2% reduction in delivery stability for 5% AI adoption increase — attributed to larger changesets
• 44% of AI code hallucinations involve errors no automated check reliably surfaces
• Implication for us: Reviewers are already less critical of AI-generated code. This amplifies the anchoring/confirmation bias problem and strengthens the case for mandatory severity calibration.

9. Agentic AI Security Threats Survey

arxiv:2510.23883v2 (updated 2026)

• Comprehensive taxonomy of attacks on multi-agent systems
• Protocol-level threats in agent-to-agent (A2A) communication
• Cascading failures, prompt injection across agents, discovery/authentication exploits
• Implication for us: If we ever expose the Divisor Council to external inputs (e.g., user-submitted repos), we need to consider agent-to-agent security. Currently low risk since all inputs are trusted, but worth tracking.

---

Updated Recommendations

Based on the new research, the following additions to the "What to Track" table are warranted:

Item	Rationale	Priority
Metadata redaction for adversary review	arxiv:2603.18740 shows 88% bypass rate via PR metadata framing	High
Detection benchmark for Divisor Council	Quorum's planted-defect methodology; needed to measure changes	Medium
Model tiering per persona	Quorum routes stronger models to security; current system uses same model for all	Medium
SAST verification targets in prompt	arxiv:2602.16741: SAST cross-ref is #1 defense (96.9% detection) — our Step 8e should explicitly inject SAST findings into the adversary prompt, not just cross-reference	Medium
Agent-to-agent security model	arxiv:2510.23883v2: relevant when/if council accepts external inputs	Low

The original decision — enhance orchestration, not add personas — is now even more strongly supported. AgentFlow demonstrated that the harness alone can change success rates by several-fold. Co-RedTeam's Validation Agent pattern is exactly the "critic pass" we identified as the top priority gap.