Enhance PR Review Agent pipeline

[don-petry]

Top tier code review agents have a robust and proven approach to finding bugs in code. Research, analyze, review and enhance the pr review agent to be a world class big hunter.

Learning sources:
https://medium.com/data-science-collective/how-coderabbit-actually-works-331aeab55ec8

https://www.greptile.com/what-is-ai-code-review

https://baz.co/resources/engineering-intuition-at-scale-the-architecture-of-agentic-code-review

https://www.coderabbit.ai/blog/how-coderabbits-agentic-code-validation-helps-with-code-reviews

https://www.qt.io/blog/introducing-the-qt-code-review-skills-for-agentic-development

[github-actions[bot]]

Enhancement: Enhance PR Review Agent Pipeline (#906)

Sharpened problem & goal

The request is to systematically benchmark this repo's PR review agent against the architectures used by top-tier commercial reviewers (CodeRabbit, Greptile, baz.co, Qt) and close the most impactful gaps — specifically around bug detection depth, not just style/linting signal.

One clarifying question worth answering upfront: is the primary goal (a) better bug-finding per review (prompt/tier quality), (b) broader context retrieval (cross-file semantic understanding), or (c) iterative validation (running the code, not just reading it)? The answer determines where to invest first.

Context

The existing cascade already implements several patterns described in the linked sources:

• Multi-tier depth: engine.sh runs triage → deep → audit → action tiers, each with escalating model/tool permissions.
• Cross-engine adversarial: the rubber-duck pass (run_duck, prompts/rubber-duck.md) routes a second opinion through a different provider (Copilot/o4-mini) — matching the "multi-model ensemble" pattern CodeRabbit uses.
• MCP opt-in: repos can mount GitHub MCP tools for richer context (issue [Phase 1] Surface the MCP knob to the review workflow without breaking template sync #679 / .github/review-mcp.json).
• LSP pilot: an opt-in setup-lsp-pilot.sh step adds symbol-resolution context to the deep/audit tiers (epic Initiative: Pilot LSP (language-server) code intelligence in the agent harness — pr-review Shell first, measured go/no-go #839).
• Downstream-impact pass: scripts/lib/downstream-impact.sh annotates reviews with consumer-repo blast radius.

What the linked sources highlight that the current agent likely doesn't do:

1. Semantic code graph retrieval (Greptile's core differentiator): fetch callers, callees, and type definitions for touched symbols before writing the review prompt — so the agent sees why a function exists, not just its diff hunk.
2. Issue-type routing: classifying the diff into a type (security, logic, perf, style) and dispatching a specialist prompt per type, rather than a single deep-review prompt for everything.
3. Iterative / agentic validation: after identifying a suspected bug, running the test suite or a lint tool to confirm the hypothesis before reporting it — reducing false-positive rate.
4. Learning from closed PRs: few-shot examples drawn from past review→merge histories for this repo, injected into the deep-tier prompt.

Impact / Effort

Approach	Impact	Effort	Why
Semantic symbol context (LSP or GitHub MCP get_file_contents across callers)	HIGH	M	Most gap vs CodeRabbit; LSP pilot scaffold already exists
Issue-type classifier + specialist prompts	HIGH	M	Splits one monolithic prompts/deep-review.md into typed variants; reuses cascade infra
Agentic iterative validation (run lint/tests in the action tier)	HIGH	L	Needs sandbox or safe-exec guard; high signal gain on logic bugs
Few-shot examples from merged PRs	MEDIUM	S	Prompt-only; can be done as a context-injection script

Suggested acceptance criteria

• A written audit comparing prompts/deep-review.md and engine.sh tier structure against at least 2 of the linked architectures, with a prioritised gap list posted as a follow-up comment or issue.
• The deep-review tier receives at minimum 3 caller/callee symbol contexts for any function touched in the diff (via LSP MCP or GitHub MCP search_code), measurable as a new log line in the review output.
• An issue-type classifier runs before the deep tier and gates which specialist prompt file is loaded; at minimum: security, logic, performance, style.
• False-positive rate (reviewer flags a non-bug) is tracked via the existing token-report infrastructure and shows no regression versus the baseline.
• A dry_run pass with the enhanced prompts on at least 5 real merged PRs (from this repo) is reviewed by the author before the feature is enabled in LIVE_MODE.

[github-actions[bot]]

📋 Initiative planned by the BMAD Scrum Master (Bob).

Epic #1088 — Initiative: Enhance the PR-review pipeline into a world-class bug hunter — semantic symbol context, issue-type specialist routing, agentic validation, and merged-PR few-shot, eval-gated and dry-run-verified

6 stories created (inert — labelled initiative, NOT initiative:auto):

• [Phase 1] Benchmark audit + prioritized gap list + frozen deep-review baseline #1089 (M) — [Phase 1] Benchmark audit + prioritized gap list + frozen deep-review baseline
• [Phase 2] Feed semantic symbol context (callers/callees/type-defs) into the deep + duck tiers #1090 (M) — [Phase 2] Feed semantic symbol context (callers/callees/type-defs) into the deep + duck tiers
• [Phase 2] Issue-type classifier + specialist deep-review prompts via the rubric registry #1091 (L) — [Phase 2] Issue-type classifier + specialist deep-review prompts via the rubric registry
• [Phase 3] Agentic iterative validation: confirm suspected bugs by running lint/tests before reporting #1092 (M) — [Phase 3] Agentic iterative validation: confirm suspected bugs by running lint/tests before reporting
• [Phase 2] Inject merged-PR few-shot examples into the deep tier (dev-split only) #1093 (S) — [Phase 2] Inject merged-PR few-shot examples into the deep tier (dev-split only)
• [Phase 4] Held-out eval regression gate + 5-PR dry-run + human go/no-go before LIVE_MODE #1094 (M) — [Phase 4] Held-out eval regression gate + 5-PR dry-run + human go/no-go before LIVE_MODE

Open questions for review:

• Primary-investment ordering (from the idea-enhancer's clarifying question): is the top priority (a) better bug-finding per review, (b) broader semantic context retrieval, or (c) iterative validation? The plan sequences all four gaps behind the Phase-1 audit, but the audit should confirm which to land first if capacity is constrained.
• Semantic-context mechanism for story 2: build on the LSP pilot (Initiative: Pilot LSP (language-server) code intelligence in the agent harness — pr-review Shell first, measured go/no-go #839) navigation tools, or use GitHub MCP search_code (MCP-powered review enrichment for the self-hosted Claude review engine (engine.sh) #676) — pending Initiative: Pilot LSP (language-server) code intelligence in the agent harness — pr-review Shell first, measured go/no-go #839's go/no-go outcome. The plan supports either via the existing MCP wiring; the Phase-1 audit picks the mechanism.

---

Review the epic and its sub-issue DAG, adjust as needed, then add initiative:auto to epic #1088 to hand it to initiative-driver for auto-implementation.