Enhance PR Review Agent pipeline

[don-petry]

Top tier code review agents have a robust and proven approach to finding bugs in code. Research, analyze, review and enhance the pr review agent to be a world class big hunter.

Learning sources:
https://medium.com/data-science-collective/how-coderabbit-actually-works-331aeab55ec8

https://www.greptile.com/what-is-ai-code-review

https://baz.co/resources/engineering-intuition-at-scale-the-architecture-of-agentic-code-review

https://www.coderabbit.ai/blog/how-coderabbits-agentic-code-validation-helps-with-code-reviews

https://www.qt.io/blog/introducing-the-qt-code-review-skills-for-agentic-development

[github-actions[bot]]

Enhancement: Enhance PR Review Agent Pipeline (#906)

Sharpened problem & goal

The request is to systematically benchmark this repo's PR review agent against the architectures used by top-tier commercial reviewers (CodeRabbit, Greptile, baz.co, Qt) and close the most impactful gaps — specifically around bug detection depth, not just style/linting signal.

One clarifying question worth answering upfront: is the primary goal (a) better bug-finding per review (prompt/tier quality), (b) broader context retrieval (cross-file semantic understanding), or (c) iterative validation (running the code, not just reading it)? The answer determines where to invest first.

Context

The existing cascade already implements several patterns described in the linked sources:

• Multi-tier depth: engine.sh runs triage → deep → audit → action tiers, each with escalating model/tool permissions.
• Cross-engine adversarial: the rubber-duck pass (run_duck, prompts/rubber-duck.md) routes a second opinion through a different provider (Copilot/o4-mini) — matching the "multi-model ensemble" pattern CodeRabbit uses.
• MCP opt-in: repos can mount GitHub MCP tools for richer context (issue [Phase 1] Surface the MCP knob to the review workflow without breaking template sync #679 / .github/review-mcp.json).
• LSP pilot: an opt-in setup-lsp-pilot.sh step adds symbol-resolution context to the deep/audit tiers (epic Initiative: Pilot LSP (language-server) code intelligence in the agent harness — pr-review Shell first, measured go/no-go #839).
• Downstream-impact pass: scripts/lib/downstream-impact.sh annotates reviews with consumer-repo blast radius.

What the linked sources highlight that the current agent likely doesn't do:

1. Semantic code graph retrieval (Greptile's core differentiator): fetch callers, callees, and type definitions for touched symbols before writing the review prompt — so the agent sees why a function exists, not just its diff hunk.
2. Issue-type routing: classifying the diff into a type (security, logic, perf, style) and dispatching a specialist prompt per type, rather than a single deep-review prompt for everything.
3. Iterative / agentic validation: after identifying a suspected bug, running the test suite or a lint tool to confirm the hypothesis before reporting it — reducing false-positive rate.
4. Learning from closed PRs: few-shot examples drawn from past review→merge histories for this repo, injected into the deep-tier prompt.

Impact / Effort

Approach	Impact	Effort	Why
Semantic symbol context (LSP or GitHub MCP get_file_contents across callers)	HIGH	M	Most gap vs CodeRabbit; LSP pilot scaffold already exists
Issue-type classifier + specialist prompts	HIGH	M	Splits one monolithic prompts/deep-review.md into typed variants; reuses cascade infra
Agentic iterative validation (run lint/tests in the action tier)	HIGH	L	Needs sandbox or safe-exec guard; high signal gain on logic bugs
Few-shot examples from merged PRs	MEDIUM	S	Prompt-only; can be done as a context-injection script

Suggested acceptance criteria

• A written audit comparing prompts/deep-review.md and engine.sh tier structure against at least 2 of the linked architectures, with a prioritised gap list posted as a follow-up comment or issue.
• The deep-review tier receives at minimum 3 caller/callee symbol contexts for any function touched in the diff (via LSP MCP or GitHub MCP search_code), measurable as a new log line in the review output.
• An issue-type classifier runs before the deep tier and gates which specialist prompt file is loaded; at minimum: security, logic, performance, style.
• False-positive rate (reviewer flags a non-bug) is tracked via the existing token-report infrastructure and shows no regression versus the baseline.
• A dry_run pass with the enhanced prompts on at least 5 real merged PRs (from this repo) is reviewed by the author before the feature is enabled in LIVE_MODE.