Trial Qodo Merge / PR-Agent (OSS) as a self-hosted Claude-powered PR reviewer

[don-petry]

Idea: trial Qodo Merge / PR-Agent (open source) as a self-hosted PR reviewer on our Claude key

Surfaced while evaluating Cursor Bugbot (dead end — not available on Free plan). The genuinely-new option worth considering is PR-Agent, the open-source engine behind Qodo Merge. It's a different harness from anything we currently run, so it's the only candidate here that isn't already in our advisory-bot stack.

Why it's different from what we already run

Not a monolithic "one bot comments" reviewer. PR-Agent is a command/tool-based pipeline — discrete agent tools invoked per-PR:

• /review — bug / security / quality findings (the Bugbot-equivalent)
• /improve — concrete code-change suggestions
• /describe — auto-writes the PR description
• /ask <question> — Q&A against the diff
• plus /add_docs, /update_changelog, /test

Layered architecture (UI adapters → orchestration → tools → platform abstraction for GitHub/GitLab/Bitbucket/Azure). Runs as a GitHub Action, webhook server, or CLI (enables local /review pre-push). Key trick: a PR-compression strategy that ranks/trims large diffs to fit context — handles big PRs that naive tools choke on.

Models — fully model-agnostic via LiteLLM

Routes through a LiteLLMAIHandler: OpenAI, Anthropic/Claude, Gemini, DeepSeek, Azure, Bedrock, Vertex, local models. Set config.model + drop in the key.

• Self-hosted (free path): point it at our Claude key — claude-opus-4-8 / claude-sonnet-5.
• For reference, hosted Qodo Merge default (Dec 2025) is a blend: GPT-5.2 + Haiku 4.5 + Gemini 2.5 Pro. Qodo 2.0 (Feb 2026) moved the hosted product to a multi-agent architecture (parallel bug/security/quality/test agents) — conceptually similar to our dev-lead fan-out. The OSS engine we'd self-host is the single-pipeline design.

Cost

Free software (OSS, self-host). Real cost = Anthropic API tokens per review — same cost model as our existing dev-lead pr-review-agent. No SaaS/per-seat fee, no dependency on a paid Cursor/Qodo plan.

Proposed trial

Stand up PR-Agent as a GitHub Action on Claude in one repo and compare its /review output head-to-head against dev-lead pr-review-agent. The likely delta worth measuring: polished /improve and /describe tools out of the box vs. maintaining our own.

Open question

Is there enough delta between PR-Agent's /review + /improve and our existing dev-lead reviewer to justify maintaining another bot in the fleet — or is this a source of ideas (the /improve, /describe, PR-compression patterns) to fold into dev-lead rather than a bot to adopt?

[github-actions[bot]]

Enhancement: Market & Feasibility Deep-Dive

Urgent: Gemini Code Assist Consumer Sunset (July 17, 2026)

The free (consumer) version of Gemini Code Assist on GitHub shuts down July 17, 2026 — effectively now. New installations were already blocked since June 18, 2026. Only the enterprise version continues, at $19/seat/month (Standard) or $45/seat/month (Enterprise).

This invalidates option 2 from the proposed direction ("Verify Gemini Code Assist is Developer-Connect-linked across all org repos"). Even if it were linked, the free tier ceases to function. Additionally, Gemini Code Assist was never installed as a GitHub App in petry-projects — the github-settings.md installed-apps table does not list it. The dev-lead engine gemini option is a different thing: it routes the existing dev-lead workflow through Google's Gemini API models, not the Gemini Code Assist GitHub App.

Net effect: Gemini Code Assist is no longer a viable free option. Remove it from the free-stack ledger.

Sources: Google deprecation notice, Google Cloud pricing page

---

Prior Art: PR-Agent Community Fork Has Matured

Since this Discussion was written, PR-Agent has undergone significant changes:

Event	Date	Impact
Qodo hands PR-Agent to community governance	April 2026	Now at the-pr-agent/pr-agent, Apache 2.0 restored
v0.39 ships AGENTS.md auto-inclusion	July 2025	Reads AGENTS.md by default in /review, /improve, /describe — direct compatibility with petry-projects standards
Claude Sonnet 5 & Opus 4.8 model support	Recent	Can run on the same Anthropic API key the dev-lead already uses
GitHub Checks API integration	v0.39	Review results surface as GitHub Checks, not just PR comments

The community fork is actively maintained (v0.39.0 released July 5, 2025; LiteLLM bumped to v1.84.0). Key differentiator vs. the dev-lead: PR-Agent's /improve and /describe commands produce structured, formatted output (inline suggestions, auto-generated PR descriptions) that complement rather than duplicate the dev-lead's review-and-fix workflow.

Sources: PR-Agent releases, Community governance announcement, PR-Agent docs

---

Updated Options Table

Option	Actual Cost	Unique Value-Add Over Dev-Lead	Risk	Recommendation
A. Close the loop — current stack is complete (dev-lead + CodeRabbit + SonarCloud)	$0 incremental	None needed — already covers review, quality, security	Missed delta from /improve-style suggestions	✅ Default if stack is satisfactory
B. Trial PR-Agent on Claude in one repo (the-pr-agent/pr-agent GitHub Action)	Anthropic API tokens only (~$0.02–0.15/PR)	/improve inline suggestions, /describe auto-descriptions, AGENTS.md-aware context, /add_docs	Multi-bot noise; maintaining a second review config; community fork longevity	⚠️ Worth a 2-week trial if /improve delta is the open question
C. Gemini Code Assist Enterprise	$19–45/seat/month	Google-native review with Gemini 2.5	Paid; requires Google Cloud Developer Connect setup; cost scales with org	❌ Not justified — dev-lead already supports Gemini models via vars.DEV_LEAD_ENGINE=gemini at zero marginal cost
D. Cursor Bugbot (usage-based)	$1.00–1.50/PR or $40/user/month	Cursor-ecosystem integration	Paid; vendor lock-in to Cursor ecosystem	❌ Already ruled out — confirmed not free

Sources: CodeRabbit pricing (free tier: 200 private reviews/month), Bugbot pricing, AI code review comparison

---

Risk: Multi-Bot Comment Fatigue

Industry experience shows that running 2+ AI review bots on the same PR creates "comment fatigue" — developers mute bots entirely, and legitimate findings get buried alongside false positives. One team's retrospective: "We ran 11 AI PR bots and kept 1" (Indie Hackers). The current stack (dev-lead + CodeRabbit + SonarCloud) already has three reviewers. Adding PR-Agent would be a fourth.

Mitigation if trialing Option B: Configure PR-Agent with auto_review: false and auto_improve: false so it only responds to explicit /review or /improve commands, avoiding unsolicited comment noise. This makes it opt-in per-PR rather than always-on.

---

Answering the Open Question

"Is there enough delta between PR-Agent's /review and our existing dev-lead reviewer to justify maintaining another bot?"

Probably not for /review — the dev-lead already does Claude-powered review with AGENTS.md context, CodeRabbit catches complementary issues, and SonarCloud handles static analysis. The review layer is well-covered.

The only testable delta is /improve — PR-Agent's inline code-improvement suggestions are a distinct capability the dev-lead doesn't replicate. A time-boxed trial (Option B, one repo, 2 weeks, command-only mode) would answer this empirically at near-zero cost.

---

Suggested Next Steps

1. Immediate: Remove Gemini Code Assist from the free-stack ledger — consumer version sunsets July 17, 2026
2. If trialing PR-Agent: Add the-pr-agent/pr-agent@v0.39 as a GitHub Action in one repo (e.g., .github itself), configured with ANTHROPIC.KEY, config.model: "anthropic/claude-sonnet-4-20250514", and auto_review: false / auto_improve: false (command-only mode). Run /improve on 5–10 PRs and compare output quality against dev-lead
3. If not trialing: Close this Discussion — the free stack (dev-lead + CodeRabbit + SonarCloud) is complete, Bugbot is confirmed paid, and Gemini Code Assist free is dead

Success Metrics for Trial (if pursued)

• Actionable suggestion rate: % of /improve suggestions that lead to accepted code changes (target: >40%)
• Unique finding rate: % of findings not already surfaced by dev-lead or CodeRabbit (target: >25%)
• Developer sentiment: Does the team find /improve output valuable or noisy? (qualitative, 2-week retro)