Skip to content

v2.0.1 — accuracy-per-dollar

Choose a tag to compare

@moazmo moazmo released this 11 Jun 18:45
· 36 commits to main since this release

PR Sentinel v2: review accuracy as a systems problem, not a model-size problem. The same cheap model, wrapped in a system that samples, anchors, and verifies, beats its own naive single pass — measured, with published false-positive rates.

The leaderboard (same deepseek-v4-flash, 51 fixture-runs)

Config Caught Clean-fixture false positives Cost/review
Naive single pass 47/51 (92%) 2 ~$0.002
v2 (ensemble + verifier) 49/51 (96%) 0 ~$0.005

What is new in v2

  • Line-numbered diffs + evidence anchoring — every finding must quote a line that literally exists in the diff, or it is dropped before posting. Hallucinations become structurally impossible.
  • Self-consistency ensemble — each analyst reviews 3× and findings are majority-voted (DeepSeek caching keeps this ~1.3x cost, not 3x).
  • Verifier pass — a dedicated agent adjudicates every finding against the code before the reviewer writes prose.
  • Inline review comments — findings post anchored to diff lines; the rest stay in one sticky summary.
  • On-demand commands — @pr-sentinel review | ask | describe, gated to repo owners/members/collaborators.
  • Native Anthropic provider, two-tier model routing, JSON-mode with fallback, pooled connections, cache-aware cost reporting.
  • 17-fixture / 5-language eval suite (Python/JS/TS/Go/Java) with hard negatives and two prompt-injection vectors.

163 tests, all LLM/GitHub mocked. Live-verified end-to-end: inline comments, cross-agent attribution, and the ask command answering a grounded question about a real PR.

Install: uses: moazmo/pr-sentinel@v2. See the README for the workflow and config.