Skip to content

Prism Review v1.0.0 - Production Release & Empirical Benchmark Verification

Latest
Latest

Choose a tag to compare

View all tags
@binodbishwakarama-max binodbishwakarama-max released this 18 Oct 12:00
· 8 commits to main since this release
v1.0.0
632687b

Prism Review v1.0.0

Prism Review is an autonomous, standards-driven AI code review platform verified by an empirical evaluation benchmark on frozen open-source repositories.

⚡ Headline Benchmark Results (Frozen Test Split: 120 PRs across trpc/trpc and vitejs/vite)

  • Review F1 Score: 0.9304 (capturing 96.8% of Claude 3.5 Sonnet's 0.9613 F1, 95% CI: [0.8926, 0.9592]).
  • Serving Cost per Review: $0.00036 (37.1x cheaper than Claude 3.5 Sonnet at $0.01334/PR).
  • Clean-PR False Positive Rate: 0.2353 (68.0% noise reduction compared to Base Qwen-7B at 0.7353).
  • Serving Latency: 40.5 ms p50 (vLLM 4-bit AWQ on NVIDIA A10G).
  • Webhook Ingestion: 11.8 ms p95 via FastAPI, Redis, and ARQ async worker pool.

🏗️ Architecture & Features

  • Distributed Ingestion & Queue: FastAPI webhook gateway with constant-time HMAC-SHA256 signature verification, idempotency deduplication, and ARQ async worker pool.
  • Enterprise Safety & Hardening: Automated secret redactor (scrubs API keys/tokens before LLM ingestion), untrusted XML encapsulation against prompt injection, daily cost budget cap, and per-installation kill switches.
  • Repository Standards as Code: Evaluates PR diffs against .prism/standards.yml with hybrid BM25 + dense embedding retrieval.
  • Self-Verification Filter: Aggressively eliminates hallucinations and speculative nitpicks before posting inline comments.
  • Multi-Matcher Evaluation Benchmark: Empirical harness featuring +/-3 line window tolerance, semantic text embedding similarity (>= 0.82), and B=1,000 paired bootstrap confidence intervals.
  • 4-Bit AWQ Quantization & Serving: Model footprint compressed from 16.2 GB to 5.8 GB VRAM with zero loss in review quality (0.9304 vs 0.9297).
  • Telemetry & Closed-Loop Feedback: Live acceptance tracking via developer reactions (thumbs up/down) streaming to Prometheus (/metrics) and Grafana.

📦 Quickstart

git clone https://github.com/binodbishwakarama-max/codeagent.git
cd codeagent
uv sync --extra dev

# Run unit and smoke tests (313 passed)
make test

# Simulate local GitHub App webhook & review pipeline
make simulate-webhook

# Reproduce benchmark results
make reproduce
Assets 2
Loading