PRIMER v1.0.0

PRIMER v1.0.0

A measurement harness for AI coding agent context files. PRIMER runs a controlled before/after experiment to answer: does your CLAUDE.md (or AGENTS.md) actually improve your AI coding agent's performance on your repository?

Install

pip install primer-eval

What's in this release

Evaluation CLI

• primer init — analyse a repository and generate a context file (supports Ollama, Anthropic, OpenAI, Gemini, OpenRouter)
• primer eval — run the before/after evaluation inside Docker containers
• primer report — render the latest score (text or --json)
• primer history — list all past evaluations
• primer compare — diff two evaluations side-by-side
• primer export — write scores.json + dashboard JSON tree

Evaluation Engine

• Docker isolation — hermetic containers per evaluation arm; optional egress proxy blocks outbound agent calls during runs
• Two task types derived from git history: revert_reimplement and stub_function
• Deterministic verification — success = pytest passes; no LLM judge
• Measurement model — signed success-rate delta with noise-envelope threshold; 4-verdict taxonomy (Helped ▲ / Hurt ▼ / No measurable effect ≈ / Not comparable ⊘)
• SQLite persistence — all evaluation runs stored and queryable

Agent Support

• Claude Code (claude_code) — default, stable
• Gemini CLI (gemini) — experimental

Dashboard (Next.js 15)

• Static export deployed to GitHub Pages
• 7 routes: overview, evaluation detail, compare, trends, methodology, score guide, export
• Dark/light theme toggle, mobile-responsive layout
• WCAG AA colour tokens

Infrastructure

• GitHub Actions CI/CD — tests + dashboard deploy on push to main
• Shields.io badge served from scores.json
• detect-secrets baseline + pre-commit hooks
• MIT license

Test Suite

• Python: 550 passed, 4 skipped (Docker/live-API integration tests)
• Dashboard: 11/11 TypeScript tests passed

Related Work

ETH Zurich SRI Lab + LogicStar.ai (arXiv:2602.11988, Feb 2026) — "Evaluating AGENTS.md." PRIMER is a per-repo measurement tool building on the questions this paper raised.