plugin-forge v0.1.0 — the eval-first plugin builder

plugin-forge v0.1.0 — the eval-first plugin builder

The eval suite is the contract; green is the definition of done.

plugin-forge builds Claude Code plugins the way you'd want production software built: nothing is scaffolded until a design contract exists, nothing counts as done until a frozen evaluation suite passes, and the builder is mechanically prevented from grading its own homework.

The pipeline

interview → contract → evals → armed → building → smoke → verify → ship

Phase	What happens	Gate
Interview	Runtime environment first (local / cloud / CI), external systems, side-effect inventory, success criteria	design/INTENT.md complete
Contract	A Primitive Decision Record: per capability, a six-step consequence chain (trigger → context economics → enforcement → failure modes → testability → distribution) plus a 14-slot ledger over every plugin primitive	Adversarial spec-critic agent writes APPROVED
Evals	The eval builder generates 20–50 tasks with reference solutions, deterministic-first graders, LLM judges with mandatory Unknown, positive and negative trigger cases	Suite complete, antipattern lints clean
Armed	Four arming gates: reference solutions pass their own graders; RED baseline (bare Claude must fail); judge calibration ≥ 0.85 vs human labels; sha256 freeze	.forge/freeze.json written
Building	Headless /goal loop in an isolated git worktree. PreToolUse + Stop guard hooks make the frozen suite un-editable — even under --dangerously-skip-permissions	Authoritative full-suite re-run prints RESULT=PASS
Smoke → Verify	Init-gate probe, per-skill probes, hook fixture pipes, trigger sanity; then fresh pass^k trials, a holdout tranche the builder never saw, trigger precision/recall under crowded-listing pressure, triage of failures	All green
Ship	claude plugin validate --strict, freshness lints, README with permission story, versioning decision, distribution menu, and a trace-capture flywheel hook installed into the generated plugin	You release

What "unverifiable → unbuildable" means

The contract's testability column is a hard gate: every capability must name the grader type and task IDs that will prove it. The eval builder derives its task list from that column — skip the lookahead thinking and there is literally nothing to generate.

What ships in the box

• 9 skills — forge (conductor), write-contract, build-evals, arm-evals, build-loop, verify, ship, plus two knowledge decks: a 12-card primitive catalog (each card ends with Consequence prompts / Reject when / Eval hooks) and generation standards with freshness-guard lints
• 2 agents — spec-critic (read-only, three-lens adversarial gate) and triage (classifies failures as agent-error / grader-error / task-ambiguity)
• 3 guard hooks — Edit/Write + Bash freeze guards and a Stop-hook sha256 sweep, all <10 ms no-ops outside a build
• A stdlib-only eval harness (bin/forge-eval → scripts/harness/run.py) — vendored into every generated plugin, so suites outlive the builder. Per-trial worktree isolation, system/init gating, timeout/api-retry fairness, --max-cost-usd hard abort, pass@k and pass^k, HTML reports
• 5 grader CLIs + smoke runner, an eval-run dynamic workflow, full eval/scaffold templates, and a self-verifying dogfood suite (8/8 green, zero tokens, CI-ready)

Validation on this release

• claude plugin validate --strict — clean
• Dogfood regression suite — 8/8 PASS (guard hooks, sweep, compaction re-injection, plus meta-evals proving the harness refuses tolerance-less numeric graders and emits the exact scoreboard grammar)
• Three-lens adversarial review — 34 evidenced findings, all resolved
• Live headless smoke — plugin loads with zero plugin_errors; all commands and agents registered

Install

/plugin marketplace add Aznatkoiny/plugin-forge
/plugin install plugin-forge@plugin-forge

Then, in the project where your new plugin should live:

/plugin-forge:forge "your plugin idea"

Honest limitations

Eval-first is the most expensive way to build a plugin (trials × tasks × judges × build iterations) — --max-cost-usd and --dry-run are load-bearing. Judge calibration needs 10–20 hand-labeled transcripts per judge, and arm-evals refuses to freeze uncalibrated judges. Overfitting to visible graders is mitigated (holdout tranche, reference solutions staged outside the workspace, triage sampling), not solved: the honest claim is "green + holdout + triage", never "correct".

Grounded in Anthropic's agent-eval methodology, LangChain's eval-engineering automation, and OpenAI's evals registry patterns.