v0.1.0 — cap-evolve initial release

cap-evolve v0.1.0 — initial release

cap-evolve improves an AI agent's prompts, tools, and skills by learning from failed evaluation traces.

You bring the agent and the eval you already have. cap-evolve runs the loop — evaluate → diagnose failures → propose an edit → keep it only if it beats a held-out split by a significant margin → commit — and reports one honest number. It optimizes what your agent reads, not its weights.

---

What is cap-evolve?

cap-evolve is a skills-native, host-agnostic harness for honestly optimizing AI agent capabilities. It is designed around three principles:

• Optimize more than prompts. System prompts, executable tool code, MCP tool surfaces, and whole Agent Skill packages — pick one or several and optimize them jointly.
• Learn from real agent failures. Every iteration reads full trajectories and per-task causal feedback (which task ids a prior edit broke and fixed), so edits are large, targeted, and don't regress the wins.
• Keep evaluation honest — in code, not docs. Acceptance is a val-only paired significance gate (Δ > k·SE); the test split is sealed and scored exactly once. Both invariants live in the core, not in editable skill markdown.

It is built on a pure Python stdlib core (cap-evolve-core, zero runtime dependencies) wrapped by a library of 19 Agent Skills that implement the full pipeline. Any coding agent CLI — Claude Code, Codex, Gemini, Cursor, Copilot, and more — can serve as the optimizer via a one-row registry entry.

---

How it works

The pipeline is linear:

intake → implement-and-check → baseline → algorithm → finalize → report

Phase	Skill	What happens
intake	intake	Interviews you (or reads your brief), installs the benchmark, wires the adapter, and authors a capability-scoped optimizer prompt — all before any budget is spent.
implement-and-check	implement-and-check	Hard gate. cap-evolve check refuses to proceed until every required adapter method is real and score() is deterministic. No spend happens against a stub.
baseline	baseline	Freezes the seeded train/val/test split (written once; test sealed) and scores the unmodified seed on val — the candidate every iteration must beat. Confirms there is headroom to optimize.
algorithm	hill-climb / gepa / skillopt / agent-optimize	Each iteration: diagnose failing val traces into failure clusters → propose a large, multi-part edit → evaluate on val → paired significance gate accepts or rejects → commit to git and update memory.
finalize	finalize	Scores the best candidate on the sealed test split exactly once. The run dir enforces the seal — a second finalize raises an error.
report	report	Writes report.md and a self-contained dashboard.html (KPI strip, stair chart, heatmap, lineage tree, diffs).

What the optimizer sees each iteration

The harness assembles a capability-scoped working directory per iteration:

• The selected capability skill(s) — as ./guidance/<cap>/ and natively in the agent's own skills dir so a headless agent auto-loads them. Each carries a "What you can change here" menu and edit boundaries.
• The diagnose method — how to cluster failures into a reflective dataset (per failing task: Inputs, Generated Outputs, Feedback).
• Only the current best step's full trajectories — never the seed + every rejected attempt.
• Supporting sources / data model — the capability_sources files copied verbatim so new tool code is written against the real types.
• Per-task causal impact of prior candidates — which task ids each prior edit broke and fixed, plus the currently-passing set to protect. Real causal feedback so a known regression is never re-introduced.

Cross-iteration memory

File	Owner	Purpose
LEDGER.md	framework	Facts: every iteration's outcome + exact tasks it broke/fixed
JOURNAL.md	optimizer	Append-only handover across the run (tried / worked / regressed / refuted / focus-next)
PROCESS.md	optimizer	Explainability, snapshotted per candidate
RUNMAP.md + prior_iterations/	framework	Manifest + every prior iteration's PROCESS.md and capability diff

Because it sees all failure clusters, the protect-set, and the prior causal impact at once, the optimizer produces one bold, multi-part candidate per iteration that addresses every cluster without regressing the wins — not a one-line tweak.

---

What cap-evolve can optimize

Capability	What the optimizer may change
System prompts	Rewrite / consolidate / add rules, examples, output contracts — never drop a needed rule
Tool implementations	Edit tool code for deterministic enforcement; add / wrap / swap tools (never bare-remove)
MCP tool surfaces	Safe edits only — tool docs, in-description examples, which tools are exposed
Skill packages	An Agent Skill dir — SKILL.md bodies, references, and executable scripts

Combine them, e.g. capabilities: [system-prompt, tools].

---

Optimization algorithms

hill-climb — simple global climber (recommended for first runs)

• Parent: always current best
• Focus modes: --focus all (whole train set), --focus cyclic (one task at a time), --focus hardest-first (lowest-scoring tasks first)
• Gate: val paired significance (Δ > k·SE)
• Best for: broad capability gaps; first baseline runs; feedback-poor tasks

gepa — Pareto frontier, trace-based reflection (flagship for expensive rollouts)

Independent implementation of the GEPA algorithm (arXiv:2507.19457).

• Parent: sampled from a per-instance Pareto frontier (frequency-weighted)
• Economy: cheap minibatch local gate first; expensive full-val gate on pass
• Reflection: per-task trace-based dataset fed back into the optimizer prompt
• Merge: system-aware merge across complementary lineages
• Budget: --max-metric-calls (primary) or --max-iterations
• Best for: expensive rollouts; rich feedback; max quality per evaluation dollar

skillopt — annealed epochs × minibatches (flagship for disciplined runs)

Independent implementation of the SkillOpt algorithm (arXiv:2605.23904).

• Parent: always current best (single lineage)
• Learning rate: textual edit budget (integer) decaying on constant | linear | cosine schedule
• Buffer: per-epoch rejected edits + failure patterns injected into the next optimizer prompt
• Meta-update: gated epoch-boundary slow/meta update that fixes longitudinal regressions
• Gate: val paired significance
• Best for: disciplined annealing; per-epoch consolidation

agent-optimize — free-form agentic loop

The agent drives the loop itself (orchestration_mode: agent). The agent proposes candidates, runs the baseline, gates every candidate on the full val split, and seals the test once. Honesty invariants (accept/reject always on full val; test sealed; no overfitting) are enforced in core-owned scripts, not editable markdown.

---

19 Agent Skills

Component	Skill	Purpose
Orchestration	orchestrate	End-to-end pipeline driver — sequences all phases and algorithms
	using-cap-evolve	Session-start router — "optimize X" → intake
Phases	intake	Interview + scaffold .capevolve/project/
	implement-and-check	Hard gate — cap-evolve check must be green
	baseline	Freeze split (seeded, once); score seed on val
	evaluate	Multi-trial scoring with variance (mean + SE + pass^k)
	diagnose	Cluster failures into learning signal
	gate	Paired significance test (Δ > k·SE, val-only)
	finalize	Seal test; score best candidate exactly once
	report	Generate dashboard.html + report.md
Capabilities	system-prompt	Prompt editing — rewrite / consolidate / add, never drop
	tools	Tool code editing — add / wrap / swap, never bare-remove
	mcp-tool	MCP tool surface — safe edits only
	skill-package	Whole Agent Skill packages
Algorithms	hill-climb	Simple global climber (3 focus modes)
	gepa	Pareto frontier + trace-based reflection
	skillopt	Annealed epochs × minibatches
	agent-optimize	Free-form agentic loop
Optimizers	run-optimizer	Dispatcher to 14 coding-agent backends via optimizers/registry.yaml

---

14 Optimizer backends

Any coding-agent CLI can serve as the optimizer — resolved by a single row in optimizers/registry.yaml. Adding a new one is one YAML row.

Backend	Notes
claude-code	Anthropic Claude Code
codex	OpenAI Codex
gemini-cli	Google Gemini CLI
opencode	Meta opencode
openclaw	OpenClaw
ibm-bob	IBM Bob
generic	Any OpenAI-compatible CLI
mock	Deterministic, zero-API ✅ — used in CI and toy_calc
cursor	Cursor cursor-agent
droid	Factory Droid
copilot	GitHub Copilot CLI
kimi	Moonshot Kimi
pi	earendil-works Pi
antigravity	Google agy (configurable wrapper)

---

Benchmark results

Numbers are cross-checked against committed run artifacts under examples/*/run_full/. Each result is labeled by split discipline:

• fit metric — train == val == test (no holdout). Shows the loop works; not a generalization claim.
• held-out — test ids the optimizer never saw, scored exactly once at finalize.

Benchmark	Split discipline	Baseline → Optimized	Gain
toy_calc (zero-API, mock optimizer)	sealed test	0.0 → 1.0	deterministic proof of loop
τ²-bench airline — policy + tools, 50 tasks × 10 trials	val — fit metric	0.536 → 0.712	+0.176 / +32.8% relative
τ²-bench airline — held-out 30/10/10 split	sealed *test...