Repository navigation
cap-evolve v0.1.0 — initial release
cap-evolve improves an AI agent's prompts, tools, and skills by learning from failed evaluation traces.
You bring the agent and the eval you already have. cap-evolve runs the loop — evaluate → diagnose failures → propose an edit → keep it only if it beats a held-out split by a significant margin → commit — and reports one honest number. It optimizes what your agent reads, not its weights.
What is cap-evolve?
cap-evolve is a skills-native, host-agnostic harness for honestly optimizing AI agent capabilities. It is designed around three principles:
- Optimize more than prompts. System prompts, executable tool code, MCP tool surfaces, and whole Agent Skill packages — pick one or several and optimize them jointly.
- Learn from real agent failures. Every iteration reads full trajectories and per-task causal feedback (which task ids a prior edit broke and fixed), so edits are large, targeted, and don't regress the wins.
- Keep evaluation honest — in code, not docs. Acceptance is a val-only paired significance gate (Δ > k·SE); the test split is sealed and scored exactly once. Both invariants live in the core, not in editable skill markdown.
It is built on a pure Python stdlib core (cap-evolve-core, zero runtime dependencies) wrapped by a library of 19 Agent Skills that implement the full pipeline. Any coding agent CLI — Claude Code, Codex, Gemini, Cursor, Copilot, and more — can serve as the optimizer via a one-row registry entry.
How it works
The pipeline is linear:
intake → implement-and-check → baseline → algorithm → finalize → report
| Phase | Skill | What happens |
|---|---|---|
| intake | intake |
Interviews you (or reads your brief), installs the benchmark, wires the adapter, and authors a capability-scoped optimizer prompt — all before any budget is spent. |
| implement-and-check | implement-and-check |
Hard gate. cap-evolve check refuses to proceed until every required adapter method is real and score() is deterministic. No spend happens against a stub. |
| baseline | baseline |
Freezes the seeded train/val/test split (written once; test sealed) and scores the unmodified seed on val — the candidate every iteration must beat. Confirms there is headroom to optimize. |
| algorithm | hill-climb / gepa / skillopt / agent-optimize |
Each iteration: diagnose failing val traces into failure clusters → propose a large, multi-part edit → evaluate on val → paired significance gate accepts or rejects → commit to git and update memory. |
| finalize | finalize |
Scores the best candidate on the sealed test split exactly once. The run dir enforces the seal — a second finalize raises an error. |
| report | report |
Writes report.md and a self-contained dashboard.html (KPI strip, stair chart, heatmap, lineage tree, diffs). |
What the optimizer sees each iteration
The harness assembles a capability-scoped working directory per iteration:
- The selected capability skill(s) — as
./guidance/<cap>/and natively in the agent's own skills dir so a headless agent auto-loads them. Each carries a "What you can change here" menu and edit boundaries. - The diagnose method — how to cluster failures into a reflective dataset (per failing task: Inputs, Generated Outputs, Feedback).
- Only the current best step's full trajectories — never the seed + every rejected attempt.
- Supporting sources / data model — the
capability_sourcesfiles copied verbatim so new tool code is written against the real types. - Per-task causal impact of prior candidates — which task ids each prior edit broke and fixed, plus the currently-passing set to protect. Real causal feedback so a known regression is never re-introduced.
Cross-iteration memory
| File | Owner | Purpose |
|---|---|---|
LEDGER.md |
framework | Facts: every iteration's outcome + exact tasks it broke/fixed |
JOURNAL.md |
optimizer | Append-only handover across the run (tried / worked / regressed / refuted / focus-next) |
PROCESS.md |
optimizer | Explainability, snapshotted per candidate |
RUNMAP.md + prior_iterations/ |
framework | Manifest + every prior iteration's PROCESS.md and capability diff |
Because it sees all failure clusters, the protect-set, and the prior causal impact at once, the optimizer produces one bold, multi-part candidate per iteration that addresses every cluster without regressing the wins — not a one-line tweak.
What cap-evolve can optimize
| Capability | What the optimizer may change |
|---|---|
| System prompts | Rewrite / consolidate / add rules, examples, output contracts — never drop a needed rule |
| Tool implementations | Edit tool code for deterministic enforcement; add / wrap / swap tools (never bare-remove) |
| MCP tool surfaces | Safe edits only — tool docs, in-description examples, which tools are exposed |
| Skill packages | An Agent Skill dir — SKILL.md bodies, references, and executable scripts |
Combine them, e.g. capabilities: [system-prompt, tools].
Optimization algorithms
hill-climb — simple global climber (recommended for first runs)
- Parent: always current best
- Focus modes:
--focus all(whole train set),--focus cyclic(one task at a time),--focus hardest-first(lowest-scoring tasks first) - Gate: val paired significance (Δ > k·SE)
- Best for: broad capability gaps; first baseline runs; feedback-poor tasks
gepa — Pareto frontier, trace-based reflection (flagship for expensive rollouts)
Independent implementation of the GEPA algorithm (arXiv:2507.19457).
- Parent: sampled from a per-instance Pareto frontier (frequency-weighted)
- Economy: cheap minibatch local gate first; expensive full-val gate on pass
- Reflection: per-task trace-based dataset fed back into the optimizer prompt
- Merge: system-aware merge across complementary lineages
- Budget:
--max-metric-calls(primary) or--max-iterations - Best for: expensive rollouts; rich feedback; max quality per evaluation dollar
skillopt — annealed epochs × minibatches (flagship for disciplined runs)
Independent implementation of the SkillOpt algorithm (arXiv:2605.23904).
- Parent: always current best (single lineage)
- Learning rate: textual edit budget (integer) decaying on
constant | linear | cosineschedule - Buffer: per-epoch rejected edits + failure patterns injected into the next optimizer prompt
- Meta-update: gated epoch-boundary slow/meta update that fixes longitudinal regressions
- Gate: val paired significance
- Best for: disciplined annealing; per-epoch consolidation
agent-optimize — free-form agentic loop
The agent drives the loop itself (orchestration_mode: agent). The agent proposes candidates, runs the baseline, gates every candidate on the full val split, and seals the test once. Honesty invariants (accept/reject always on full val; test sealed; no overfitting) are enforced in core-owned scripts, not editable markdown.
19 Agent Skills
| Component | Skill | Purpose |
|---|---|---|
| Orchestration | orchestrate |
End-to-end pipeline driver — sequences all phases and algorithms |
using-cap-evolve |
Session-start router — "optimize X" → intake |
|
| Phases | intake |
Interview + scaffold .capevolve/project/ |
implement-and-check |
Hard gate — cap-evolve check must be green |
|
baseline |
Freeze split (seeded, once); score seed on val | |
evaluate |
Multi-trial scoring with variance (mean + SE + pass^k) | |
diagnose |
Cluster failures into learning signal | |
gate |
Paired significance test (Δ > k·SE, val-only) | |
finalize |
Seal test; score best candidate exactly once | |
report |
Generate dashboard.html + report.md |
|
| Capabilities | system-prompt |
Prompt editing — rewrite / consolidate / add, never drop |
tools |
Tool code editing — add / wrap / swap, never bare-remove | |
mcp-tool |
MCP tool surface — safe edits only | |
skill-package |
Whole Agent Skill packages | |
| Algorithms | hill-climb |
Simple global climber (3 focus modes) |
gepa |
Pareto frontier + trace-based reflection | |
skillopt |
Annealed epochs × minibatches | |
agent-optimize |
Free-form agentic loop | |
| Optimizers | run-optimizer |
Dispatcher to 14 coding-agent backends via optimizers/registry.yaml |
14 Optimizer backends
Any coding-agent CLI can serve as the optimizer — resolved by a single row in optimizers/registry.yaml. Adding a new one is one YAML row.
| Backend | Notes |
|---|---|
claude-code |
Anthropic Claude Code |
codex |
OpenAI Codex |
gemini-cli |
Google Gemini CLI |
opencode |
Meta opencode |
openclaw |
OpenClaw |
ibm-bob |
IBM Bob |
generic |
Any OpenAI-compatible CLI |
mock |
Deterministic, zero-API ✅ — used in CI and toy_calc |
cursor |
Cursor cursor-agent |
droid |
Factory Droid |
copilot |
GitHub Copilot CLI |
kimi |
Moonshot Kimi |
pi |
earendil-works Pi |
antigravity |
Google agy (configurable wrapper) |
Benchmark results
Numbers are cross-checked against committed run artifacts under examples/*/run_full/. Each result is labeled by split discipline:
- fit metric —
train == val == test(no holdout). Shows the loop works; not a generalization claim. - held-out — test ids the optimizer never saw, scored exactly once at finalize.
| Benchmark | Split discipline | Baseline → Optimized | Gain |
|---|---|---|---|
| toy_calc (zero-API, mock optimizer) | sealed test | 0.0 → 1.0 |
deterministic proof of loop |
| τ²-bench airline — policy + tools, 50 tasks × 10 trials | val — fit metric | 0.536 → 0.712 |
+0.176 / +32.8% relative |
| τ²-bench airline — held-out 30/10/10 split | sealed test — held-out | 30.0 → 47.5 |
+17.5 pp / +58.3% relative |
| τ²-bench airline — agent-orchestration mode, held-out 30(val)/20(test) | sealed test — held-out (single-trial) | 0.400 → 0.550 |
+0.150 / +37.5% relative |
| SkillsBench — 4 skill packages, 7 val / 3 test tasks | sealed test — held-out | 0.556 → 0.667 |
+0.111 / +20.0% relative |
τ²-bench airline detail (fit metric run)
- Capability: airline policy + tools optimized jointly (
[system-prompt, tools]) - Optimizer:
claude-code@claude-opus-4-6 - Runner + user simulator:
openai/gpt-oss-120bvia IBM RITS - Algorithm:
hill-climb --focus all, 10 iterations, paired significance gatek_se 0.2 - Accepted iterations: 5 of 10 (iter 1, 3, 5, 6, 7 — the rest were within-noise rejections)
- What changed: deep in-code tool edits (
tools.py593 → 832 lines; policy 166 → 233 lines) — not just prompt tweaks
SkillsBench detail (held-out)
- Capability: four shared office-document skill packages (
docx/pptx/xlsx/pdf) - Agent under test:
claude-sonnet-4-6in a Docker sandbox - Optimizer:
claude-code@claude-opus-4-8 - Iterations: 7 (best
cand_0004, 4 accepted); stopped on a real ceiling (two unsolved tasks diagnosed as broken oracles, not overfitting candidates)
External context (not apples-to-apples)
Different benchmark versions, models, splits, simulators — contextual evidence only:
| Work | Benchmark | Result | Relative |
|---|---|---|---|
| EvoTool (arXiv:2603.04900) | original τ-Bench airline, GPT-4.1 | 35.9 → 39.1 | ~+8.9% |
| EvoTool | original τ-Bench airline, Qwen3-8B | 14.4 → 15.7 | ~+9.0% |
| Evolutionary Context Search | τ²-Bench | reported +23.3% | +23.3% (unverified citation) |
| cap-evolve | τ²-Bench airline, held-out 30/10/10 | 30.0 → 47.5 (+17.5 pp) | +58.3% |
Honest caveat: EvoTool uses the original τ-Bench (different, easier benchmark). These rows are not directly comparable. cap-evolve's +58.3% is a within-run relative improvement on its own held-out split.
Key engineering decisions
| Decision | Rationale | Implementation |
|---|---|---|
| Zero runtime dependencies | Portable, reproducible, auditable | Pure Python 3.10+ stdlib (dataclasses, json, pathlib, subprocess) |
| Seeded splits, sealed test | Honest test number; no data leakage | splits.json written once; test_used flag prevents re-scoring |
| Paired significance gate | Rejects marginal noise gains | stats.combined_stderr() + gate.decide(Δ > k·SE) — in core, not editable docs |
| Git-backed iteration store | Full auditability + resume capability | Each candidate a commit; RunDir checkpoints entire run state |
| Adapter contract | Agent-agnostic, benchmark-agnostic | 3 abstract methods (tasks, run_target, score) + 2 optional hooks (materialize, live) |
| Skill library | Composable, swappable pipeline components | 19 skills; each has scripts/run.py + SKILL.md + references; extending is one folder or one registry row |
| Multi-trial + pass^k | Handles stochastic agents honestly | run_trials_pool() + stats.pass_k() / pass_at_k() + bootstrap CIs |
| Optimizer memory | Never re-proposes dead ends | RejectedMemory + History track and inject rejected edits and failure patterns |
| Consuming-LLM profiles | Adapts optimizer prompt to the reader tier | target_model tier (frontier / strong / mid / weak); weaker readers get more explicit rules and worked examples |
--resume support |
Survives pod evictions, crashes, timeouts | Reopens run dir via exist_ok=True; GEPA reconstructs Pareto frontier from gepa_state.json checkpoint |
Try it — zero API key required
toy_calc is a deterministic stand-in agent that only answers correctly when its system prompt contains a [CALC] marker. The mock optimizer adds it, so the score provably rises — no model is called.
git clone https://github.com/skillberry-ai/cap-evolve.git
cd cap-evolve
python3 -m venv .venv && source .venv/bin/activate
pip install ./core # cap-evolve-core · CLI: cap-evolve · zero runtime deps
bash examples/toy_calc/run.shExpected output:
baseline_val 0.0 -> test_reward 1.0 (gate-accepted, test sealed) + dashboard.html
Open the printed dashboard.html in any browser.
Three adoption paths
| Path | Use it when | Start |
|---|---|---|
| Claude Code plugin | You use Claude Code and want slash commands + honesty hooks | claude --plugin-dir ./plugins/cap-evolve then follow RUN.md |
| Another coding-agent host | Codex, Gemini, opencode, Cursor, Droid, Copilot, Kimi, Pi, Antigravity, openclaw, IBM Bob | ./install.sh --host <name> then follow RUN.md |
| Manual adapter + CLI | You want to wire the adapter yourself and drive cap-evolve directly |
See docs/OPTIMIZE_YOUR_OWN.md |
The adapter contract (3 required methods)
from cap_evolve import CapabilityAdapter, Task, Rollout, Score
class MyAdapter(CapabilityAdapter):
def tasks(self, split: str) -> list[Task]:
"""Return tasks for 'train' | 'val' | 'test'."""
def run_target(self, task: Task, ctx, *, seed: int = 0) -> Rollout:
"""Run your agent with the candidate LIVE as ctx. Return Rollout."""
def score(self, task: Task, rollout: Rollout) -> Score:
"""Score in [0, 1] + natural-language feedback. Never leak the gold."""Everything else — splits, trials, gating, pass^k, sealed test, memory, and dashboard — is provided by the core.
For common cases, don't write an adapter from scratch — copy a ready-made template (templates/adapters/): JSONL, HuggingFace, tau2-bench, SWE-bench, SkillsBench.
Claude Code plugin
Install with claude --plugin-dir ./plugins/cap-evolve. Provides:
- Every skill as a
/cap-evolve:<skill>slash command (dual-mode: standalone + orchestrator-callable + headless JSON) - Honesty hooks (in core-owned scripts, never in editable markdown):
PreToolUse— denies edits to sealed test/gold filesStop/SubagentStop— block finishing untilcap-evolve checkand the gate are green
- Read-only diagnoser + writer-proposer subagents
using-cap-evolverouter for "optimize X" requests
What's new in v0.1.0
Added
- 19 Agent Skills — phases (intake, implement-and-check, baseline, evaluate, diagnose, gate, finalize, report), capabilities (system-prompt, tools, mcp-tool, skill-package), algorithms (hill-climb · gepa · skillopt · agent-optimize), run-optimizer backed by
registry.yaml, orchestrate + using-cap-evolve router. gepa— real GEPA: two-stage minibatch-then-full-val economy, per-instance Pareto frontier with frequency-weighted parent sampling, trace-based reflective dataset, round-robin component focus, system-aware merge across lineages, rollout/metric-call budget, eval cache (arXiv:2507.19457).skillopt— epochs × mini-batches with a textual learning rate (integer edit budget on constant | linear | cosine schedule), within-epoch rejected-edit buffer, gated epoch-boundary slow/meta update (arXiv:2605.23904).- Self-contained
dashboard.html(no CDN): KPI strip, cumulative-best stair, tasks × iterations pass/fail heatmap, per-iteration diff, lineage tree (merges as multi-parent), optimizer-vs-runner cost / tokens / latency, annotations. cap-evolve report --terminal— ANSI in-chat progress report.- Claude Code plugin (
plugins/cap-evolve/): every skill as/cap-evolve:<skill>, honesty hooks (PreToolUse / Stop / SubagentStop) in core-owned scripts, diagnoser + writer subagents, using-cap-evolve router. - Git-backed iteration store (default) + optimizer memory (
MEMORY.md/STATE.md/rejected.jsonl). - Host-agnostic installer —
./install.sh --host <name>wires any coding-agent's skills dir. cap-evolve run --resume— continue an interrupted run from its last completed state. Picks up at iteration N+1; GEPA reconstructs its full pool / lineage / frontier fromgepa_state.json+ rollouts. Explicit budget flags extend a resumed run.- Consuming-LLM profiles (
target_modelincapevolve.yaml) — adapts optimizer prompt and guidance to the runtime/consuming model's capability tier (frontier | strong | mid | weak). Distinct fromoptimizer_model. - Six more optimizer backends —
cursor,droid,copilot,kimi,pi,antigravity— bringing the supported set to 14, matchingobra/superpowers. Adding one is a single row inregistry.yaml. - Per-iteration optimizer dollar cap (
optimizer_usd_per_iterincapevolve.yaml) — threaded intorun-optimizer --usd-budget, enforced natively where the CLI supports it (e.g.claude-code --max-budget-usd). - Examples: toy_calc (zero-API deterministic proof), tau2_airline (real τ²-bench run with committed artifact + static dashboard), skillsbench (skill-package optimization in Docker).
Fixed
- Benchmark CI robustness — three fixes so a broken runner or gateway is loud, not a silent all-0.000 "success": (1) optimizer CLI hard-verified at setup; (2) model-gateway budget preflight (one tiny call that aborts with a clear error on
429 budget_exceeded); (3)run_suite.shiterates task list on FD 3 so an optimizer reading stdin can no longer drain the here-string and cut the suite off after one task.metrics.pydetects infra-dominated evals (majority trials errored + reward ≈ 0) and renders them as⚠️ infra-error, excluding from suite mean/flip counts. - Adapter template — scaffolded
adapter.pynow matches the realCapabilityAdaptercontract (run_target(task, ctx, *, seed),materialize/liveas optional). Old stub used a stale signature that let a filled-in body silently pass the stub probe. - Honest-eval core — seeded splits with sealed test, paired significance gate (default), multi-trial variance, pass^k + pass@k, bootstrap CIs.
Changed
- Skill library collapsed 26 → 19 — 8 per-CLI optimizer skills became one
run-optimizerskill + one-row-per-optimizerregistry.yaml; three hill-climb clones (all-at-once / cyclic / hardest-first) became onehill-climbskill with--focus. Adding an optimizer is now one YAML row. - Adapter contract updated to
tasks(split)·run_target(task, ctx, *, seed)·score(task, rollout)· purematerialize(candidate_dir, edits)+live(candidate_dir)context manager.apply()retained as back-compat hook. Per-trial seed threaded intorun_targetso pass^k measures real variance. - Honest-eval upgrades — paired significance gate is now the default; test seal is seal-on-success (a finalize crash no longer burns the headline); infra-vs-capability failures use structured
Rollout.errorsignal instead of substring-matching feedback.
Documentation
| Document | Use it when |
|---|---|
| Getting started | You want your first successful run (toy_calc walkthrough) |
| Installation | Host-specific setup, credentials, dashboard |
| Optimize your own agent | Integrate your agent or benchmark |
| Adapter templates | Copy-and-run adapters (JSONL, HuggingFace, tau2-bench, SWE-bench, SkillsBench) |
| Adapter contract | Implementing an adapter |
| Architecture | Pipeline, optimizer context, skill library |
| Agent orchestration | Agent-driven loop (orchestration_mode: agent) |
| Honest evaluation | Splits, gates, sealing, honesty invariants |
| Results | Full experiments with models, task counts, costs, and artifacts |
| Comparison | Positioning vs DSPy, GEPA, promptfoo; external result context |
| Extending cap-evolve | Adding a capability, optimizer, or algorithm |
| Roadmap | Planned work |
| Troubleshooting | Setup failures and common issues |
| RUN.md | CLI guide — sequence of skills for any agent host |
Project details
- Python: 3.10+
- Runtime dependencies: 0 (pure stdlib)
- License: Apache-2.0
- Status: Beta (0.x) — anything may change per semver 0.x rules
- Repository: https://github.com/skillberry-ai/cap-evolve
- Site: https://skillberry-ai.github.io/cap-evolve/
cap-evolve includes no third-party code. The gepa and skillopt skills are independent implementations of the GEPA and SkillOpt papers. The bundled example uses tau2-bench (MIT). Full citations: docs/sources.bib.
Citation
@software{cap-evolve,
title = {cap-evolve: a skills-native, host-agnostic harness for honestly
optimizing AI-agent capabilities},
year = {2026},
note = {https://github.com/skillberry-ai/cap-evolve}
}