💡 SkillOpt-Style Self-Improving Skills: Eval-Gated, Human-Reviewed Agent Skill Evolution

[don-petry]

Summary

Stand up an eval-gated, human-reviewed self-improvement loop for our agent skills and prompts, modeled on Microsoft's open-source SkillOpt pattern. Today every skill and prompt in frameworks/, prompts/, and agents/ is hand-authored markdown that only changes when a human writes a commit — there is no feedback loop turning agent run outcomes back into better skills. The proposal: let agents propose bounded edits to their own skill/prompt markdown, validate each candidate against a held-out eval set, and route the winner through the existing pr-review + release-channel gates as a normal PR. Weight-free (no fine-tuning), fully version-controlled, and human-gated — improvement becomes a reviewable diff, never a silent mutation.

Market Signal

Self-evolving agent skills (not weights) became a distinct, productized category in 2025–2026:

• Microsoft SkillOpt (open-source) optimizes agent skills without touching model weights: it makes bounded edits to a markdown skill file, gates every candidate behind a strict held-out validation set, and ships only the winner as best_skill.md — author-reported 52/52 wins (microsoft/SkillOpt, MS Research: "Executive Strategy for Self-Evolving Agent Skills", VentureBeat).
• GEPA (reflective prompt evolution) beats RL (GRPO) while using up to ~35× fewer rollouts, and ships inside DSPy as a drop-in optimizer (GEPA paper 2507.19457, gepa-ai/gepa, DSPy GEPA). ACE (Agentic Context Engineering) evolves the agent's "playbook" context the same way (2510.04618).
• Anthropic's skill-creator explicitly closes the loop with eval + blind A/B testing to measure and refine Agent Skills — which are themselves composable markdown folders, exactly our format (Improving skill-creator, Agent Skills, anthropics/skills).
• Research lineage: Reflexion and Voyager's ever-growing skill library established experience-driven self-improvement (Voyager); 2026 production guidance converges on eval gates + prompt versioning + canary in CI (AI-native CI/CD eval gates, prompt versioning & change management, MLflow: evaluating skills).

Hype filter: fully agent-authored skills remain aspirational. What is shippable today is the SkillOpt/skill-creator shape — agent proposes a bounded edit, an offline eval gate decides, a human reviews the PR (GitHub: reviewing agent PRs).

User Signal

This org already has every prerequisite except the loop itself:

• Markdown skills under version control: frameworks/bmad-method/ (25+ skills), prompts/*.md (triage, deep-review, synthesize, dev-lead phases), agents/*.md — SkillOpt's exact input format.
• A human-gate pipeline that's idle for this: Ideas → idea:approved → initiative-planner → epic/DAG → dev-lead → pr-review. We can run skill improvements through it unchanged.
• Guardrail alignment: the Safe Release Strategy (Initiative: Safe Release Strategy for Agentic Workflows (versioning · rings · canary) #495) already blocks agents from moving their own release tags. A weight-free, PR-gated improvement loop respects that boundary by construction — no self-promotion, only proposed diffs.
• Conspicuous gap: zero feedback collection exists today. Idea approval rates, initiative completion, pr-review human-override rates, and Token Cost Observatory spend are all captured-but-unused signals that could seed an eval set.

Shadow-mode dual-run (Idea 566) and health-gated promotion (#501) are the natural enforcement layer for rolling out an improved skill safely.

Technical Opportunity

A minimal, weight-free loop reusing what we already run:

1. Eval harness (the keystone): a skills/evals/<skill>/cases.jsonl held-out set per high-traffic skill (start with prompts/triage.md and prompts/deep-review.md), scored by a deterministic check or a Haiku-tier LLM-judge. No gate, no loop — so this lands first.
2. Proposer: a scheduled agent reads recent run outcomes (override rates, escalations, failures) and emits bounded candidate edits to one skill file — diff-only, size-capped, à la SkillOpt's edit budget.
3. Validation gate: each candidate runs against the held-out set; it must strictly beat the incumbent (no ties promoted) before anything is opened. Reward-hacking guard: eval cases are versioned separately and never edited by the proposer.
4. Human gate: the winner is opened as a normal PR (or an Idea here for larger skills), reviewed by pr-review + a human CODEOWNER — identical to any other change.
5. Safe rollout: merged skill rides shadow-mode dual-run (Idea 566) / health-gated promotion ([Phase 2] Replace PUT-contents clobber deploy with versioned, ring-staged, health-gated promotion #501) before becoming stable.

This is additive — no changes to the release-tag ruleset, no agent self-promotion, no model fine-tuning.

Assessment

Dimension	Score	Rationale
Feasibility	med	The hard part is building a trustworthy held-out eval set per skill; the loop plumbing reuses the existing Ideas → initiative → pr-review pipeline and weight-free markdown edits. Start with one skill end-to-end.
Impact	high	Converts unused outcome signals (override rates, approvals, cost) into compounding skill quality; SkillOpt/GEPA show double-digit gains, and it scales across every agent in the org.
Urgency	med	No fire today, but skill quality is hand-tuned and static; the longer we run agents without a feedback loop, the more captured signal we waste. Sequence after eval-harness groundwork.

Adversarial Review

Strongest objection: This is a reward-hacking and drift magnet. An agent optimizing its own skill against a metric will learn to game the eval rather than genuinely improve — and a self-modifying skill loop is exactly the kind of "agent edits its own infrastructure" the org deliberately fenced off with the release-channel ruleset.

Rebuttal: The design neutralizes both. (1) Reward hacking is bounded because the eval set is version-controlled separately and never writable by the proposer — the agent cannot edit the test, only the skill, and a strict-improvement gate rejects ties/regressions; SkillOpt's held-out-validation discipline and Anthropic's blind-A/B exist precisely for this. (2) Self-modification is not what's proposed — the agent never moves a tag or merges anything; it opens a diff that goes through the identical human + pr-review + promotion gates as any contributor's PR. It is strictly more conservative than today, where a human can hand-edit a skill with no eval gate at all. The genuine cost is building and maintaining honest eval sets — which is why the first deliverable is the eval harness alone, with the loop gated on it proving stable.

Suggested Next Step

Pilot on a single high-traffic skill (prompts/triage.md). Deliverable 1: a versioned held-out eval set + scorer wired into CI as a non-blocking report (no loop yet). Deliverable 2: a manual "propose → validate → PR" dry run by a human operator to prove the gate rejects regressions. Only then automate the proposer. Gate the rollout layer on Safe Release Strategy Phase 2 (#501) and shadow-mode dual-run (Idea 566).

[github-actions[bot]]

📋 Initiative planned by the BMAD Scrum Master (Bob).

Epic #581 — Initiative: Eval-gated, human-reviewed self-improving skills (SkillOpt-style)

6 stories created (inert — labelled initiative, NOT initiative:auto):

• [Phase 1] Define held-out eval-case format + seed prompts/triage.md case set #582 (M) — [Phase 1] Define held-out eval-case format + seed prompts/triage.md case set
• [Phase 1] Build the deterministic eval scorer + offline tests (triage) #583 (M) — [Phase 1] Build the deterministic eval scorer + offline tests (triage)
• [Phase 1] Wire the eval scorer into CI as a non-blocking report #584 (S) — [Phase 1] Wire the eval scorer into CI as a non-blocking report
• [Phase 2] Extend the harness to prompts/deep-review.md via an LLM-judge scorer #585 (M) — [Phase 2] Extend the harness to prompts/deep-review.md via an LLM-judge scorer
• [Phase 2] Strict-improvement gate + manual propose->validate->PR runbook with regression-rejection proof #586 (M) — [Phase 2] Strict-improvement gate + manual propose->validate->PR runbook with regression-rejection proof
• [Phase 3] Automate the scheduled proposer behind the safe-rollout gates #587 (L) — [Phase 3] Automate the scheduled proposer behind the safe-rollout gates

Open questions for review:

• Directory naming: the idea specifies skills/evals//cases.jsonl, but this repo's skills live under prompts/ and frameworks/ (there is no skills/ dir today). Confirm skills/evals/ vs evals/ or prompts/evals/ before Story 1 creates the tree.
• The safe-rollout layer depends on shadow-mode dual-run (Discussion 💡 Shadow Mode Dual-Run for Safe Agent Canary Validation #566), which is not yet an epic/issue — Story 6's activation prerequisite is only partially tracked ([Phase 2] Replace PUT-contents clobber deploy with versioned, ring-staged, health-gated promotion #501). Should 💡 Shadow Mode Dual-Run for Safe Agent Canary Validation #566 be promoted to an initiative before Phase 3 starts?
• Held-out set sizing/sourcing: how many cases per skill, and may we seed them from real historical triage/deep-review decisions (and how should we de-identify PR content)? This directly affects how much trust the gate's verdict earns.
• This initiative's Phase 3 rollout overlaps the Safe Release Strategy epic (Initiative: Safe Release Strategy for Agentic Workflows (versioning · rings · canary) #495 / [Phase 2] Replace PUT-contents clobber deploy with versioned, ring-staged, health-gated promotion #501). Confirm Phase 3 should consume that ring/health-gating machinery rather than re-implement any of it.

---

Review the epic and its sub-issue DAG, adjust as needed, then add initiative:auto to epic #581 to hand it to initiative-driver for auto-implementation.

[github-actions[bot]]

Enhancement: SkillOpt Loop — Held-Out Set Immutability & Human Gate Mechanics

Sharpened problem & goal
The proposal is to stand up an eval-gated, human-reviewed self-improvement loop for skills/prompts in frameworks/, prompts/, and agents/. The highest-risk ambiguity: what is the held-out eval set and how is it kept immutable from the skill-proposer? A self-improving loop that can influence its own test set violates the fundamental assumption of the evaluation. The eval set must be owned by humans, not the proposer, and immutable during any run.

Context
The org already has the key infrastructure for this:

• evals/ directory + scripts/evals/run-eval.sh — the eval framework exists; the skill-opt held-out set is a new subdirectory within this existing structure
• holdout-guard.yml + scripts/lib/holdout-guard.sh — the CI gate that prevents the automated skill-proposer identity from touching evals/ paths (AGENTS.md: "fails any PR authored by the automated skill-proposer identity that changes a held-out path"); this IS the immutability mechanism the proposal needs at the infrastructure layer — it already exists
• scripts/evals/notify-eval-health.sh — eval health notifications already in place
• The existing pr-review pipeline + scripts/cut-release.sh / channel-tag promotion is already the "human review" step the proposal references — skill edits should flow through this same path

Microsoft SkillOpt's key insight (applicable here): the skill-proposer should propose bounded edits (parameter adjustments, example additions, instruction appends) not full rewrites — this limits the search space and makes human review tractable. The proposal should specify what edit boundaries are enforced.

Impact / Effort

• Impact: HIGH — a feedback loop turning real run outcomes into better skills compounds over time; current hand-authored skills have no systematic improvement mechanism
• Effort: XL — eval set design, held-out guard extension, proposer agent, gated promotion pipeline, and human review workflow together are a multi-sprint initiative; the SkillOpt paper itself took a dedicated team

Suggested acceptance criteria

• A held-out eval set (evals/skill-opt/) is authored by a human covering at least 10 scenarios per target skill; holdout-guard.yml is extended to protect this specific path from the skill-proposer identity (the guard already exists — this is a path addition)
• The skill-proposer is constrained to bounded edits only: appending examples, adjusting model-routing parameters, adding acceptance criteria — not rewriting skill identity, rubric structure, or held-out eval scenarios
• Each proposed skill edit is evaluated against the held-out set before a PR is opened; a PR is only opened if the candidate scores ≥ current skill on ≥ 80% of scenarios
• Skill edit PRs flow through the existing pr-review pipeline with artifact_type=skill (per Discussion 💡 Generalize pr-review into a Context-Adaptive "Review" Agent (PRs, Plans, Skills) #607's review registry) before any human merges
• A dedicated proposer identity (service account or GitHub App) authors all skill-edit PRs; the holdout-guard.yml pattern (author + path checks) enforces this separation of concerns

[github-actions[bot]]

Weekly Update

What Changed

Both major platforms have now shipped the "agents improve their own instructions offline, with human review" pattern this idea proposes:

• GitHub Copilot "dreaming": public preview, 2026-10-28 (Universe). It reviews past agent work and proposes checked-in skill and instruction changes as pull requests, plus code-review customization. It is opt-in, consumes AI credits, and is admin-configurable. See the changelog entry and the dreaming docs. Copilot Memory went GA at the same time.
• Anthropic Dreams (Managed Agents research preview). It consolidates memory between sessions into a separate output memory store that can be inspected and diffed before being attached. On 2026-10-01 it gained support for Opus 5.5, Fable 5.1 and Sonnet 5.5 (release notes).

What this tells us for this repo: the market settled on proposals as reviewable diffs, separated from the live artifact. That matches the held-out-set immutability and human-gate mechanics in the June enhancement comment above. Neither vendor product covers what this idea uniquely needs: eval-gating against a held-out set before the PR exists, and channel-tag ring promotion after merge. Copilot dreaming only proposes; it doesn't prove.

Updated Assessment

Dimension	Previous	Current	Delta
Feasibility	med	med	→
Impact	high	high	→
Urgency	med	med	→ (vendor validation strengthens the case but creates no deadline)

Recommendation

Keep watching, with a cheap experiment. Turn on Copilot dreaming's repository-customization mode on one low-risk repo for 2 weeks. Count how many of its proposed instruction PRs the maintainer would merge, then run those same proposals through the existing skill-eval pipeline. If dreaming's suggestions often fail our evals, that confirms the eval gate is the differentiated part to build here. If they mostly pass, adopt the vendor proposer and build only the gate.