Recursively self-improving skill files — runs only inside a Claude Code session, no API key.
A skill file (SKILL.md) sometimes contains wrong or incomplete instructions.
An agent follows them, fails, improvises something that works — and that
learning is lost. poseidon closes the loop: it runs an agent against a
skill, measures how reliably the task succeeds against a real test,
diagnoses which instruction caused failures, rewrites the skill, and
re-measures — until the task passes deterministically across N runs.
There is no separate service and no ANTHROPIC_API_KEY. The engine is your
own running claude session: the /poseidon:forge command is the
orchestrator, "a fresh attempt with the new skill" is a subagent
(forge-runner) it spawns each iteration, and the deterministic bookkeeping
(versioning, running the test, computing success rate) is a small bundled
Python helper (plugin/lib/forge.py) that touches only the filesystem.
┌─────────────────────────────────────┐
│ Task (per skill) + an approved │
│ test (the oracle) │
└──────────────────┬──────────────────┘
┌──────────┐ run N times ┌───▼─────────┐ trace+verdict ┌────────────┐
│ Skill ├────────────────►│ forge-runner├────────────────►│ orchestrator│
│ (v_k) │ │ subagent │ │ (diagnose) │
└────▲─────┘ └─────────────┘ └─────┬──────┘
│ accept iff converged AND no regression │ edit
│ ▼
┌────┴───────┐ re-measure with N fresh subagents ┌──────────────────┐
│ Promote │◄────────────────────────────────────────┤ Skill v_{k+1} │
│ + version │ (suggest mode never writes) └──────────────────┘
└────────────┘
Five design rules do the real work:
- External oracle. A skill edit is only "good" if an approved automated test says the outcome is correct. The LLM never grades itself.
- Rate, not anecdote. Every skill version is run N times; we track the success rate and whether it's deterministic. "Worked once" is rejected.
- No regression. A candidate is accepted only if it improves the failing task without lowering any other task's success rate.
- Test-approval gate. The tool may draft a test, but a test can only gate
a promotion after a human approves it (
task.json: "approved": true). This is what stops the loop from "passing" by writing a weak test. - Session budget. Each
/poseidon:forgeinvocation does at most one diagnose → fix → validate cycle and checkpoints progress to disk, so a single session stays under roughly 40% of the context window. If more iterations are needed, you re-run the command and it resumes from the checkpoint in a fresh session.
# Try it locally (dev):
claude --plugin-dir /path/to/poseidon/plugin
# Inside Claude Code, reload after edits:
/reload-pluginsRequires python on PATH and pytest (the test oracle): pip install -r requirements.txt.
The bundled csv-stats example skill has a deliberately wrong instruction
(float(row["amount"]), which crashes on currency like "$1,234.56"). Inside
a Claude Code session with this plugin loaded:
/poseidon:forge csv-stats --mode suggest --n 3
This spawns 3 fresh forge-runner subagents against the current skill, then
3 more against a proposed fix, and reports before/after success rate +
whether each run had to deviate from the literal instructions.
| mode | writes skill? | stops when |
|---|---|---|
suggest |
no | a validated candidate is found |
auto |
yes (versioned, rollback-able) | one accepted promotion |
autonomous |
yes | converged, or one checkpointed cycle (re-run to continue) |
- Locate the skill and establish the oracle — an approved automated test.
If none exists,
/poseidon:forgedrafts one and asks you to approve it before it can gate any change (so it can't "pass" by writing a weak test). - Baseline: spawn the
forge-runnersubagent N times (fresh sandbox each), record PASS/FAIL, compute success rate + determinism. - Diagnose: read the failing runs' traces, identify the single instruction that failed or had to be deviated from.
- Edit: rewrite
SKILL.mdminimally and concretely; show a diff. - Validate: re-run N fresh subagents with the candidate skill. Accept only if it converges (rate ≥ threshold AND deterministic) with no regression.
- Apply per
--mode; report before→after stats, the diff, and rationale. Rolls back automatically if a promotion regresses.
If your pipeline spawns several agents, each following its own skill, you
don't have to notice which one misbehaved. A bundled hook watches every tool
call (including inside subagents) and attributes failures to whichever
skill/subagent was active at the time, using the transcript. If one skill
racks up repeated failures in a session, the hook makes the agent
self-trigger /poseidon:forge <that-skill> --mode suggest before it
finishes — once per skill per session — so agent A's broken skill and agent
B's broken skill get flagged separately instead of one vague "something broke"
message. Attribution is heuristic and best-effort; the approved-test oracle
(rule 4) still gates anything ever being written, so a mis-attributed nudge
can at worst run a no-op suggest pass, never a bad promotion.
A task is a prompt + input files + an approved test:
tasks/<name>/
task.json # name, prompt, input_files, artifact, test, approved
<input files> # copied into a fresh sandbox per run
test_outcome.py # pytest; runs with cwd = sandbox, asserts on the artifact
The contract: the agent writes its result to the artifact (e.g. output.json)
in the sandbox; the test asserts on it. /poseidon:forge will draft this for
you and ask for approval if it doesn't exist yet.
plugin/
├── .claude-plugin/plugin.json # manifest
├── skills/forge/SKILL.md # /poseidon:forge — the orchestrator (the loop)
├── agents/forge-runner.md # the clean-room "fresh session" subagent
├── hooks/
│ ├── hooks.json # PostToolUse / PostToolUseFailure / Stop / SubagentStop
│ └── detect_friction.py # keyless, per-skill friction detector + self-trigger
└── lib/forge.py # keyless helper: versioning, oracle, determinism, checkpoints
examples/
skills/csv-stats/SKILL.md # ships with a deliberate flaw
tasks/csv-stats/ # task + sales.csv + approved test
See ARCHITECTURE.md for the mechanism in more detail.