Show and tell: dsh-rule-evolve ? verification-driven self-evolution loop (rules are only as good as their checks)

[zoahdev]

dsh-evolve ? a verification-driven self-evolution loop for dsh plugin maintenance.

Repo: https://github.com/zoahdev/dsh-evolve ? Zero-dependency CLI

The loop

experience (markdown)  ?  learn  ?  rules (AGENTS.md)  ?  verify (real checks)

Every rule carries its source and its last verification result. Evolution is auditable; nothing is "learned" without being checked.

Proof it actually evolves (dogfood run, today)

1. learn extracted 5 real lessons from our doctor experience doc (Bug: installing any plugin that depends on @deepseek-ai/dsh-tools breaks every tool call (Cannot read properties of undefined (reading 'prepare')) #1697/Bug: profile package.json with UTF-8 BOM crashes dsh web (Unexpected token ... is not valid JSON) #1842/【Bug】Windows 上官方 minimal（极简模式）预设的 bash 无法启动：默认 shellPath /bin/bash 不可解析（实测+修复建议） #1856/Bug: session search crashes with `RangeError: Invalid string length` on large sessions (root cause + fix) #1859/[插件安全边界] tools/pre-execute 在 ask 审批前已执行，可先完成独立 Node.js 子进程 #1863)
2. rules rendered them into an AGENTS.md block
3. verify ran the real dsh-plugin-doctor check against its own repo ? and it failed: the pre-execute-side-effects lint false-positived on doctor's own source (comments + its own regex matched)
4. We fixed the lint (only quoted event names / method calls count; tests dirs skipped; regex no longer self-matches), released v1.10.1, re-ran verify ? 5/5 entries verified ?, doctor passes its own check (exit 0)

That is the loop working: a tool checking itself, finding its own bug, and the fix being captured as a verified rule.

Why this slot is empty

The ecosystem has 12+ memory plugins and #1881 proposes triple-layer persistent memory. Its Layer 1 is Git-tracked Markdown rules. dsh-evolve is the production + validation half of that layer: where rules come from and how they stay honest. Complementary to #1881, not a duplicate.

Try it

node scripts/dsh-evolve.mjs loop --from docs/troubleshooting.md --out experience.jsonl --dir ./my-plugin

Feedback welcome ? especially on the rule-extraction heuristics and the verify contract.

[zoahdev]

v0.2.0 ? full self-improvement loop (release: https://github.com/zoahdev/dsh-evolve/releases/tag/v0.2.0)

New commands:

• reflect --task <t> --result <md> --out <jsonl> ? turn a task retrospective into experience entries (with task/goal context)
• evolve --experience <jsonl> --dir <repo> --profile web --log EVOLUTION.md ? verify against the real repo, install verified rules into the dsh profile's AGENTS.md (with backup), append one round to the evolution log
• install-rules --profile web ? idempotent rules install

Live demo (Round 1, real repo): reflect ? 5 rules ? real dsh-plugin-doctor check on its own repo ? 5/5 verified ? ? rules installed into profile ? EVOLUTION.md logged: https://github.com/zoahdev/dsh-evolve/blob/main/examples/demo/EVOLUTION.md

The loop is now closed: reflection ? rules ? verification ? profile activation ? logged history. Next iterations: auto-extracting rules from failure logs, and a dsh plugin surface so agents can evolve themselves mid-session.

[zoahdev]

v0.3.0 ? the loop now learns from raw failure logs (release: https://github.com/zoahdev/dsh-evolve/releases/tag/v0.3.0)

• extract --task <t> --from <log> --hint <remedy> ? error lines become conditional rules (When "<error>" occurs, <hint>.); progress/noise lines skipped, dedup + tags
• audit --experience <jsonl> ? rule-library health: verified/unverified, tag & source distribution, near-duplicate pairs (Jaccard ? 0.7)

Dogfood: extracted 6 rules from our real npm ENEEDAUTH failure log (configure NPM_TOKEN and retry) in one command. 8/8 tests.

The self-improvement path is now: failure log ? extract ? verify ? install into profile AGENTS.md ? EVOLUTION.md round. Next: a dsh plugin surface so agents call this loop mid-session.

[zoahdev]

v0.4.0 ? the loop is now a dsh plugin: agents evolve themselves mid-session (release: https://github.com/zoahdev/dsh-evolve/releases/tag/v0.4.0)

dsh plugin --profile web add github:zoahdev/dsh-evolve#path:/plugin

Two tools registered on the agent:

• evolve_learn(task, experience, hint, profile) ? failure log / retrospective ? rules persisted to the profile experience store
• evolve_apply(profile, verifyCmd, dryRun) ? refuses unverified installs; with a passing verifyCmd, installs rules into the profile's AGENTS.md (backup created) and stamps every rule verified

So an agent can now, inside a session: hit a failure ? learn a rule ? run a check ? install the rule for all future sessions. 4/4 plugin tests + packed runtime smoke (fresh project, real register + real handler execution).

[zoahdev]

cc @imccyu @tianyicui ? this is the AGI-direction piece that closes the loop between dsh's own verification culture and agent self-improvement:

• Rules are produced from real failure logs, verified by real checks (our 5 upstream patches pass the official suite; doctor passes its own check), and only then installed into the profile ? unverified rules are refused
• Runs both as CLI and as a plugin (evolve_learn / evolve_apply), so agents evolve mid-session
• Full evidence chain: https://github.com/zoahdev/dsh-evolve/blob/main/examples/demo/EVOLUTION.md + packed runtime smoke

If maintainers see a place for a first-class rule lifecycle (produce ? verify ? install ? audit) in the official profile/preset system, we'll adapt the contract ? the shape is cheap to change now.

[zoahdev]

Naming clarity (ecosystem hygiene): another community project (william-jin-cmu/dsh-evolve) already uses dsh-evolve for in-session plugin hot-mounting. Different focus ? they evolve which plugins are mounted; this project evolves which rules are trusted, with verification gates. Both are complementary (hot-mount + verified rules). We are tracking a rename to dsh-rule-evolve; in the dsh-subscribe registry this project is listed as dsh-evolve (zoahdev).

[zoahdev]

v0.5.0 ? tool learning gate (release: https://github.com/zoahdev/dsh-rule-evolve/releases/tag/v0.5.0)

tool-verify --dir <plugin> runs the real readiness pipeline (doctor check), renders a readiness report (checks table + READY/ISSUES), and learns from its own failures: every failing/warning check becomes an experience entry stamped with the verification result. A tool that ships with bugs teaches the next tool how not to ship.

Dogfood: our doctor repo reports 8/8 PASS ? READY ? ? and before v1.10.1, this same gate caught its own lint bug.

The self-evolution path is now: failure log ? extract ? verify ? install into profile AGENTS.md ? EVOLUTION.md, plus tool ? tool-verify ? readiness report ? lessons learned. 10/10 tests.

[zoahdev]

v0.6.0 ? rule lifecycle: the library now maintains itself (release: https://github.com/zoahdev/dsh-rule-evolve/releases/tag/v0.6.0)

• score ? every rule gets a health score (verified count, usage count, recency) ? active / stale
• prune --min-score N ? soft-retires low-score rules (retired: true + reason; auditable, never deleted)
• merge-duplicates ? near-duplicate pairs merge into the higher-scoring rule (mergedInto kept)

Dogfood: 5/5 active, zero duplicates ? a healthy library. 11/11 tests.

This completes the self-improving memory loop: produce ? verify ? install ? use ? score ? merge/retire. Rules accumulate, but the library tells you what to keep ? "memory that forgets usefully" is the long-term half of agent learning.

[zoahdev]

v0.7.0 ? cross-session reinforcement (release: https://github.com/zoahdev/dsh-rule-evolve/releases/tag/v0.7.0)

Rules now carry usage and verification history:

• touch / plugin tool evolve_touch ? usageCount + lastUsedAt
• every successful verify / evolve / tool-verify ? verifiedCount + 1
• health score rewards both ? repeatedly verified + repeatedly used rules survive pruning; stale ones fade

That closes the long-term loop: produce ? verify ? install ? use ? reinforce ? score ? merge/retire. The rule library literally remembers what works across sessions. CLI 11/11 + plugin 5/5 + packed runtime smoke.

[zoahdev]

Two updates:

v0.8.0 ? local recall (https://github.com/zoahdev/dsh-rule-evolve/releases/tag/v0.8.0): recall --query "windows bash" returns the right rules first, via zero-dependency BM25-lite ? Layer-2-style retrieval without an embedding model, offline; verified + used rules rank higher. 12/12 tests.

Sixth upstream patch ? #1891 (https://github.com/zoahdev/deepseek-harness/tree/fix/session-persistence-recreate-on-enoent): external session-directory deletion no longer kills the process ? appendLines recreates the missing directory and retries once. Verified against the official suite: session-persistence-jsonl 239/239 baseline and 239/239 with patch.

The self-evolution stack is now: learn ? verify ? install ? use ? reinforce ? recall ? score ? merge/retire ? plus a tool-ready gate and a growing cherry-pick fleet.

[zoahdev]

v0.9.1 ? self-evolution marathon (https://github.com/zoahdev/dsh-rule-evolve/releases/tag/v0.9.1)

Ran the loop end-to-end with real commands and recorded the growth diary: https://github.com/zoahdev/dsh-rule-evolve/blob/main/examples/marathon/EVOLUTION-HISTORY.md

• Round 1: reflect ? 5 rules (doctor lessons) ? verified + installed
• Round 2: extract ? +6 rules from a real npm ENEEDAUTH log ? 11 rules verified
• Reinforcement: touch + tool-verify (8/8 READY) + score (11 active, 0 duplicates)

The run also surfaced a real bug ? merged batches could duplicate EXP ids ? fixed with a regression test (CLI 13/13, plugin 6/6). No manual curation anywhere: the library learned, verified, installed, reinforced, and reported its own health.

[zoahdev]

v0.10.0 ? Agent Growth Dashboard (https://github.com/zoahdev/dsh-rule-evolve/releases/tag/v0.10.0)

node scripts/dsh-evolve.mjs dash --experience experience.jsonl --evolution EVOLUTION.md --out agent-growth.html

One self-contained HTML page (opens offline, zero deps, bilingual): rule growth curve, library health, what the agent learned, most-used rules, evolution timeline, and a shareable Agent Growth Report card ("My agent learned N rules, all verified by real checks").

Live example from the real marathon: https://github.com/zoahdev/dsh-rule-evolve/blob/main/examples/marathon/agent-growth.html ? this is what "your agent is learning" looks like when it's visible. CLI 15/15 + plugin 6/6 + packed runtime smoke.

[zoahdev]

v0.10.1 ? shareable growth card (https://github.com/zoahdev/dsh-rule-evolve/releases/tag/v0.10.1)

The Agent Growth Report now exports a 1200?630 PNG share card (canvas, zero dependencies): "AGENT GROWTH REPORT ? 11/11 rules verified ? 3 uses ? 11 rules learned ? 2 evolution rounds". One click ? download ? post it anywhere.

This is the visibility half of self-evolution: the work is invisible until you can screenshot the growth. Live example includes the button: https://github.com/zoahdev/dsh-rule-evolve/blob/main/examples/marathon/agent-growth.html

[zoahdev]

v0.10.2 ? in-session growth report (https://github.com/zoahdev/dsh-rule-evolve/releases/tag/v0.10.2)

New plugin tool evolve_report: the agent itself summarizes the library and returns ready-to-post share text ? "My agent learned N rules, M/N verified by real checks, used X times". Five in-session tools now: learn / apply / touch / recall / report. The full loop ? learn, verify, apply, use, recall, and show the user it worked ? happens inside the conversation.

Plugin 7/7 tests + packed runtime smoke.

[zoahdev]

v0.11.0: the evolution loop is now embeddable anywhere with a one-line SVG badge.

dsh-evolve badge --experience experience.jsonl --evolution EVOLUTION.md --out badge.svg

Tiers by verified rules: starting (0) · learning (1-4) · building (5-9) · growing (10-19) · legend (20+). --json returns the machine-readable summary plus the SVG payload for CI/README pipelines.

So the loop is now: learn -> verify -> evolve -> show (dashboard + share card + badge). Release: https://github.com/zoahdev/dsh-rule-evolve/releases/tag/v0.11.0