DSH Harness Evolution

English | 简体中文

Methods, evolution curves, reproducible configurations, and measured plugin artifacts.

Freeze the actor and evolve its runtime harness. Inspect what each loop changes, how candidates are selected, what happened, and which reproduction steps are actually supported.

Architecture lab (中文) · Interactive summary and all curves · Methods · Reproduce · Leaderboard

These artifacts come from different methods and protocols, not one unified algorithm. Each final artifact follows its campaign selection rule; cross-campaign ranking and retrospective peaks use observed tests. Paper budgets, repeats, and Telecom simulator/access differ. The full leaderboard retains ALIGN, AgentSquare, Meta-Harness, AHE, and historical Life-Harness versions.

• More loop machinery did not reliably improve results. L4/L5 remain strong; L8 delivered experience without consistent gains, and L9 did not dominate the earlier checkpoints.

• Protocol changes are additional variables. DB/Web L4 use 300 training tasks and no separate validation; L5 uses 240+60. Their score difference is not a clean component ablation.

• Selection leaves headroom. Web retrospective 47.00% exceeds that campaign final 44.33%; OS early 39.35% exceeds final 38.89%. Original selections are preserved.

• Overall gain and mechanism support are separate. Loading, triggering, and experience delivery are execution facts; intended repairs and global outcome changes require additional evidence.

• Mean success and reliability can diverge. OS mechanism feedback reached 32.87% versus summary 29.40%, yet all-three success was 25.00% versus 26.39%. This is a single-search observation.

evolution/ supplies shared three-benchmark L1–L9 code, 36 profiles, task IDs and a small offline example. Each method documents inputs → edits → evaluation → adoption → final selection, with explicit reproduction coverage.

python -m pip install -e evolution
dsh-evolve profiles
python evolution/scripts/smoke.py ./workspaces/smoke

This scripted-model example exercises real DSH hooks and one adoption cycle; it does not remeasure benchmark scores. Formal runs require models, data and environments. ALFWorld/Telecom and feedback-extension integration boundaries are documented.

Round records and detailed statistics · Metric definitions · All artifacts · Publishing

Training, validation and test measurements remain separate. Inspect repeats, valid coverage, adoption/regression, models and splits. Time fields state their scope; missing costs stay null; three test trials are not three independent searches. New measured records rebuild the site, which shows a collection date rather than pretending to be a live monitor.

MIT · Third-party notices · DSH