Repository navigation
First experiment results
Experiment 001 (CLAUDE.md effect) ran to completion - 5 runs per condition on claude-haiku-4-5 against the typescript-eslint codebase. Results are now tracked in the repo alongside experiment configs.
Headline numbers: hand-curated CLAUDE.md improved pass@1 from 60% to 100%, cut duration by 15%, and reduced token usage by 13%. Effect sizes are large (Cohen's d around -0.85 to -1.0), though N=5 means confidence intervals still overlap. Full report in experiments/001-claude-md-effect/results/report.md.
Results tracking changed: .gitignore no longer blanket-ignores results/ directories. Metrics, diffs, reports, and summaries are now committed. Only session.json files (400-800KB agent logs) stay ignored.
Fixes
- Report median was wrong: the Duration (median) row displayed the average for every condition except the first, due to a broken lambda that only resolved against
conditions[0]. Median is now computed directly per condition, matching how the CI row already worked. - Dead code in row(): removed a conditional where both branches did the same thing (
avg if "avg" not in label else avg) and an unusedboldparameter that wrapped labels in**nobody consumed.
Housekeeping
- Added a task backlog (
.tasks/) with 8 items tracking planned framework improvements, using YAML frontmatter for metadata - Fixed step numbering in the experiment creation skill - rounds now number from 1 instead of continuing a global counter across all rounds
- Applied ruff formatting across
report_generator.py - Cleaned up project documentation