Skip to content

v1.1.1

Latest

Choose a tag to compare

@sergeyklay sergeyklay released this 04 Mar 22:56

First experiment results

Experiment 001 (CLAUDE.md effect) ran to completion - 5 runs per condition on claude-haiku-4-5 against the typescript-eslint codebase. Results are now tracked in the repo alongside experiment configs.

Headline numbers: hand-curated CLAUDE.md improved pass@1 from 60% to 100%, cut duration by 15%, and reduced token usage by 13%. Effect sizes are large (Cohen's d around -0.85 to -1.0), though N=5 means confidence intervals still overlap. Full report in experiments/001-claude-md-effect/results/report.md.

Results tracking changed: .gitignore no longer blanket-ignores results/ directories. Metrics, diffs, reports, and summaries are now committed. Only session.json files (400-800KB agent logs) stay ignored.


Fixes

  • Report median was wrong: the Duration (median) row displayed the average for every condition except the first, due to a broken lambda that only resolved against conditions[0]. Median is now computed directly per condition, matching how the CI row already worked.
  • Dead code in row(): removed a conditional where both branches did the same thing (avg if "avg" not in label else avg) and an unused bold parameter that wrapped labels in ** nobody consumed.

Housekeeping

  • Added a task backlog (.tasks/) with 8 items tracking planned framework improvements, using YAML frontmatter for metadata
  • Fixed step numbering in the experiment creation skill - rounds now number from 1 instead of continuing a global counter across all rounds
  • Applied ruff formatting across report_generator.py
  • Cleaned up project documentation