Repository navigation
v1.1.0
What's new
Worktree dependency setup
Worktrees created by git worktree add only contain tracked files — dependencies like node_modules or venv are absent. Experiments can now declare a project.setup_command in experiment.yaml that runs in each worktree before the agent starts:
project:
setup_command: "pnpm install --frozen-lockfile"An optional project.setup_timeout (defaults to verification timeout) prevents hung installs from blocking the experiment.
Quick iteration with --runs
No more editing experiment.yaml to do a smoke test. Override runs per condition from the CLI:
bash framework/orchestrator.sh experiments/001-claude-md-effect/ --runs 1The report now shows actual vs configured count (e.g., Runs/condition | 2 (config: 5)) so partial runs are transparent.
Cost estimation
Reports now include estimated USD cost per run and per condition, computed from official Claude API pricing. Token breakdown covers input, output, cache read, and cache creation separately. Pricing lives in framework/pricing.json for easy updates.
Fixes
- Agent SIGTTIN hang — the
timeoutcommand creates a new process group, causingclaudeto lose terminal access. Any stdin read during init stopped the process permanently with zero output. Fixed by redirecting stdin from/dev/null. - Report per_condition mismatch — the report displayed the YAML config value even when
--runsoverrode it. Now derived from actual metrics data. - Setup errors silent — project setup previously swallowed stderr. Errors are now visible for debugging.
Internal improvements
- Report generator extracted from a 390-line bash heredoc into standalone
report_generator.pywithpricing.json - Agent and verification commands wrapped in configurable
timeout(defaults: agent 3600s, verification 600s) - Orchestrator uses proper option parsing instead of positional args
- Creating-experiment skill updated with setup_command interview question and template field
- CLAUDE.md artifact for typescript-eslint experiment simplified and reorganized