Skip to content

v1.1.0

Choose a tag to compare

@sergeyklay sergeyklay released this 04 Mar 12:50
· 6 commits to main since this release

What's new

Worktree dependency setup

Worktrees created by git worktree add only contain tracked files — dependencies like node_modules or venv are absent. Experiments can now declare a project.setup_command in experiment.yaml that runs in each worktree before the agent starts:

project:
  setup_command: "pnpm install --frozen-lockfile"

An optional project.setup_timeout (defaults to verification timeout) prevents hung installs from blocking the experiment.

Quick iteration with --runs

No more editing experiment.yaml to do a smoke test. Override runs per condition from the CLI:

bash framework/orchestrator.sh experiments/001-claude-md-effect/ --runs 1

The report now shows actual vs configured count (e.g., Runs/condition | 2 (config: 5)) so partial runs are transparent.

Cost estimation

Reports now include estimated USD cost per run and per condition, computed from official Claude API pricing. Token breakdown covers input, output, cache read, and cache creation separately. Pricing lives in framework/pricing.json for easy updates.


Fixes

  • Agent SIGTTIN hang — the timeout command creates a new process group, causing claude to lose terminal access. Any stdin read during init stopped the process permanently with zero output. Fixed by redirecting stdin from /dev/null.
  • Report per_condition mismatch — the report displayed the YAML config value even when --runs overrode it. Now derived from actual metrics data.
  • Setup errors silent — project setup previously swallowed stderr. Errors are now visible for debugging.

Internal improvements

  • Report generator extracted from a 390-line bash heredoc into standalone report_generator.py with pricing.json
  • Agent and verification commands wrapped in configurable timeout (defaults: agent 3600s, verification 600s)
  • Orchestrator uses proper option parsing instead of positional args
  • Creating-experiment skill updated with setup_command interview question and template field
  • CLAUDE.md artifact for typescript-eslint experiment simplified and reorganized