Skip to content

v0.8.0: Token and wall-clock tracking

Choose a tag to compare

@edonadei edonadei released this 05 Jul 20:21
· 240 commits to main since this release

Token & wall-clock tracking, raw-rate scoring, and a unified compare view (#46, closes #45).

Token & wall-clock usage

  • Per-attempt TokenUsage (disjoint input/output/cache fields + computed total) and wall-clock time, captured across claude-code, codex, pi, hermes; malformed payloads degrade to null rather than crashing a run.
  • report shows per-task Tokens/Wall + a per-run cost line; --baseline shows the skill-vs-bare-agent token/wall delta. No dollar cost tracked (deliberate).

Metrics: raw success rate is now primary

  • The headline is the raw per-attempt success rate (successes / usable) — how often a single run works.
  • pass@k and pass^k are kept as secondary views (in JSON as pass_at_k/pass_hat_k, and under --verbose), with docs on when to use each.
  • Breaking (JSON): AggregateScore.avg_pass_at_k → avg_score; TaskResult gains score/usable/pass_hat_k. Old results JSON still loads.

Report & compare

  • --baseline now renders through the same compare view (one diff path; inherits regression detection).
  • Comparison tables read before → after (no A/B legend); dropped the positional task_id column.