Repository navigation
v0.8.0: Token and wall-clock tracking
Token & wall-clock tracking, raw-rate scoring, and a unified compare view (#46, closes #45).
Token & wall-clock usage
- Per-attempt
TokenUsage(disjoint input/output/cache fields + computed total) and wall-clock time, captured across claude-code, codex, pi, hermes; malformed payloads degrade tonullrather than crashing a run. reportshows per-task Tokens/Wall + a per-run cost line;--baselineshows the skill-vs-bare-agent token/wall delta. No dollar cost tracked (deliberate).
Metrics: raw success rate is now primary
- The headline is the raw per-attempt success rate (
successes / usable) — how often a single run works. - pass@k and pass^k are kept as secondary views (in JSON as
pass_at_k/pass_hat_k, and under--verbose), with docs on when to use each. - Breaking (JSON):
AggregateScore.avg_pass_at_k→avg_score;TaskResultgainsscore/usable/pass_hat_k. Old results JSON still loads.
Report & compare
--baselinenow renders through the samecompareview (one diff path; inherits regression detection).- Comparison tables read
before → after(no A/B legend); dropped the positionaltask_idcolumn.