Skip to content

v0.2.0 — active time metric + benchmark

Choose a tag to compare

@mertdotdev mertdotdev released this 20 Jul 14:29

Added

  • agentxray benchmark [dir] — runs agentxray across every transcript and prints aggregate findings, so the numbers in the README are checkable rather than trusted. Aggregates only: no file paths, prompts or tool inputs leave your machine.
  • SessionStats.activeMs — time the agent actually spent working (sum of gaps under 5 minutes).

Fixed

  • stats reported first-event-to-last-event wall clock as session duration. On resumed transcripts this was wildly wrong — across the benchmark set it overstated agent time by 41× (1,455 h reported vs 35.6 h actual). stats now leads with active time and shows the span only when the two differ:

    session  2156 events, 1291 assistant turns, 7h 27m active (spanning 246h 28m)
    

    Found by writing the benchmark, not by a bug report.

  • Durations over an hour rendered as inflated minute counts (14751m 29s); they now render as 246h 28m.

New benchmark

56 sessions, 9 projects, 20 days: 1 in 7 tool calls was a failure or a repeat. Full method and caveats: docs/BENCHMARK.md.

Reproduce on your own machine:

npx agentxray@latest benchmark