Problem
Every command loads every run of every transcript into memory before doing anything.
def parse_all(root: Path | None = None) -> list[Session]:
return [parse_session(p) for p in find_sessions(root)]
def _load(args) -> list[AgentRun]:
...
runs = [r for s in sessions for r in s.runs]
And parse_session itself buffers the whole file's worth of state:
uses: dict[str, dict] = {}
results: dict[str, dict] = {}
results holds the full content of every tool_result in the transcript, including results from tools that are not Agent delegations, because the elif btype == "tool_result" branch stores everything and only later checks which ids matter.
The module docstring cites a real 34MB session with 96 Agent invocations. find_sessions globs **/*.jsonl under ~/.claude/projects, which for a regular Claude Code user is dozens to hundreds of such files. agentrace stats over that whole directory materializes every prompt and every result of every run in every session at once, in order to print nine aggregate numbers.
Why it matters
stats needs sums. list needs one row per run. Neither needs the prompt and result bodies retained. Only show needs the full text, and it needs it for exactly one run. The current design pays the worst case cost for all four commands, and it scales with total transcript history rather than with what was asked for.
Suggested approach
- Fix the cheap part first: in
parse_session, only retain a tool_result if its tool_use_id is a known Agent use, or do the filtering in the second phase. Right now every unrelated tool result in the file is held in memory for the duration.
- Make
parse_all a generator (Iterator[Session]) so sessions are processed and released one at a time instead of accumulated. _load returning a list is fine for list and show, but stats and check should be able to consume a stream.
- Give
AgentRun a lightweight mode, or add a summary dataclass carrying only tool_use_id, description, timestamps, is_error, background, and the two character counts. list and stats use that; show re-parses the one file it needs.
- Measure before and after with a synthetic directory of large transcripts, and post the numbers on the PR. A short script under
tests/ that generates them is welcome.
Done when
agentrace stats over a large transcript directory has memory use roughly independent of transcript size.
- Non-Agent tool results are never retained.
- The improvement is demonstrated with measured numbers.
If you want to take this on, comment on the issue to claim it and it will be assigned. Please keep to a maximum of 2 open claims per person at a time so other contributors get a chance.
Problem
Every command loads every run of every transcript into memory before doing anything.
And
parse_sessionitself buffers the whole file's worth of state:resultsholds the fullcontentof everytool_resultin the transcript, including results from tools that are not Agent delegations, because theelif btype == "tool_result"branch stores everything and only later checks which ids matter.The module docstring cites a real 34MB session with 96 Agent invocations.
find_sessionsglobs**/*.jsonlunder~/.claude/projects, which for a regular Claude Code user is dozens to hundreds of such files.agentrace statsover that whole directory materializes every prompt and every result of every run in every session at once, in order to print nine aggregate numbers.Why it matters
statsneeds sums.listneeds one row per run. Neither needs the prompt and result bodies retained. Onlyshowneeds the full text, and it needs it for exactly one run. The current design pays the worst case cost for all four commands, and it scales with total transcript history rather than with what was asked for.Suggested approach
parse_session, only retain atool_resultif itstool_use_idis a known Agent use, or do the filtering in the second phase. Right now every unrelated tool result in the file is held in memory for the duration.parse_alla generator (Iterator[Session]) so sessions are processed and released one at a time instead of accumulated._loadreturning a list is fine forlistandshow, butstatsandcheckshould be able to consume a stream.AgentRuna lightweight mode, or add a summary dataclass carrying onlytool_use_id,description, timestamps,is_error,background, and the two character counts.listandstatsuse that;showre-parses the one file it needs.tests/that generates them is welcome.Done when
agentrace statsover a large transcript directory has memory use roughly independent of transcript size.If you want to take this on, comment on the issue to claim it and it will be assigned. Please keep to a maximum of 2 open claims per person at a time so other contributors get a chance.