Skip to content

Unify CalculateTokenUsage#509

Merged
squishykid merged 5 commits intomainfrom
gtrrz-victor/cleanup-token-calculation
Feb 26, 2026
Merged

Unify CalculateTokenUsage#509
squishykid merged 5 commits intomainfrom
gtrrz-victor/cleanup-token-calculation

Conversation

@gtrrz-victor
Copy link
Contributor

Summary

  • Delete duplicated token usage calculation logic that was scattered across condensation, lifecycle, and hook code
  • Centralize agent dispatch into a single agent.CalculateTokenUsage() function
  • Agent token calculators now accept []byte instead of file paths, letting callers own the file I/O

🤖 Generated with Claude Code

@gtrrz-victor gtrrz-victor requested a review from a team as a code owner February 26, 2026 03:00
Copilot AI review requested due to automatic review settings February 26, 2026 03:00
@cursor
Copy link

cursor bot commented Feb 26, 2026

PR Summary

Medium Risk
Touches core checkpoint/condensation paths and token accounting across multiple agent integrations; risk is mainly behavioral regressions in offset handling or token/subagent aggregation rather than data safety/security.

Overview
Refactors agent token accounting to be bytes-first: TokenCalculator/SubagentAwareExtractor now take transcriptData []byte and callers own transcript file I/O.

Introduces agent.CalculateTokenUsage() to centralize dispatch (preferring subagent-aware extraction), and updates lifecycle + manual-commit condensation/hooks to use it; Claude Code’s subagent file/token extraction now parses the main transcript from bytes (slicing by offset) and Gemini/OpenCode token calculators are moved to agent methods instead of file-based helpers.

Simplifies transcript.ParseFromFileAtLine to return only parsed lines (no total count) and adjusts all downstream call sites/tests accordingly.

Written by Cursor Bugbot for commit e0119c1. Configure here.

@gtrrz-victor gtrrz-victor changed the title Unify CalculateTokenUsage to accept []byte instead of file paths Unify CalculateTokenUsage Feb 26, 2026
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR centralizes token usage calculation behind agent.CalculateTokenUsage(...) and updates agent token calculators/extractors to accept transcript []byte instead of file paths, shifting transcript file I/O responsibilities to callers.

Changes:

  • Updated transcript parsing (ParseFromFileAtLine) to return only parsed lines (removed totalLines return value) and adjusted call sites/tests.
  • Refactored agent interfaces (TokenCalculator, SubagentAwareExtractor) to accept transcript bytes, and updated Claude Code / Gemini CLI / OpenCode implementations and tests accordingly.
  • Added a shared agent.CalculateTokenUsage(...) dispatcher and integrated it into lifecycle + condensation paths.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
cmd/entire/cli/transcript/parse.go Changes ParseFromFileAtLine signature to return only ([]Line, error).
cmd/entire/cli/transcript/parse_test.go Updates tests to match new parse signature.
cmd/entire/cli/strategy/common.go Updates transcript preparation helper to accept an agent.Agent instance.
cmd/entire/cli/strategy/manual_commit_hooks.go Adjusts Claude Code modified-file extraction to pass transcript bytes.
cmd/entire/cli/strategy/manual_commit_condensation.go Removes duplicated token-usage logic and routes through agent.CalculateTokenUsage(...).
cmd/entire/cli/strategy/manual_commit_condensation_test.go Updates tests to resolve agents via registry and use agent.CalculateTokenUsage(...).
cmd/entire/cli/lifecycle.go Switches token usage + subagent extraction paths to pass transcript bytes and use centralized dispatcher.
cmd/entire/cli/agent/token_usage.go Introduces centralized token-usage dispatch helper.
cmd/entire/cli/agent/agent.go Updates token-related interfaces to accept transcript bytes.
cmd/entire/cli/agent/claudecode/claude.go Updates Claude Code subagent-aware extractor methods to accept transcript bytes.
cmd/entire/cli/agent/claudecode/lifecycle.go Updates Claude Code token calculator to accept transcript bytes.
cmd/entire/cli/agent/claudecode/transcript.go Refactors Claude Code token/subagent helpers to parse from bytes and slice by line.
cmd/entire/cli/agent/claudecode/transcript_test.go Updates tests to use transcript bytes and adds a JSONL builder helper.
cmd/entire/cli/agent/geminicli/lifecycle.go Implements Gemini token calculation directly from transcript bytes.
cmd/entire/cli/agent/geminicli/transcript.go Removes old package-level/file-based Gemini token usage helpers.
cmd/entire/cli/agent/geminicli/transcript_test.go Updates Gemini tests to use the agent method returning (usage, error).
cmd/entire/cli/agent/opencode/transcript.go Updates OpenCode token calculator to accept transcript bytes.
cmd/entire/cli/agent/opencode/transcript_test.go Updates OpenCode tests to use transcript bytes and replaces nonexistent-file test with empty-data test.

Prevents constructing relative paths like "agent-xyz.jsonl" that would
read from the process working directory.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: 782908946233
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable autofix in the Cursor dashboard.

Comment @cursor review or bugbot run to trigger another review on this PR

…nup-token-calculation

# Conflicts:
#	cmd/entire/cli/agent/geminicli/lifecycle.go
#	cmd/entire/cli/strategy/common.go
#	cmd/entire/cli/strategy/manual_commit_condensation.go

Entire-Checkpoint: 45003cf5a34f
@squishykid squishykid merged commit fec13da into main Feb 26, 2026
3 checks passed
@squishykid squishykid deleted the gtrrz-victor/cleanup-token-calculation branch February 26, 2026 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants