fix(codebuddy): count function_call usage, fix ~60% token undercount

[HandSonic]

Summary

The CodeBuddy jsonl parser only aggregates type=="message" && role=="assistant" records, but every LLM round-trip that ends in a tool call is recorded as type=="function_call" with full providerData.rawUsage. On a real install those records are ~93% of all round-trips and carry ~14x the assistant-message token volume — the parser silently drops them.

Verified on real ~/.codebuddy data (109 projects, 2,245 jsonl files, 33k deduped LLM round-trips):

Metric	Before (main)	After (this PR)	Ground truth
net input	66.5M	172.8M	172.2M
output	5.3M	9.6M	9.5M
cache read	806M	2.42B	2.37B

(Residual <0.1% is incremental-scan timing — new sessions written between the two scans.)

The WorkBuddy reader already aggregates every usage-carrying record for exactly this reason (rollout.js WorkBuddy header comment: "Usage lives on function_call records too"). CodeBuddy uses the identical transcript format — the fix was simply never ported back.

Root cause

// rollout.js (main): drops 93% of LLM round-trips
if (!entry || entry.type !== "message" || entry.role !== "assistant") continue;

A single user "turn" in CodeBuddy is many LLM round-trips: model → tool call → tool result → model → … → final text. Only the final text reply is a message record; every intermediate round-trip is a function_call record with its own providerData.rawUsage.

Fix

1. Aggregate ANY record with providerData.rawUsage, regardless of record type (mirrors the WorkBuddy reader).
2. Dedup per round-trip via providerData.messageId (the response-level id shared by the function_call/message pair of the same round-trip) before falling back to entry.uuid/entry.id. Per-record ids would count a round-trip once per record type.
3. reasoning_tokens from completion_tokens_details (prompt-side is always 0 on real data), subtracted from completion_tokens to avoid double-counting output.
4. Cache split also reads prompt_cache_hit_tokens / prompt_cache_write_tokens (DeepSeek-style mirrors present in real rawUsage); net input subtracts both cache reads and cache writes.

Note on the trace-based approach (#399)

Closed #399 in favor of this. Real-data verification showed trace.modelInfo covers only ~64% of a session's jsonl usage (17%–100% per session — one trace per agent workflow, while sessions accumulate many workflows + sub-agent round-trips), so traces cannot be the authoritative source. The "90% of messages lack rawUsage" observation from #399 was correct, but the usage isn't missing — it lives on function_call records, which this PR reads directly. No trace files needed.

Tests

• test/rollout-parser.test.js: 229 pass / 2 fail — both pre-existing on main (#141 codebuddy resolver fixture, #154 kiro), unrelated to this change
• Real-data verification: parser output matches an independent jsonl-only ground-truth scan within 0.1%

Scope

• CLI (src/lib/rollout.js — parseCodebuddyIncremental only; WorkBuddy untouched)

Summary by CodeRabbit

• Bug Fixes
  • Improved usage tracking for function-call and other response records.
  • Prevented duplicate usage entries by consistently recognizing shared response identifiers.
  • Improved token reporting, including cached reads, cached writes, reasoning tokens, and completion output.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: d4436b21-ae0b-4187-86dc-d9c7c15d9964

📥 Commits

Reviewing files that changed from the base of the PR and between 79a1e86 and 4ea3347.

📒 Files selected for processing (1)

• src/lib/rollout.js

🚧 Files skipped from review as they are similar to previous changes (1)

• src/lib/rollout.js

---

📝 Walkthrough

Walkthrough

CodeBuddy parsing now includes function-call records with raw usage. It uses response-level message IDs for deduplication and normalizes cache read, cache write, reasoning, prompt, and completion tokens.

Changes

CodeBuddy usage parsing

Layer / File(s)	Summary
Raw usage aggregation and token normalization src/lib/rollout.js	The parser processes object records with providerData.rawUsage, prioritizes providerData.messageId, combines alternate cache fields, removes cache tokens from prompt tokens, and separates reasoning tokens from completion tokens.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related issues

• #403: The changes address CodeBuddy parser handling, deduplication, cache-token normalization, and reasoning-token accounting.

Possibly related PRs

• xiufengsun/TokenTracker#399: Both changes update CodeBuddy parsing in src/lib/rollout.js, including raw usage handling and token normalization.

Suggested reviewers: xiufengsun

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	The PR improves JSONL usage parsing but does not implement the linked issue's trace scanning, trace deduplication, WorkBuddy, or extension-log requirements [#399].	Implement the omitted trace, deduplication, WorkBuddy, and extension-log requirements, or update the linked issue to reflect the reduced scope.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the CodeBuddy function-call usage and token undercount fix.
Out of Scope Changes check	✅ Passed	The changes remain within CodeBuddy usage parsing and token accounting, with no unrelated changes evident.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

src/lib/rollout.js (1)

8203-8218: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Use a shared dedup path for CodeBuddy round-trips.

parseCodebuddyIncremental() builds its own provider-message/fallback dedup key here. Route the stable response-level provider.messageId fallback sequence through the parser dedup helper, or extend that helper to accept this CodeBuddy key shape before creating the local key.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/rollout.js` around lines 8203 - 8218, The local dedup-key
construction in parseCodebuddyIncremental must use the shared parser dedup path
instead of independently selecting provider.messageId, entry.uuid, entry.id, and
the timestamp fallback. Update the relevant dedup helper or extend it to accept
CodeBuddy’s key shape, then have parseCodebuddyIncremental reuse that helper
while preserving provider.messageId as the preferred stable round-trip key.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/rollout.js`:
- Around line 8250-8251: Update the completion-token filtering logic around
reasoningTokens and completionTokens so records with reasoning-only usage are
retained: the zero-token discard guard must also consider reasoningTokens before
marking a record as empty. Preserve discarding records where both
completionTokens and reasoningTokens are zero.

---

Nitpick comments:
In `@src/lib/rollout.js`:
- Around line 8203-8218: The local dedup-key construction in
parseCodebuddyIncremental must use the shared parser dedup path instead of
independently selecting provider.messageId, entry.uuid, entry.id, and the
timestamp fallback. Update the relevant dedup helper or extend it to accept
CodeBuddy’s key shape, then have parseCodebuddyIncremental reuse that helper
while preserving provider.messageId as the preferred stable round-trip key.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 34c32ce9-1002-4f60-8219-f6015fc14f5a

📥 Commits

Reviewing files that changed from the base of the PR and between f54ee17 and 79a1e86.

📒 Files selected for processing (1)

• src/lib/rollout.js

[xiufengsun]

Reviewed exact head 4ea3347 against main f54ee17. Parser suite 231/231 and an independent function_call/cache/reasoning/dedup fixture pass; no blocking review threads remain.