Skip to content

fix(llmcore): record Claude messages-API output_tokens instead of zero - #761

Open
Kailigithub wants to merge 1 commit into
lsdefine:mainfrom
Kailigithub:optimize/llmcore-messages-out-tokens
Open

fix(llmcore): record Claude messages-API output_tokens instead of zero#761
Kailigithub wants to merge 1 commit into
lsdefine:mainfrom
Kailigithub:optimize/llmcore-messages-out-tokens

Conversation

@Kailigithub

Copy link
Copy Markdown
Contributor

Summary

llmcore._record_usage(usage, api_mode='messages') hard-coded out = 0. As a result:

  • STATS['out'] was never updated for Claude (Anthropic Messages API) sessions — the user-facing cost / running-stats display stayed stale.
  • The [Output] tokens=... log line never printed for Claude sessions, while the chat_completions and responses branches both emit it.

This PR reads usage['output_tokens'] through the same _i(...) helper the other branches already use (which coerces None and non-int values to 0) and adds the missing log line so the three branches stay symmetric.

Diff

--- a/llmcore.py
+++ b/llmcore.py
@@ -375,8 +375,9 @@ def _record_usage(usage, api_mode):
         if out: print(f"[Output] tokens={out}")
     elif api_mode == 'messages':
         ci, cr, raw_inp = _i(usage.get("cache_creation_input_tokens")), _i(usage.get("cache_read_input_tokens")), _i(usage.get("input_tokens"))
-        inp, cached, out = raw_inp + ci + cr, cr, 0
+        inp, cached, out = raw_inp + ci + cr, cr, _i(usage.get("output_tokens"))
         print(f"[Cache] input={raw_inp} creation={ci} read={cr}")
+        if out: print(f"[Output] tokens={out}")
     else: return
     STATS.update(inp=inp, cached=cached, out=out)

2 lines added, 1 changed. No behavior change on existing inputs except that Claude output_tokens are now counted.

Tests

tests/test_llmcore_record_usage_messages.py covers:

  1. messages happy path — STATS['out'] populated, [Output] line emitted.
  2. messages with output_tokens key absent — degrades to 0, no log line.
  3. messages with output_tokens: null — _i coerces to 0 (consistent with the other branches).
  4. chat_completions regression — untouched, still correct.
  5. responses regression — untouched, still correct.

The regression tests for the other branches are intentional: _record_usage is shared, and the diff is small enough that a single typo could break all three modes. Keeping a 5-case suite means future refactors that touch one branch can't silently regress the others.

Verification

python tests/test_llmcore_record_usage_messages.py
# PASS test_messages_mode_records_output_tokens
# PASS test_messages_mode_handles_missing_output_tokens
# PASS test_messages_mode_handles_null_output_tokens
# PASS test_chat_completions_mode_still_works
# PASS test_responses_mode_still_works
# ALL PASS

# Same suite against the unfixed llmcore.py (git checkout main -- llmcore.py):
# FAIL test_messages_mode_records_output_tokens: messages API: STATS['out'] not updated from output_tokens; got 0
# PASS test_messages_mode_handles_missing_output_tokens
# PASS test_messages_mode_handles_null_output_tokens
# PASS test_chat_completions_mode_still_works
# PASS test_responses_mode_still_works

This PR is the 'optimize' branch from a saturated-repo 3-bucket cron run; it pairs with #760 (fix(fsapp) #685) for bug-fix coverage on the same tick.

_record_usage(usage, api_mode='messages') hard-coded out = 0, so STATS['out']
was never updated for Claude (Anthropic Messages API) sessions and the
'[Output] tokens=...' log line never printed. Claude's usage dict emits
output_tokens just like the OpenAI chat_completions and responses branches
already do, so the messages branch should pick it up too.

Read usage['output_tokens'] through the same _i(...) helper that coerces
None and non-int values to 0, and add the missing [Output] log line so the
three branches stay symmetric.

Adds tests/test_llmcore_record_usage_messages.py with 5 cases covering
happy path, missing key, null value, and regression checks on the
chat_completions and responses branches.

Verified by reverting llmcore.py to upstream/main: tests pass on the
fix and the happy-path messages case fails on the unfixed code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant