fix(llma): two $ai_generation fidelity bugs in the Claude Code session ingest - #170
Open
josephruocco wants to merge 2 commits into
Open
fix(llma): two $ai_generation fidelity bugs in the Claude Code session ingest#170josephruocco wants to merge 2 commits into
josephruocco wants to merge 2 commits into
Conversation
build_ai_generation emitted cache_read_input_tokens and cache_creation_input_tokens without the $ai_ prefix that every other property in that dict carries. PostHog's LLM Analytics pipeline reads $ai_cache_read_input_tokens and $ai_cache_creation_input_tokens when computing cost; unprefixed keys are ingested as ordinary custom properties and ignored by cost calculation. For prompt-cached workloads this silently understates spend by a large factor. Reported in PostHog#92: cache_read=150109, cache_write=75729, input=4, output=926 costed at ~$0.014 against an actual ~$0.343. Property names verified against posthog-python (posthog/ai/utils.py), which also always emits both fields for the anthropic provider even at zero -- that behaviour is preserved here. test_cache_tokens asserted the unprefixed names, so it encoded the bug rather than catching it. Updated, plus two regression tests covering the namespace and the always-emit-at-zero case. Fixes PostHog#92 (bug 1)
_finalize_generation appended thinking and text blocks into one
text_parts list, and build_events then wrapped the joined result in a
single {"type": "text"} block. Extended thinking therefore arrived in
PostHog as ordinary assistant output: it renders inline with the answer
and cannot be filtered on.
_finalize_generation now also builds output_blocks, a structured list
that preserves each block's type in first-seen order, and build_events
prefers it. output_text keeps the flattened join, so nothing downstream
of it changes, and build_events falls back to the old single-text-block
path when output_blocks is absent (older parsed payloads).
Block shape matches posthog-python: {"type": "thinking", "thinking": ...}
(posthog/ai/anthropic/anthropic_converter.py).
test_thinking_preserved_when_split_across_chunks asserted that thinking
content came back as a text block -- the buggy behaviour -- so it is
updated to assert the typed block instead. Two further tests cover
mixed thinking/text output and block ordering.
Fixes PostHog#92 (bug 2)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #92. Both bugs are still on main as of 257a7d5. One commit each.
Cache tokens missing the
$ai_prefixbuild_ai_generationemits these two unprefixed while every other key in the dict is namespaced:They land as ordinary custom properties, so cost calculation ignores them.
I checked the expected names against posthog-python rather than taking them from the issue. Both are in
_TOKEN_PROPERTY_KEYSinposthog/ai/utils.py, and around line 754 the anthropic provider always emits them even at zero, unlike the other providers. Kept that behaviour here since this builder only ever runs for anthropic.Thinking blocks arriving as assistant text
_finalize_generationappended thinking and text into the sametext_partslist, andbuild_eventswrapped the joined result in a single{"type": "text"}block. Reasoning ends up indistinguishable from the actual answer in$ai_output_choices.Added
output_blocksalongsideoutput_text, keeping each block's type in the order they first appear:output_textstill holds the flattened join, so nothing reading it changes, and theelifkeeps older parsed payloads working. Block shape matchesanthropic_converter.py.Note on the test diff
Two passing tests asserted the old behaviour, so the diff touches tests that were previously green:
test_cache_tokenschecked the unprefixed keytest_thinking_preserved_when_split_across_chunksexpected thinking to come back as a text blockBoth updated. Probably part of why these lasted as long as they did.
Testing
89 passing, up from 86. Ran both commands from
tests.ymllocally:The two commits are independent, so happy to split this into separate PRs or drop the second one if you'd rather take the prefix fix on its own.