Skip to content

fix(llma): two $ai_generation fidelity bugs in the Claude Code session ingest - #170

Open
josephruocco wants to merge 2 commits into
PostHog:mainfrom
josephruocco:fix/ai-generation-fidelity
Open

fix(llma): two $ai_generation fidelity bugs in the Claude Code session ingest#170
josephruocco wants to merge 2 commits into
PostHog:mainfrom
josephruocco:fix/ai-generation-fidelity

Conversation

@josephruocco

Copy link
Copy Markdown

Fixes #92. Both bugs are still on main as of 257a7d5. One commit each.

Cache tokens missing the $ai_ prefix

build_ai_generation emits these two unprefixed while every other key in the dict is namespaced:

"$ai_project_name": project_name,
"$ai_agent_name": agent_name,
"cache_read_input_tokens": cache_read_tokens,
"cache_creation_input_tokens": cache_creation_tokens,

They land as ordinary custom properties, so cost calculation ignores them.

I checked the expected names against posthog-python rather than taking them from the issue. Both are in _TOKEN_PROPERTY_KEYS in posthog/ai/utils.py, and around line 754 the anthropic provider always emits them even at zero, unlike the other providers. Kept that behaviour here since this builder only ever runs for anthropic.

Thinking blocks arriving as assistant text

_finalize_generation appended thinking and text into the same text_parts list, and build_events wrapped the joined result in a single {"type": "text"} block. Reasoning ends up indistinguishable from the actual answer in $ai_output_choices.

Added output_blocks alongside output_text, keeping each block's type in the order they first appear:

if gen.get("output_blocks"):
    content_blocks.extend(gen["output_blocks"])
elif gen["output_text"]:
    content_blocks.append({"type": "text", "text": gen["output_text"]})

output_text still holds the flattened join, so nothing reading it changes, and the elif keeps older parsed payloads working. Block shape matches anthropic_converter.py.

Note on the test diff

Two passing tests asserted the old behaviour, so the diff touches tests that were previously green:

  • test_cache_tokens checked the unprefixed key
  • test_thinking_preserved_when_split_across_chunks expected thinking to come back as a text block

Both updated. Probably part of why these lasted as long as they did.

Testing

89 passing, up from 86. Ran both commands from tests.yml locally:

python3 -m pytest tests/test_session_parser.py tests/test_posthog_llma.py
./tests/test_gate_exec_write.sh

The two commits are independent, so happy to split this into separate PRs or drop the second one if you'd rather take the prefix fix on its own.

build_ai_generation emitted cache_read_input_tokens and
cache_creation_input_tokens without the $ai_ prefix that every other
property in that dict carries. PostHog's LLM Analytics pipeline reads
$ai_cache_read_input_tokens and $ai_cache_creation_input_tokens when
computing cost; unprefixed keys are ingested as ordinary custom
properties and ignored by cost calculation.

For prompt-cached workloads this silently understates spend by a large
factor. Reported in PostHog#92: cache_read=150109, cache_write=75729, input=4,
output=926 costed at ~$0.014 against an actual ~$0.343.

Property names verified against posthog-python (posthog/ai/utils.py),
which also always emits both fields for the anthropic provider even at
zero -- that behaviour is preserved here.

test_cache_tokens asserted the unprefixed names, so it encoded the bug
rather than catching it. Updated, plus two regression tests covering the
namespace and the always-emit-at-zero case.

Fixes PostHog#92 (bug 1)
_finalize_generation appended thinking and text blocks into one
text_parts list, and build_events then wrapped the joined result in a
single {"type": "text"} block. Extended thinking therefore arrived in
PostHog as ordinary assistant output: it renders inline with the answer
and cannot be filtered on.

_finalize_generation now also builds output_blocks, a structured list
that preserves each block's type in first-seen order, and build_events
prefers it. output_text keeps the flattened join, so nothing downstream
of it changes, and build_events falls back to the old single-text-block
path when output_blocks is absent (older parsed payloads).

Block shape matches posthog-python: {"type": "thinking", "thinking": ...}
(posthog/ai/anthropic/anthropic_converter.py).

test_thinking_preserved_when_split_across_chunks asserted that thinking
content came back as a text block -- the buggy behaviour -- so it is
updated to assert the typed block instead. Two further tests cover
mixed thinking/text output and block ordering.

Fixes PostHog#92 (bug 2)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Two $ai_generation fidelity bugs in the Claude Code session ingest

1 participant