feat(replay-vision): capture gemini reasoning and group scan traces in llma

[fivestarspicy]

Problem

Replay Vision's scanner runs are instrumented with LLM analytics, but two things keep the traces from being useful for debugging scans:

• No reasoning visible. The Gemini calls never set thinking_config, so the model's thought summaries are never returned or captured — you can see reasoning token counts on generations, but not what the model was actually thinking.
• Fragmented traces. No posthog_trace_id is passed, so the SDK generates a fresh trace id per generate_content call. A scan is a multi-step, tool-using conversation (preamble → mission steps → tool round-trips → validation retries), and in LLM analytics today nearly every generation lands as its own single-event trace — you can't read a scan as one conversation, and there's no way to find the trace for a given observation.

Changes

• _step_config now sets thinking_config=ThinkingConfig(include_thoughts=True), so thought summaries land in the captured generation output. Answer parsing is unaffected: response.text skips thought parts, and carrying them forward in the conversation is the already-recommended pattern (the code already preserves thought signatures).
• Every generate_content call in a scan now shares one posthog_trace_id — the observation id — so all steps, tool round-trips, and retries of a scan read as a single trace, and the observation id doubles as the trace search key. Evaluation re-runs (snapshot_override) get a fresh uuid so they don't interleave with the real scan's trace.
• Each generation is named after its mission step via $ai_span_name, so the trace tree reads as the mission plan.
• AI observability normalizer: a bare {type: function} block gets a rule in compat_array, so a thinking block sitting next to a Gemini-style tool-call block no longer knocks the tool call into stringified-JSON salvage. Needed because a thinking block takes the message off compat_array's envelope allowlist onto per-block delegation, where nothing claimed the function block.

Companion SDK PR that types Gemini thought summaries as thinking blocks (so the UI renders them as reasoning rather than plain text): PostHog/posthog-python#827

How did you test this code?

• pytest products/replay_vision/backend/tests/test_call_scanner_provider.py — 23 passed. Extended the existing _step_config test to assert the thinking config is present (catches a regression where a config refactor silently drops thought capture).
• AIO frontend: new messageNormalization test proving a thinking block + typed function block normalizes into a thinking message plus a typed tool call (fails with stringified JSON without the new rule). Full AIO frontend jest suite: 631 passed across 45 suites.
• hogli ci:preflight --fix — clean.
• Verified against the pinned SDKs: posthoganalytics==7.30.1's Gemini wrapper captures thought parts as output content blocks and already maps thoughts_token_count to reasoning tokens; google-genai==1.46.0's response.text excludes thought=True parts, so the JSON parsing path is unchanged.
• Not tested against the live Gemini API from this environment; the flash-lite tier has thinking off by default and simply returns no thought parts.

Automatic notifications

• Publish to changelog?

Docs update

N/A — internal instrumentation only.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

• Authored with Claude Code (PostHog Code cloud task). Investigated the existing instrumentation in call_scanner_provider.py, the posthoganalytics Gemini wrapper, and production LLM analytics data, which confirmed generations ≈ traces (full fragmentation) and that reasoning tokens are reported while reasoning content is absent.
• Considered gating include_thoughts per model tier, but all models in billing.GEMINI_MODELS support thinking configuration, and thinking-off models return no thoughts rather than erroring, so it's unconditional.
• Trace id is the observation id (rather than a random uuid per activity attempt) so Temporal retries of the same observation land in the same trace and the id is searchable from the observation row.

---

Created with PostHog Code

[trunk-io]

Merging to master in this repository is managed by Trunk.

• To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

[trunk-io]

   

View Full Report ↗︎ ⋅ Docs

[github-actions]

🦔 Hogbox preview · ✅ ready

▶ Open the preview

🔑 Login	test@posthog.com / 12345678 (demo data)
🧩 Running	this PR's backend and frontend, on the PostHog :master base
🔗 Link	stable across rebuilds — a re-push swaps the box underneath, the URL stays
🔒 Access	tailnet only (PostHog VPN)
🛠️ Admin	inspect & debug state in hogland
💤 Idle	sleeps after ~30 min idle (snapshot to S3, zero node cost) and wakes on your next visit in ~30s, behind a brief "waking up" screen

_{commit e2b7710 · box box-d7a5e9940431 · ready in 970s (push → usable) · build log · rebuilds on every push, torn down on close}

[github-actions]

🤖 CI report

⚠️ Bundle size — 🔺 +3.8 KiB (+0.0%)

Uncompressed size of every built .js bundle, compared against the base branch.

Total: 65.58 MiB · 🔺 +3.8 KiB (+0.0%)

File	Size	Δ vs base
render-query/src/render-query/render-query.js	21.32 MiB	🔺 +2.6 KiB (+0.0%)
posthog-app/_parent/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.js	23.2 KiB	🔺 +1.2 KiB (+5.4%)

_{Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report}

✅ Eager graph — within budget

How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.

Root	Eager (shipped)	Δ vs base	Budget
entry (logged-out pages, app bootstrap) src/index.tsx	1.25 MiB · 22 files	no change	███░░░░░░░ 27.7% of 4.51 MiB
authenticated shell (every logged-in page) src/scenes/AuthenticatedShell.tsx	8.14 MiB · 3,036 files	🔺 +1.2 KiB (+0.0%)	████████░░ 83.8% of 9.71 MiB

🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx

Largest files eagerly shipped from src/index.tsx

Size	File
126.8 KiB	../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js
24.6 KiB	../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js
6.3 KiB	../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js
4.5 KiB	../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js
3.9 KiB	../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js
1.4 KiB	../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js
1.3 KiB	src/RootErrorBoundary.tsx
912 B	../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js
789 B	src/scenes/ChunkLoadErrorBoundary.tsx
762 B	src/index.tsx

Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx

Size	File
285.5 KiB	../node_modules/.pnpm/posthog-js@1.410.1/node_modules/posthog-js/dist/rrweb.js
267.7 KiB	../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js
235.5 KiB	src/taxonomy/core-filter-definitions-by-group.json
231.5 KiB	../node_modules/.pnpm/posthog-js@1.410.1/node_modules/posthog-js/dist/module.js
154.3 KiB	../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js
126.8 KiB	../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js
104.4 KiB	src/lib/api.ts
94.7 KiB	../packages/quill/packages/quill/dist/index.js
93.3 KiB	../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js
90.6 KiB	../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js

_{Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479}

✅ Toolbar bundle — eager 2.19 MiB within budget

What the toolbar ships to customer pages, measured from the esbuild output (minified, post-tree-shake). The eager set is the entry plus everything statically imported from it — fetched before any feature runs; deferred chunks load lazily. The eager guardrail is 5.72 MiB. Each output file must also stay below 10 MB, where CloudFront stops compressing it. The module boundary is enforced separately by check-toolbar-graph.

Metric	Size	Δ vs base	Budget
Eager (shipped) entry + static imports	2.19 MiB · 17 files	no change	████░░░░░░ 38.3% of 5.72 MiB
Deferred (lazy)	2.08 MiB · 33 files	🔺 +903 B (+0.0%)	n/a — loads on demand
Loader dist/toolbar.js	1.1 KiB	no change	█░░░░░░░░░ 5.8% of 19.5 KiB

Largest eagerly-shipped chunks

Size	File
718.1 KiB	dist/toolbar/toolbar-app-6ZY7LRCK.css
551.5 KiB	dist/toolbar/chunk-chunk-UMQTZ4EG.js
484.6 KiB	dist/toolbar/chunk-chunk-ORHT67OV.js
133.6 KiB	dist/toolbar/chunk-chunk-Z7BTLT7U.js
131.8 KiB	dist/toolbar/chunk-chunk-T5KY5WYR.js
71.0 KiB	dist/toolbar/toolbar-app-2LEWJD2B.js
69.0 KiB	dist/toolbar/chunk-chunk-27JL52RE.js
35.6 KiB	dist/toolbar/chunk-chunk-KBKYVHTL.js
20.9 KiB	dist/toolbar/chunk-chunk-7YA5UZUN.js
12.2 KiB	dist/toolbar/chunk-chunk-PIK3PADE.js

_{Posted automatically by check-toolbar-size · sizes are toolbar output bytes (shipped, post-tree-shake) from the esbuild metafile}

✅ Dist folder size — 🔺 +227.6 KiB (+0.0%)

Total size of the built frontend/dist folder (all assets), compared against the base branch.

Total: 1389.58 MiB · 🔺 +227.6 KiB (+0.0%)

⚠️ Backend coverage — 83.0% of changed backend lines covered — 1 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): █████████████████░░░ 83.0% (5 / 6)

File	Patch	Uncovered changed lines
products/replay_vision/backend/temporal/activities/call_scanner_provider.py	80.0%	159

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 30847003852 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
platform_features	██░░░░░░░░░░░░░░░░░░ 12.1%	7 / 58
batch_exports	████████░░░░░░░░░░░░ 39.5%	8,808 / 22,287
demo	███████████░░░░░░░░░ 56.3%	1,497 / 2,661
warehouse_sources_queue	████████████░░░░░░░░ 59.2%	148 / 250
data_tools	██████████████░░░░░░ 70.0%	63 / 90
tasks	██████████████░░░░░░ 70.4%	33,513 / 47,609
ai_gateway	███████████████░░░░░ 75.0%	9 / 12
signals	████████████████░░░░ 81.5%	25,463 / 31,226
cdp	████████████████░░░░ 82.1%	3,320 / 4,042
data_modeling	█████████████████░░░ 85.7%	7,756 / 9,054
notebooks	█████████████████░░░ 86.0%	7,794 / 9,060
managed_warehouse	█████████████████░░░ 86.5%	5,875 / 6,790
actions	█████████████████░░░ 86.6%	717 / 828
wizard	█████████████████░░░ 86.7%	1,090 / 1,257
data_warehouse	█████████████████░░░ 87.5%	11,011 / 12,591
cohorts	██████████████████░░ 87.6%	6,484 / 7,398
exports	██████████████████░░ 87.9%	7,078 / 8,054
product_tours	██████████████████░░ 87.9%	1,303 / 1,482
business_knowledge	██████████████████░░ 89.0%	4,384 / 4,928
engineering_analytics	██████████████████░░ 89.4%	6,551 / 7,331
dashboards	██████████████████░░ 89.4%	5,983 / 6,693
visual_review	██████████████████░░ 89.5%	5,870 / 6,558
conversations	██████████████████░░ 90.0%	18,294 / 20,337
alerts	██████████████████░░ 90.3%	4,482 / 4,966
links	██████████████████░░ 90.6%	183 / 202
streamlit_apps	██████████████████░░ 90.7%	2,630 / 2,901
error_tracking	██████████████████░░ 91.0%	10,981 / 12,070
slack_app	██████████████████░░ 91.2%	9,802 / 10,751
stamphog	██████████████████░░ 91.3%	4,505 / 4,936
marketing_analytics	██████████████████░░ 91.5%	12,700 / 13,874
mcp_store	██████████████████░░ 92.2%	6,525 / 7,076
product_analytics	███████████████████░ 92.5%	5,849 / 6,321
managed_migrations	███████████████████░ 92.6%	1,556 / 1,681
early_access_features	███████████████████░ 92.6%	1,287 / 1,390
ai_observability	███████████████████░ 92.6%	16,036 / 17,319
mcp_analytics	███████████████████░ 92.6%	3,775 / 4,076
notifications	███████████████████░ 92.6%	1,017 / 1,098
surveys	███████████████████░ 93.2%	5,853 / 6,281
posthog_ai	███████████████████░ 93.2%	1,326 / 1,422
approvals	███████████████████░ 93.3%	3,437 / 3,682
reminders	███████████████████░ 93.4%	468 / 501
web_analytics	███████████████████░ 93.5%	15,393 / 16,470
legal_documents	███████████████████░ 93.8%	1,628 / 1,736
endpoints	███████████████████░ 94.2%	8,655 / 9,192
workflows	███████████████████░ 94.2%	7,559 / 8,021
tracing	███████████████████░ 94.5%	2,671 / 2,827
review_hog	███████████████████░ 94.6%	8,246 / 8,715
skills	███████████████████░ 94.6%	3,158 / 3,337
messaging	███████████████████░ 94.7%	2,885 / 3,048
logs	███████████████████░ 95.5%	10,491 / 10,990
experiments	███████████████████░ 95.6%	26,982 / 28,230
annotations	███████████████████░ 96.2%	732 / 761
revenue_analytics	███████████████████░ 96.3%	1,887 / 1,960
growth	███████████████████░ 96.3%	4,767 / 4,949
feature_flags	███████████████████░ 96.4%	17,488 / 18,144
replay_vision	███████████████████░ 96.4%	16,620 / 17,235
user_interviews	███████████████████░ 96.5%	2,638 / 2,734
access_control	███████████████████░ 96.9%	870 / 898
customer_analytics	███████████████████░ 97.1%	10,427 / 10,737
warehouse_sources	███████████████████░ 97.3%	360,793 / 370,784
data_catalog	████████████████████ 97.8%	2,589 / 2,648
analytics_platform	████████████████████ 98.0%	2,153 / 2,197
metrics	████████████████████ 98.2%	2,491 / 2,536
pulse	████████████████████ 98.4%	2,017 / 2,049
live_debugger	████████████████████ 99.2%	613 / 618
field_notes	████████████████████ 99.4%	158 / 159

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

[TueHaulund]

PR looks good 👍