Skip to content

feat(billing): add posthog code token spend analysis banner#2224

Merged
pauldambra merged 6 commits into
mainfrom
posthog-code/llma-spend-analysis-banner
May 21, 2026
Merged

feat(billing): add posthog code token spend analysis banner#2224
pauldambra merged 6 commits into
mainfrom
posthog-code/llma-spend-analysis-banner

Conversation

@pauldambra
Copy link
Copy Markdown
Member

@pauldambra pauldambra commented May 19, 2026

Problem

PostHog Code users on the Plan & usage page have no way to see how their LLM token spend breaks down. The data exists in PostHog's LLM analytics — we just haven't surfaced it back inside the app. This PR adds an in-app banner that runs the analysis on demand and prints the result inline, doubling as an upsell for PostHog's LLM analytics product.

Changes

  • New TokenSpendAnalysisBanner rendered at the top of Settings -> Plan & usage. Idle state is an upsell callout with an "Analyse my spend" button; click runs the analysis, swaps to inline tables and heuristic suggestions on completion.
  • New useTokenSpendAnalysis hook (renderer-side useState/useEffect pattern matching other cloud-API callers) and a new getPostHogCodeSpendAnalysis(days) method on PostHogAPIClient that hits GET /api/llm_analytics/posthog_code_spend/.
  • Response types in apps/code/src/renderer/features/billing/types/spend-analysis.ts, mirroring the backend serializer.
  • Footer links: PostHog LLM analytics docs and the exploring-llm-costs skill in the skill store.
  • Gated behind posthog-code-spend-analysis (always on in dev). The BILLING_FLAG that gates Plan & usage itself is also forced on in dev so the section is reachable without per-developer flag setup.

Backend endpoint: PostHog/posthog#59017.

Screenshots

Banner idle state — Settings → Plan & usage:

CleanShot 2026-05-19 at 16 19 20@2x

How did you test this code?

I'm an agent. I did not perform manual UI testing — the screenshot above is from the human author's local run.

  • pnpm --filter=@posthog/code typecheck passes on the rebuilt branch.
  • pnpm biome check --write --unsafe clean on the touched files.

The banner is reachable in dev at Cmd+, → Plan & usage.

Publish to changelog?

no — backend dependency lands first; user-facing announcement comes later.

🤖 Agent context

Authored with PostHog Code (Claude Opus 4.7).

  • Approaches considered and rejected:
    • Letting an agent task handle the analysis (background task with results in side panel) — adds friction for a one-shot upsell view.
    • Calling MCP directly from the renderer — wrong scope; MCP is per-agent auth.
    • Querying the customer's own PostHog project's HogQL — wrong project; the data lives in PostHog's internal cloud analytics team.
  • Chose a fixed backend report + inline tables. Faster, deterministic, no LLM in the loop for data fetching.
  • Suggestions are hardcoded heuristics (Bash share, no-tool share, top-trace share). Easy to swap for an LLM-generated narrative later if useful.

Surfaces a self-serve spend analysis inside Settings -> Plan & usage.
Calls the new /api/llm_analytics/posthog_code_spend/ endpoint, renders
totals + breakdowns by ai_product / tool / model + top traces, and
generates inline heuristic suggestions on where to optimise. Footer
links to PostHog LLM analytics docs and the exploring-llm-costs skill.

Gated behind the posthog-code-spend-analysis feature flag (always on
in dev). Plan & usage section itself is also forced on in dev so the
banner is reachable without billing-flag setup.

Generated-By: PostHog Code
Task-Id: f9d5d152-49c6-46cf-8fde-079105ba2e67
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 19, 2026

Prompt To Fix All With AI
Fix the following 3 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 3
apps/code/src/renderer/features/billing/hooks/useTokenSpendAnalysis.ts:28-33
Refresh errors are silently swallowed after the first successful load. Because `data` is never cleared on failure, the banner always takes the `if (data)` branch in `TokenSpendAnalysisBanner`, so the `if (error)` branch is unreachable once data has been fetched. When a refresh fails the user sees the stale previous result with no indication anything went wrong.

```suggestion
      const result = await client.getPostHogCodeSpendAnalysis(days);
      setData(result);
      setError(null);
    } catch (err) {
      const message = err instanceof Error ? err.message : "Unknown error";
      log.warn("Failed to fetch spend analysis", { error: message });
      setData(null);
      setError(message);
```

### Issue 2 of 3
apps/code/src/renderer/features/billing/components/TokenSpendAnalysisBanner.tsx:23-24
The `amount < 1` branch is a strict subset of `amount < 100` and performs the exact same operation (`toFixed(2)`), so it is dead code that adds no distinction. This violates the "no superfluous parts" simplicity rule.

```suggestion
  if (amount < 100) return `$${amount.toFixed(2)}`;
```

### Issue 3 of 3
apps/code/src/renderer/features/billing/types/spend-analysis.ts:22-28
`SpendAnalysisModelRow` is defined and `by_model` is included in `SpendAnalysisResponse`, but the component never renders a model breakdown table. The data is fetched over the wire and typed, yet discarded. Either a `ModelTable` is missing from the banner, or the interface and field should be dropped to satisfy the "no superfluous parts" rule.

Reviews (1): Last reviewed commit: "feat(billing): add posthog code token sp..." | Re-trigger Greptile

Comment thread apps/code/src/renderer/features/billing/hooks/useTokenSpendAnalysis.ts Outdated
Comment thread apps/code/src/renderer/features/billing/components/TokenSpendAnalysisBanner.tsx Outdated
Comment thread apps/code/src/renderer/features/billing/types/spend-analysis.ts
- Clear data state on fetch error so error UI is reachable on refresh failures.
- Render the model breakdown table that was already typed and fetched but unused.
- Remove redundant amount < 1 branch in formatUsd.

Generated-By: PostHog Code
Task-Id: f9d5d152-49c6-46cf-8fde-079105ba2e67
@pauldambra pauldambra added the Stamphog This will request an autostamp by stamphog on small changes label May 19, 2026
Copy link
Copy Markdown
Contributor

@joshsny joshsny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - wondering if it should be an MCP tool and we send them into the chat, probably not since they want to look at this when they ran out of $ so that would be frustrating

Renames the client method to getPersonalSpendAnalysis, takes an options
object with optional `product` query param, and points at the new
/api/llm_analytics/personal_spend/ endpoint. Banner passes
`product: "posthog_code"` so the tool / model / trace breakdowns stay
PostHog Code-scoped; the by_product breakdown shows the cross-product
distribution.

SpendAnalysisSummary drops posthog_code_cost_usd / posthog_code_event_count
in favour of generic scoped_cost_usd / scoped_event_count fields, matching
the backend shape.

Hook renamed useTokenSpendAnalysis -> useSpendAnalysis.

Generated-By: PostHog Code
Task-Id: f9d5d152-49c6-46cf-8fde-079105ba2e67
…aginated

Mirrors the backend changes:

- URL: getPersonalSpendAnalysis now calls /api/llm_analytics/@me/spend/
- Response shape: each breakdown is `{ items, truncated }`; banner reads
  `.items` and tables stay the same.
- SpendAnalysisToolRow gains `share_of_scoped` (float 0-1). Banner
  suggestions use this directly, no longer divide cost_usd by
  scoped_cost_usd (which can over-count for multi-tool generations).

Generated-By: PostHog Code
Task-Id: f9d5d152-49c6-46cf-8fde-079105ba2e67
Tracks the backend rename:

- Client method now takes `{ dateFrom, dateTo, product }` instead of
  `{ days, product }`. Maps to the `date_from` / `date_to` query params
  the backend now accepts.
- SpendAnalysisSummary swaps `period_days` for `date_from` / `date_to`
  ISO strings.
- Banner derives the "Window" stat card from the two timestamps.

Generated-By: PostHog Code
Task-Id: f9d5d152-49c6-46cf-8fde-079105ba2e67
…nd-analysis-banner

# Conflicts:
#	apps/code/src/renderer/api/posthogClient.ts
#	apps/code/src/renderer/features/settings/components/SettingsDialog.tsx
@pauldambra pauldambra added the Create Release This will trigger a new release label May 21, 2026
@pauldambra pauldambra merged commit 025801b into main May 21, 2026
15 checks passed
@pauldambra pauldambra deleted the posthog-code/llma-spend-analysis-banner branch May 21, 2026 08:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Create Release This will trigger a new release Stamphog This will request an autostamp by stamphog on small changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants