Skip to content

chore: Speed up digest generation time - #69801

Merged
jordanm-posthog merged 2 commits into
masterfrom
jordanm-posthog/wa-digest-slow-load
Jul 9, 2026
Merged

chore: Speed up digest generation time#69801
jordanm-posthog merged 2 commits into
masterfrom
jordanm-posthog/wa-digest-slow-load

Conversation

@jordanm-posthog

Copy link
Copy Markdown
Contributor

Problem

The Web Analytics digest loads slowly on every visit

Changes

Cache + parallelize the queries

How did you test this code?

Locally

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Docs update

🤖 Agent context

Autonomy: Human-driven (agent-assisted) - or - Fully autonomous

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

🦔 Hogbox preview · ✅ ready

▶ Open the preview

🔑 Login test@posthog.com / 12345678 (demo data)
🧩 Running this PR's backend and frontend, on the PostHog :master base
🔗 Link stable across rebuilds — a re-push swaps the box underneath, the URL stays
🔒 Access tailnet only (PostHog VPN)
💤 Idle sleeps after ~30 min idle (snapshot to S3, zero node cost) and wakes on your next visit in ~30s, behind a brief "waking up" screen

commit d6aa8fb · box box-c89cd1e7da7f · ready in 486s (push → usable) · build log · rebuilds on every push, torn down on close

@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team July 9, 2026 19:21
@greptile-apps

greptile-apps Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Comments Outside Diff (1)

  1. products/web_analytics/backend/weekly_digest.py, line 272-275 (link)

    P1 Mixed freshness breaks recap ratios

    These four queries use separate cache keys but are combined as one snapshot. In the async-if-stale API path, overview can be stale while top pages or goals are fresh, so recap logic can compare values from different cache generations and assign the wrong persona or highlight.

Reviews (1): Last reviewed commit: "Speed up digest generation time" | Re-trigger Greptile

Comment thread frontend/src/scenes/web-analytics/recap/webAnalyticsRecapLogic.ts Outdated
digest = build_team_digest(self.team, days=params["days"], compare=params["compare"])
digest = build_team_digest(
self.team,
days=params["days"],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Stale cache returned as current digest

This mode blocks on cache miss, but returns stale cached results immediately when a cached digest query is past its freshness window. The old calculate() path always produced fresh data, so /weekly_digest can now return old visitors, pages, sources, or goals as the current weekly digest while the refresh runs in the background.

recap = build_team_recap(
self.team,
days=params["days"],
compare=params["compare"],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Stale cache drives recap persona

The recap endpoint now uses async-if-stale cache behavior for the digest that feeds compute_persona and highlights. When any section has a stale cache entry, the user can receive a recap persona and highlight set derived from old data instead of the current weekly recap.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Bundle size — 🔺 +136 B (+0.0%)

Uncompressed size of every built .js bundle, compared against the base branch.

Total: 69.86 MiB · 🔺 +136 B (+0.0%)

No file changed by more than 1000 B.

Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report

Eager graph — within budget

How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.

Root Eager (shipped) Δ vs base Budget
entry (logged-out pages, app bootstrap)
src/index.tsx
1.20 MiB · 22 files no change ███░░░░░░░ 28.0% of 4.29 MiB
authenticated shell (every logged-in page)
src/scenes/AuthenticatedShell.tsx
8.07 MiB · 2,956 files no change █████████░ 87.2% of 9.25 MiB

🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx

Largest files eagerly shipped from src/index.tsx
Size File
126.8 KiB ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js
24.6 KiB ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js
6.3 KiB ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js
4.5 KiB ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js
3.9 KiB ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js
1.4 KiB ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js
1.3 KiB src/RootErrorBoundary.tsx
912 B ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js
789 B src/scenes/ChunkLoadErrorBoundary.tsx
668 B src/index.tsx
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
Size File
277.1 KiB ../node_modules/.pnpm/posthog-js@1.398.2/node_modules/posthog-js/dist/rrweb.js
266.9 KiB ../node_modules/.pnpm/@posthog+icons@0.37.4_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js
224.6 KiB src/taxonomy/core-filter-definitions-by-group.json
213.6 KiB ../node_modules/.pnpm/posthog-js@1.398.2/node_modules/posthog-js/dist/module.js
164.0 KiB src/queries/validators.js
154.0 KiB ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js
126.8 KiB ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js
105.7 KiB src/lib/api.ts
93.3 KiB ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js
90.5 KiB ../node_modules/.pnpm/@tiptap+core@3.20.1_@tiptap+pm@3.20.1/node_modules/@tiptap/core/dist/index.js

Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479

⚠️ Dist folder size — 🔺 +618 B (+0.0%)

Total size of the built frontend/dist folder (all assets), compared against the base branch.

Total: 1433.70 MiB · 🔺 +618 B (+0.0%)

⚠️ Playwright — 11 failed

🎭 Playwright report · View test results →

11 failed tests:

  • create trends insight via API and snapshot (chromium)
  • See recordings action (chromium)
  • password-protected insight sharing (chromium)
  • CSS loads correctly on shared insight page (chromium)
  • CSS fallback works when hashed CSS returns error (chromium)
  • renders successfully in a logged-out browser context (chromium)
  • renders successfully in a logged-out browser context (chromium)
  • renders successfully in a logged-out browser context (chromium)
  • creates a Postgres direct source and queries it successfully (chromium)
  • plays a multi-source recording, pauses, seeks, scrubs, and reaches the end (chromium)
  • creates, launches, edits and deletes new survey (chromium)

These issues are not necessarily caused by your changes.
Annoyed by this section? Help fix flakies and failures and it will go green!

Comment thread frontend/src/scenes/web-analytics/recap/webAnalyticsRecapLogic.ts Outdated
@veria-ai

veria-ai Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

@trunk-io

trunk-io Bot commented Jul 9, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

@jordanm-posthog
jordanm-posthog merged commit 0ba58bc into master Jul 9, 2026
234 checks passed
@jordanm-posthog
jordanm-posthog deleted the jordanm-posthog/wa-digest-slow-load branch July 9, 2026 21:52
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 9, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-09 22:28 UTC Run
prod-us ✅ Deployed 2026-07-09 22:41 UTC Run
prod-eu ✅ Deployed 2026-07-09 22:44 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants