fix(data-warehouse): shrink new-source wizard payload and gzip it

[danielcarletti]

Problem

The new-source wizard page (/pipeline/new/source) blocks on GET /api/environments/:id/external_data_sources/wizard/, which returned 1.22 MB of uncompressed JSON describing all 761 registered sources. Two compounding issues:

1. 43% of the payload (521 KB) was dead weight. The tables key is the credential-free documented-table catalog built for the posthog.com docs build (served by the public /api/public_source_configs/ endpoint). Nothing on the app path reads it: the frontend SourceConfig type has no tables field, and the MCP tool's response projection (pickResponseFields in services/mcp) only passes through name, caption, docsUrl, featured, unreleasedSource, and fields.
2. The response shipped uncompressed. PostHog gzips API responses via a scoped allowlist (ScopedGZipMiddleware), and this path wasn't on it.

Server CPU was not the bottleneck (catalog build is ~35 ms warm), transfer was.

Changes

	before	after
wizard response (raw)	1,218 KB	688 KB
over the wire	1,218 KB (no gzip)	86 KB (gzip)
catalog build per request	~35 ms	~0 ms (memoized)

• build_source_configs() gains a keyword-only include_tables flag and is memoized with functools.cache (the catalog is deploy-static, it only changes when source code ships)
• the wizard action calls it with include_tables=False; the public docs endpoint is unchanged and keeps tables
• the wizard path is added to GZIP_RESPONSE_ALLOW_LIST. BREACH-safe: the response reflects no user input and contains no secrets
• the endpoint drift test now asserts public == wizard + tables, and that the wizard response carries no tables key

No frontend change needed, and the OpenAPI spec is unchanged (tables was never part of the SourceConfig schema, it was bolted onto the dict).

Estimated impact

Pages that fetch this endpoint (the new-source wizard, the source connect scene, and the source configuration tab) get roughly 14k loads per month across US and EU Cloud combined. That works out to about 16.7 GB/month of uncompressed transfer before, ~1.2 GB/month after, so ~15.5 GB/month saved (93%). At typical cloud egress rates that's only a couple of dollars a month, so the dollar savings are negligible. The real win is page load time (1.2 MB of JSON on a slow connection is seconds of blocking spinner) plus a small per-request CPU saving on the web workers.

Is the MCP tool affected?

No. The generated external-data-sources-wizard MCP tool fetches this endpoint and then applies an include-list projection before returning anything to the model. tables was fetched over the internal network and immediately discarded, so the tool's output is byte-identical after this change. The tool also declares its response as unknown (no zod validation of the response body), so removing a key cannot break parsing. Side benefit: the MCP service now downloads 688 KB instead of 1.22 MB for unfiltered calls.

How did you test this code?

Automated (run locally, all green):

• products/data_warehouse/backend/tests/api/test_public_source_configs.py (7 passed), including the updated drift test — it now catches both regressions this change creates room for: the two endpoints drifting on shared fields, and someone re-adding tables to the wizard response and doubling the payload
• products/data_warehouse/backend/tests/api/test_external_data_source.py -k wizard (15 passed)
• posthog/test/test_gzip_middleware.py — 2 passed, 2 pre-existing failures that fail identically on a clean tree (local env can't render the home page), unrelated to this change
• payload sizes and the gzip allowlist regex verified with a local script against the real registry (761 sources): pattern matches /api/environments/2/external_data_sources/wizard/ and /api/projects/2/external_data_sources/wizard and nothing else on that router

I (Claude) did not manually test the page in a browser.

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

No docs reference the wizard endpoint payload shape; the public docs endpoint that posthog.com consumes is unchanged.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

• Authored by Claude Code (Fable 5) in a Claude Code session, directed by Daniel. Skills invoked: /improving-drf-endpoints, /writing-tests.
• Investigated three options the driver raised (new endpoint, smaller response, caching) and landed on trimming + memoizing + gzip on the existing endpoint: a new endpoint would duplicate the catalog builder for no extra saving, and HTTP-level ETag/304 caching was considered and dropped as low-value once the wire size is 86 KB (would also need conditional-GET wiring this stack doesn't have today).
• Verified the two consumers of the removed field before cutting it: frontend SourceConfig type (no tables field, no .tables reads in app code) and the generated MCP tool (include-list projection drops it).
• Monthly load and transfer estimates come from querying pageviews of the three scenes that mount availableSourcesLogic, then multiplying by measured payload sizes.

🤖 Generated with Claude Code

[greptile-apps]

_{Reviews (1): Last reviewed commit: "fix(data-warehouse): shrink new-source w..." | Re-trigger Greptile}

[github-actions]

🤖 CI report

✅ Playwright — all passed

All tests passed.

View test results →

⚠️ Backend coverage — 78.0% of changed backend lines covered — 3 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ████████████████░░░░ 78.0% (11 / 14)

File	Patch	Uncovered changed lines
products/data_warehouse/backend/presentation/views/public_source_configs.py	66.7%	39–41

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29353135880 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
platform_features	██░░░░░░░░░░░░░░░░░░ 12.1%	7 / 58
batch_exports	████████░░░░░░░░░░░░ 39.7%	8,416 / 21,220
demo	███████████░░░░░░░░░ 56.2%	1,497 / 2,663
warehouse_sources_queue	████████████░░░░░░░░ 59.2%	148 / 250
tasks	█████████████░░░░░░░ 67.2%	25,262 / 37,571
data_tools	██████████████░░░░░░ 70.0%	63 / 90
ai_gateway	███████████████░░░░░ 75.0%	9 / 12
data_modeling	████████████████░░░░ 78.4%	4,696 / 5,987
signals	████████████████░░░░ 78.5%	18,105 / 23,059
cdp	████████████████░░░░ 80.7%	3,118 / 3,864
wizard	████████████████░░░░ 82.5%	772 / 936
cohorts	█████████████████░░░ 82.9%	3,048 / 3,675
notebooks	█████████████████░░░ 83.8%	6,086 / 7,259
agent_platform	█████████████████░░░ 84.1%	3,095 / 3,678
actions	█████████████████░░░ 86.6%	717 / 828
product_tours	█████████████████░░░ 87.5%	1,266 / 1,447
exports	██████████████████░░ 88.3%	6,857 / 7,763
visual_review	██████████████████░░ 88.5%	5,565 / 6,287
business_knowledge	██████████████████░░ 88.5%	4,400 / 4,969
conversations	██████████████████░░ 88.9%	15,924 / 17,921
mcp_analytics	██████████████████░░ 89.1%	2,485 / 2,790
dashboards	██████████████████░░ 89.1%	5,648 / 6,337
engineering_analytics	██████████████████░░ 89.4%	4,560 / 5,100
error_tracking	██████████████████░░ 89.5%	9,613 / 10,737
streamlit_apps	██████████████████░░ 90.4%	2,499 / 2,764
slack_app	██████████████████░░ 90.6%	9,460 / 10,444
links	██████████████████░░ 90.6%	183 / 202
marketing_analytics	██████████████████░░ 90.7%	11,476 / 12,646
alerts	██████████████████░░ 90.9%	3,416 / 3,760
product_analytics	██████████████████░░ 91.1%	5,507 / 6,048
data_warehouse	██████████████████░░ 91.9%	17,694 / 19,263
managed_migrations	██████████████████░░ 91.9%	908 / 988
workflows	██████████████████░░ 92.0%	4,795 / 5,210
mcp_store	██████████████████░░ 92.1%	3,665 / 3,981
web_analytics	███████████████████░ 92.7%	13,702 / 14,787
notifications	███████████████████░ 92.7%	1,026 / 1,107
ai_observability	███████████████████░ 92.7%	14,670 / 15,822
surveys	███████████████████░ 92.9%	5,660 / 6,094
posthog_ai	███████████████████░ 93.2%	1,311 / 1,407
tracing	███████████████████░ 93.2%	2,423 / 2,599
approvals	███████████████████░ 93.3%	3,395 / 3,640
reminders	███████████████████░ 93.4%	468 / 501
early_access_features	███████████████████░ 93.8%	848 / 904
legal_documents	███████████████████░ 94.1%	1,568 / 1,667
endpoints	███████████████████░ 94.1%	8,606 / 9,143
skills	███████████████████░ 94.4%	2,827 / 2,995
revenue_analytics	███████████████████░ 94.4%	3,586 / 3,797
messaging	███████████████████░ 94.5%	2,530 / 2,677
review_hog	███████████████████░ 94.5%	6,429 / 6,802
logs	███████████████████░ 95.3%	9,476 / 9,942
growth	███████████████████░ 95.5%	2,734 / 2,864
experiments	███████████████████░ 95.6%	24,045 / 25,152
replay_vision	███████████████████░ 95.7%	13,151 / 13,745
feature_flags	███████████████████░ 96.0%	14,600 / 15,203
warehouse_sources	███████████████████░ 96.1%	214,534 / 223,219
annotations	███████████████████░ 96.2%	732 / 761
user_interviews	███████████████████░ 96.4%	2,242 / 2,325
access_control	███████████████████░ 96.8%	849 / 877
data_catalog	███████████████████░ 97.2%	1,642 / 1,689
customer_analytics	███████████████████░ 97.3%	7,396 / 7,600
analytics_platform	████████████████████ 98.2%	2,098 / 2,137
metrics	████████████████████ 98.3%	2,363 / 2,405
live_debugger	████████████████████ 99.2%	613 / 618
field_notes	████████████████████ 99.4%	158 / 159

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

[trunk-io]

     

View Full Report ↗︎ ⋅ Docs

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-14 18:20 UTC	Run
prod-us	✅ Deployed	2026-07-14 18:35 UTC	Run
prod-eu	✅ Deployed	2026-07-14 18:37 UTC	Run