feat(data-warehouse): scaffold 3 newly-requested sources by Gilbert09 · Pull Request #68376 · PostHog/posthog

Gilbert09 · 2026-07-04T09:36:24Z

Problem

Users keep asking for warehouse connectors we don't have yet. On 2026-07-03 the posthog.com "which source do you want?" survey and the in-product "Notify me" button surfaced a fresh batch. This PR scaffolds the legitimate ones so the work to build them out can start from a registered stub.

Changes

Scaffolded three new sources as unreleased stubs (hidden from users via unreleasedSource=True, no sync logic yet):

Source	Category	Notes
Peec AI (`peec_ai`)	Analytics	AI search / brand visibility analytics
Healthchecks.io (`healthchecks`)	Engineering & monitoring	Cron / uptime monitoring
impact.com (`impact`)	Advertising	Partnership / affiliate marketing

Each follows the standard scaffold: enum value in ExternalDataSourceType, entry in externalDataSources (schema-general.ts, regenerated into schema_enums.py), a source.py stub, registration in _load_all.py, a generated config class, an icon in frontend/public/services/, and a SOURCES.md scaffolded-list entry.

All three enum additions land in a single migration (0058) that chains linearly off 0057, so there's exactly one leaf and no branching.

Skipped from the same batch:

aws — a cloud platform, not a single connector (we already have amazon_s3, aws_cloudtrail, amazon_ads, etc.), and no Logo.dev brand entry.

How did you test this code?

I (Claude) verified the scaffolding without a live database (this environment has no Postgres):

Ran pnpm run schema:build and generate:source-configs to regenerate schema_enums.py, schema.json, and generated_configs.py; confirmed each diff contains only the three new entries.
makemigrations --check --dry-run reports no pending changes and --merge --dry-run finds no conflicts, confirming 0058 is a single linear leaf.
Loaded the source registry and confirmed all three sources register, resolve their generated config class, and expose the right category / icon / unreleasedSource=True.
Ran the test_source_categories parametrized tests for the three sources (6 passed) covering valid category and lowercase-keyword rules.
ruff check / ruff format clean over the touched files.

I did not run a real sync or bring up the app - these are stubs with no transport logic.

Automatic notifications

Publish to changelog?
Alert Sales and Marketing teams?

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

I (Claude) scaffolded these following the /implementing-warehouse-sources skill, and invoked /django-migrations before generating the migration. Icons were pulled from Logo.dev.

Decisions worth flagging for review:

Treated the raw survey names strictly as data. Verified each against Logo.dev before scaffolding; skipped aws as too generic to be a single connector.
Checked open PRs (including the maintainer's own) from several angles - none already scaffold peec.ai, healthchecks.io, or impact.com.
Bundled all three into one migration rather than one-per-source. A choices change serializes the full enum list, so three chained migrations would each carry the whole list; one migration is unambiguously linear and far more reviewable, and still satisfies the "single leaf, no shared parent" requirement.
Left unreleasedSource=True on every source on purpose - these are stubs, not finished connectors.

github-actions · 2026-07-04T09:36:34Z

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

greptile-apps · 2026-07-04T09:38:59Z

_{Reviews (1): Last reviewed commit: "feat(data-warehouse): scaffold 3 newly-r..." | Re-trigger Greptile}

github-actions · 2026-07-04T09:39:36Z

Migration SQL Changes

Hey 👋, we've detected some migrations on this PR. Here's the SQL output for each migration, make sure they make sense:

`products/warehouse_sources/backend/migrations/0059_alter_externaldatasource_source_type_and_more.py`

BEGIN;
--
-- Alter field source_type on externaldatasource
--
-- (no-op)
--
-- Alter field source_type on pendingsourcecredential
--
-- (no-op)
COMMIT;

Last updated: 2026-07-08 15:01 UTC (b24c1a0)

github-actions · 2026-07-04T09:39:50Z

🔍 Migration Risk Analysis

We've analyzed your migrations for potential risks.

Summary: 0 Safe | 1 Needs Review | 0 Blocked

⚠️ Needs Review

May have performance impact

warehouse_sources.0059_alter_externaldatasource_source_type_and_more
  └─ #1 ⚠️ AlterField
     Field alteration may cause table locks or data loss (check if changing type or constraints)
     model: externaldatasource, field: source_type, field_type: CharField
  └─ #2 ⚠️ AlterField
     Field alteration may cause table locks or data loss (check if changing type or constraints)
     model: pendingsourcecredential, field: source_type, field_type: CharField

Last updated: 2026-07-08 15:01 UTC (b24c1a0)

github-actions · 2026-07-04T09:40:45Z

🤖 CI report

⚠️

Bundle size — 🔺 +99 B (+0.0%)

Uncompressed size of every built .js bundle, compared against the base branch.

Total: 78.57 MiB · 🔺 +99 B (+0.0%)

No file changed by more than 1000 B.

_{Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report}

⚠️

Eager graph — 1 over budget

How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.

Root	Eager (shipped)	Δ vs base	Budget
entry (logged-out pages, app bootstrap) `src/index.tsx`	1.20 MiB · 22 files	no change	`███░░░░░░░` 27.9% of 4.29 MiB
🟡 authenticated shell (every logged-in page) `src/scenes/AuthenticatedShell.tsx`	14.02 MiB · 2,996 files	🔺 +33 B (+0.0%)	`██████████` 108.9% of 12.87 MiB

🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx

Largest files eagerly shipped from src/index.tsx

Size	File
126.8 KiB	`../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js`
24.6 KiB	`../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js`
6.3 KiB	`../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js`
4.5 KiB	`../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js`
3.9 KiB	`../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js`
1.4 KiB	`../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js`
1.3 KiB	`src/RootErrorBoundary.tsx`
912 B	`../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js`
789 B	`src/scenes/ChunkLoadErrorBoundary.tsx`
668 B	`src/index.tsx`

Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx

Size	File
980.5 KiB	`../node_modules/.pnpm/@posthog+brand@0.6.0_react@18.3.1/node_modules/@posthog/brand/dist/generated/hoggies/svg/code-bubble.mjs`
458.3 KiB	`../node_modules/.pnpm/@posthog+brand@0.6.0_react@18.3.1/node_modules/@posthog/brand/dist/generated/hoggies/svg/driving-hogzilla.mjs`
345.4 KiB	`../node_modules/.pnpm/@posthog+brand@0.6.0_react@18.3.1/node_modules/@posthog/brand/dist/generated/hoggies/svg/wizard-hog.mjs`
302.9 KiB	`../node_modules/.pnpm/@posthog+brand@0.6.0_react@18.3.1/node_modules/@posthog/brand/dist/generated/hoggies/svg/coffee-run.mjs`
277.1 KiB	`../node_modules/.pnpm/posthog-js@1.398.2/node_modules/posthog-js/dist/rrweb.js`
266.9 KiB	`../node_modules/.pnpm/@posthog+icons@0.37.4_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js`
253.2 KiB	`../node_modules/.pnpm/@posthog+brand@0.6.0_react@18.3.1/node_modules/@posthog/brand/dist/generated/hoggies/svg/chart-hog.mjs`
239.2 KiB	`../node_modules/.pnpm/@posthog+brand@0.6.0_react@18.3.1/node_modules/@posthog/brand/dist/generated/hoggies/svg/x-ray.mjs`
238.0 KiB	`../node_modules/.pnpm/@posthog+brand@0.6.0_react@18.3.1/node_modules/@posthog/brand/dist/generated/hoggies/svg/workflows.mjs`
224.6 KiB	`src/taxonomy/core-filter-definitions-by-group.json`

_{Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479}

⚠️

Dist folder size — 🔺 +54.0 KiB (+0.0%)

Total size of the built frontend/dist folder (all assets), compared against the base branch.

Total: 1484.95 MiB · 🔺 +54.0 KiB (+0.0%)

⚠️

Playwright — 11 failed, 5 flaky

🎭 Playwright report · View test results →

❌ 11 failed tests:

create trends insight via API and snapshot (chromium)
See recordings action (chromium)
password-protected insight sharing (chromium)
CSS loads correctly on shared insight page (chromium)
CSS fallback works when hashed CSS returns error (chromium)
renders successfully in a logged-out browser context (chromium)
renders successfully in a logged-out browser context (chromium)
renders successfully in a logged-out browser context (chromium)
preserves saved insight editing state across browser back and forward (chromium)
plays a multi-source recording, pauses, seeks, scrubs, and reaches the end (chromium)
creates, launches, edits and deletes new survey (chromium)

⚠️ 5 flaky tests:

Editing an insight updates the dashboard tile (chromium)
Add insight to new dashboard and view it there (chromium)
creates a Postgres direct source and queries it successfully (chromium)
Toolbar loads (chromium)
Toolbar item in sidebar has launch options (chromium)

These issues are not necessarily caused by your changes.
Annoyed by this section? Help fix flakies and failures and it will go green!

⚠️

Backend snapshots — 1 updated (1 modified, 0 added, 0 deleted)

Query snapshots: Backend query snapshots updated

Changes: 1 snapshots (1 modified, 0 added, 0 deleted)

What this means:

Query snapshots have been automatically updated to match current output
These changes reflect modifications to database queries or schema

Next steps:

Review the query changes to ensure they're intentional
If unexpected, investigate what caused the query to change

Review snapshot changes →

tests-posthog · 2026-07-04T09:52:45Z

⏭️ Skipped snapshot commit because branch advanced to d2de975 while workflow was testing 5191902.

The new commit will trigger its own snapshot update workflow.

If you expected this workflow to succeed: This can happen due to concurrent commits. To get a fresh workflow run, either:

Merge master into your branch, or
Push an empty commit: git commit --allow-empty -m 'trigger CI' && git push

posthog · 2026-07-04T12:09:02Z

👋 Visual changes detected for this PR.

Review and approve in PostHog Visual Review

If these changes are unexpected, they may be caused by a flaky test or a broken snapshot on master. Don't approve — rerun the job or wait for a fix.

Gilbert09 · 2026-07-06T12:56:00Z

Frontend Tests (shard 1) — blocked on a known master-wide jest hang, not this PR

Jest test (FOSS - 1) and Jest test (EE - 1) hang until the 15-min timeout on this branch. This is the known master-wide regression documented in #68608: since #68425 landed, an infinite render loop in useTaxonomicLocalOverrides starves the event loop inside Combobox.test.tsx (which sits in jest shard 1 for both FOSS and EE), so chunk 1 dies on every open PR and on master, not just here. This PR only touches generated schema/config and backend source stubs and adds no frontend tests, so it can't be the cause — shards 2-4 pass in ~4-5 min.

The fix is in flight and approved: #68608 (root cause) and #68618 (excludes the flaky test). Once either lands on master I'll merge master in; shard 1 then passes (~5.5 min, as it does on branches that already carry the fix). No action needed on this PR's own code.

danielcarletti

LGTM

github-actions · 2026-07-08T12:59:21Z

🦔 Hogbox preview · ✅ ready

▶ Open the preview


🔑 Login	`test@posthog.com` / `12345678` (demo data)
🧩 Running	this PR's backend and frontend, on the PostHog `:master` base
🔗 Link	stable across rebuilds — a re-push swaps the box underneath, the URL stays
🔒 Access	tailnet only (PostHog VPN)
💤 Idle	sleeps after ~30 min idle (snapshot to S3, zero node cost) and wakes on your next visit in ~30s, behind a brief "waking up" screen

_{commit b24c1a0 · box box-69e2dcc7d45f · up in 376s · build log · rebuilds on every push, torn down on close}

trunk-io · 2026-07-08T13:10:16Z

_{View Full Report ↗︎ ⋅ Docs}

Gilbert09 · 2026-07-08T14:20:37Z

Status update (automated PR-tidy pass)

Everything within scope of an automated fix is done — the PR is approved, conflict-free, and up to date with master. The only remaining red check is the Visual regression tests pass gate, and it is failing on pre-existing master-level snapshot flakes that are unrelated to this PR.

What was fixed

Merge conflict resolved. master landed its own 0058_externaldataschemaoomevent warehouse-sources migration, which collided with this PR's 0058 source_type migration. Rebased onto master, renumbered this PR's migration to 0059, repointed its dependency onto 0058_externaldataschemaoomevent, and updated max_migration.txt. Single linear leaf, no branching.
Branch is current. Merged master in (two-parent, no linearization); master is now an ancestor, branch is 0 commits behind, and the PR's own file set is unchanged (no base-branch files leaked into the diff).
Review threads: the one thread (peec.ai keywords) is resolved.
All other CI is green (Django, Dagster, Jest, Playwright, Rust, MCP, semgrep, migrations check, etc.).

Why the Visual Review gate is red (not this PR)

This PR's three new sources are scaffolded as unreleasedSource=True, so they render nothing in Storybook — it has zero legitimate visual impact. The snapshots the gate flags are chart-rendering stories that rotate between runs and are changed on master itself with identical content hashes:

Run	Blocking snapshots	Also `changed` on `master`?
commit `cbcbde0c`	`queries-webvitals--web-vitals` (date-axis labels), `scenes-app-mcp-analytics--dashboard` (sub-pixel)	yes — same hash on master run at 12:50
commit `0fe8e1ab` (after merging master)	`scenes-app-insights-trendsvalue--trends-value` (dark/light)	yes — same hash on master commit `ec641090` at 13:26

Different charts flake each run, none touch the data-warehouse surface, and each has a long history of flipping changed/unchanged on master. Re-running (I brought master in to force a fresh run) just rotates which charts flake, so it will not converge on its own.

What's needed to green it

A maintainer with Visual Review access needs to finalize / tolerate the latest run, or the flaky snapshots need re-baselining on master:

Latest run: https://us.posthog.com/project/2/visual_review/runs/36af1df8-740c-4e0b-819b-d93a4ed6c0ef

I deliberately did not auto-approve: the Visual Review tooling (and the bot comment above) reserve the finalize/approve action for a human when the diffs are suspected master flakes, precisely so an agent doesn't rubber-stamp a shared baseline. Handing that one decision back.

Scaffold stub connectors for three data-warehouse sources requested via the posthog.com source survey and the in-product "Notify me" button: Peec AI (analytics), Healthchecks.io (engineering & monitoring), and impact.com (advertising). Each ships as an unreleased stub (unreleasedSource=True), so it stays hidden from users. No sync logic yet. A single Django migration (0058) adds all three enum values in one linear step off 0057. Skipped from the same survey batch: "aws" (a cloud platform, not a single connector, and no Logo.dev brand) and other non-source entries. Generated-By: PostHog Code Task-Id: b05e044c-3271-4a39-90f2-8e8975ed5c48

makemigrations emitted a blank line after the class declaration that ruff format rejects. Reformat so `ruff format --check` passes. Generated-By: PostHog Code Task-Id: b05e044c-3271-4a39-90f2-8e8975ed5c48

Match the healthchecks and impact stubs, which both set keywords. Adds alternate spellings so catalog search surfaces the source. Generated-By: PostHog Code Task-Id: b05e044c-3271-4a39-90f2-8e8975ed5c48

Add "AI brand visibility" and "AI search analytics" keywords so the peec.ai source stub surfaces for those catalog searches, matching the coverage of the other new stubs. Generated-By: PostHog Code Task-Id: 4e0933fc-4c6a-41f4-926d-66e597536231

Source keywords must be lowercase (enforced by test_source_keywords_are_a_list_of_strings). Lowercase the two newly added phrase keywords. Generated-By: PostHog Code Task-Id: 4e0933fc-4c6a-41f4-926d-66e597536231

deployment-status-posthog · 2026-07-08T16:11:43Z

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-08 16:11 UTC	Run
prod-us	✅ Deployed	2026-07-08 16:41 UTC	Run
prod-eu	✅ Deployed	2026-07-08 16:43 UTC	Run

pr-assigner-resolver-posthog Bot assigned Gilbert09 Jul 4, 2026

pr-assigner-resolver-posthog Bot requested a review from a team July 4, 2026 09:36

greptile-apps Bot reviewed Jul 4, 2026

View reviewed changes

Comment thread products/warehouse_sources/backend/temporal/data_imports/sources/peec_ai/source.py

Gilbert09 mentioned this pull request Jul 6, 2026

feat(data-warehouse): scaffold 2 newly-requested sources #68535

Merged

2 tasks

danielcarletti approved these changes Jul 6, 2026

View reviewed changes

Gilbert09 force-pushed the posthog-code/scaffold-newly-requested-warehouse-sources branch from f6a8084 to cbcbde0 Compare July 8, 2026 12:59

github-actions Bot deployed to preview-pr-68376 July 8, 2026 12:59 View deployment

github-actions Bot deployed to preview-pr-68376 July 8, 2026 13:44 View deployment

Gilbert09 added 7 commits July 8, 2026 15:55

chore: update OpenAPI generated types

692d34b

chore(data-warehouse): ruff-format migration 0058

022af0c

makemigrations emitted a blank line after the class declaration that ruff format rejects. Reformat so `ruff format --check` passes. Generated-By: PostHog Code Task-Id: b05e044c-3271-4a39-90f2-8e8975ed5c48

feat(data-warehouse): add search keywords to peec.ai stub

693e123

Match the healthchecks and impact stubs, which both set keywords. Adds alternate spellings so catalog search surfaces the source. Generated-By: PostHog Code Task-Id: b05e044c-3271-4a39-90f2-8e8975ed5c48

test(backend): update query snapshots

0367c1b

fix(data-warehouse): lowercase peec.ai search keywords

b24c1a0

Source keywords must be lowercase (enforced by test_source_keywords_are_a_list_of_strings). Lowercase the two newly added phrase keywords. Generated-By: PostHog Code Task-Id: 4e0933fc-4c6a-41f4-926d-66e597536231

Gilbert09 force-pushed the posthog-code/scaffold-newly-requested-warehouse-sources branch from 0fe8e1a to b24c1a0 Compare July 8, 2026 14:55

github-actions Bot deployed to preview-pr-68376 July 8, 2026 14:56 View deployment

Gilbert09 enabled auto-merge (squash) July 8, 2026 15:35

Gilbert09 merged commit 2fe8c90 into master Jul 8, 2026
280 checks passed

Gilbert09 deleted the posthog-code/scaffold-newly-requested-warehouse-sources branch July 8, 2026 15:40

Gilbert09 mentioned this pull request Jul 8, 2026

feat(data-warehouse): scaffold 25 new saas/ai source stubs #69517

Merged

2 tasks

Uh oh!

Conversation

Gilbert09 commented Jul 4, 2026

Problem

Changes

How did you test this code?

Automatic notifications

🤖 Agent context

Uh oh!

github-actions Bot commented Jul 4, 2026

Uh oh!

greptile-apps Bot commented Jul 4, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Migration SQL Changes

products/warehouse_sources/backend/migrations/0059_alter_externaldatasource_source_type_and_more.py

Uh oh!

github-actions Bot commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Migration Risk Analysis

⚠️ Needs Review

Uh oh!

github-actions Bot commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 CI report

Query snapshots: Backend query snapshots updated

Uh oh!

tests-posthog Bot commented Jul 4, 2026

Uh oh!

posthog Bot commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gilbert09 commented Jul 6, 2026

Frontend Tests (shard 1) — blocked on a known master-wide jest hang, not this PR

Uh oh!

danielcarletti left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦔 Hogbox preview · ✅ ready

▶ Open the preview

Uh oh!

trunk-io Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gilbert09 commented Jul 8, 2026

Status update (automated PR-tidy pass)

What was fixed

Why the Visual Review gate is red (not this PR)

What's needed to green it

Uh oh!

Uh oh!

deployment-status-posthog Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploy status

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jul 4, 2026 •

edited

Loading

`products/warehouse_sources/backend/migrations/0059_alter_externaldatasource_source_type_and_more.py`

github-actions Bot commented Jul 4, 2026 •

edited

Loading

github-actions Bot commented Jul 4, 2026 •

edited

Loading

posthog Bot commented Jul 4, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

trunk-io Bot commented Jul 8, 2026 •

edited

Loading

deployment-status-posthog Bot commented Jul 8, 2026 •

edited

Loading