feat(data-warehouse): scaffold 25 new saas/ai source stubs by Gilbert09 · Pull Request #69517 · PostHog/posthog

Gilbert09 · 2026-07-08T22:19:14Z

Problem

Users increasingly want warehouse imports from fast-growing SaaS and AI platforms (AI coding tools, voice agents, LLM observability, usage billing). We researched a batch of high-demand vendors with confirmed public data APIs and want them scaffolded so per-source implementation PRs stay small and conflict-free on the shared enum/schema files.

Changes

Scaffolds 25 new import sources, following the same pattern as previous scaffold batches (for example #68376):

New ExternalDataSourceType enum values plus the matching externalDataSources schema entries and regenerated schema/OpenAPI types
One stub source.py per source, registered with unreleasedSource=True and empty fields, so nothing is user-visible yet
Placeholder generated configs, _load_all.py imports, and SOURCES.md scaffolded-list entries
Migration 0060 (choices-only AlterField, no DDL)

Sources: Aikido Security, Alguna, Anthropic, Appwrite, Bland AI, Browse AI, Browser Use, ChartHop, Cody, Cursor, Decagon, Deepgram, ElevenLabs, Harvey, Hyperspell, Langfuse, Lingo.dev, m3ter, Maxio, Metorial, OpenRouter, Together AI, Vapi, Vespa, Writesonic.

Icons are intentionally not included; each implementation PR adds its own icon alongside the real config fields.

How did you test this code?

pytest products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_source_config_generator.py (15 passed, snapshot updated) — exercises the source registry, so all 25 stubs import and register correctly
pnpm schema:build, pnpm generate:source-configs, and hogli build:openapi all run clean with no drift
hogli ci:preflight --fix passes (lint, format, migration conflict checks)

No behavior change: every source is behind unreleasedSource=True with empty fields.

Automatic notifications

Publish to changelog?
Alert Sales and Marketing teams?

Docs update

No docs update needed: sources are unreleased stubs; user-facing docs land with each implementation PR.

github-actions · 2026-07-08T22:19:25Z

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

github-actions · 2026-07-08T22:19:30Z

🦔 Hogbox preview · ✅ ready

▶ Open the preview


🔑 Login	`test@posthog.com` / `12345678` (demo data)
🧩 Running	this PR's backend and frontend, on the PostHog `:master` base
🔗 Link	stable across rebuilds — a re-push swaps the box underneath, the URL stays
🔒 Access	tailnet only (PostHog VPN)
💤 Idle	sleeps after ~30 min idle (snapshot to S3, zero node cost) and wakes on your next visit in ~30s, behind a brief "waking up" screen

_{commit d549e7e · box box-07bf15129319 · ready in 625s (push → usable) · build log · rebuilds on every push, torn down on close}

greptile-apps · 2026-07-08T22:22:16Z

_{Reviews (1): Last reviewed commit: "feat(data-warehouse): scaffold 25 new sa..." | Re-trigger Greptile}

github-actions · 2026-07-08T22:22:51Z

Migration SQL Changes

Hey 👋, we've detected some migrations on this PR. Here's the SQL output for each migration, make sure they make sense:

`products/warehouse_sources/backend/migrations/0060_alter_externaldatasource_source_type_and_more.py`

BEGIN;
--
-- Alter field source_type on externaldatasource
--
-- (no-op)
--
-- Alter field source_type on pendingsourcecredential
--
-- (no-op)
COMMIT;

Last updated: 2026-07-09 10:06 UTC (d549e7e)

github-actions · 2026-07-08T22:23:06Z

🔍 Migration Risk Analysis

We've analyzed your migrations for potential risks.

Summary: 0 Safe | 1 Needs Review | 0 Blocked

⚠️ Needs Review

May have performance impact

warehouse_sources.0060_alter_externaldatasource_source_type_and_more
  └─ #1 ⚠️ AlterField
     Field alteration may cause table locks or data loss (check if changing type or constraints)
     model: externaldatasource, field: source_type, field_type: CharField
  └─ #2 ⚠️ AlterField
     Field alteration may cause table locks or data loss (check if changing type or constraints)
     model: pendingsourcecredential, field: source_type, field_type: CharField

Last updated: 2026-07-09 10:06 UTC (d549e7e)

github-actions · 2026-07-08T22:23:40Z

🤖 CI report

⚠️

Bundle size — 🔺 +941 B (+0.0%)

Uncompressed size of every built .js bundle, compared against the base branch.

Total: 71.59 MiB · 🔺 +941 B (+0.0%)

No file changed by more than 1000 B.

_{Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report}

✅ Eager graph — within budget

How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.

Root	Eager (shipped)	Δ vs base	Budget
entry (logged-out pages, app bootstrap) `src/index.tsx`	1.20 MiB · 22 files	no change	`███░░░░░░░` 27.9% of 4.29 MiB
authenticated shell (every logged-in page) `src/scenes/AuthenticatedShell.tsx`	8.05 MiB · 2,948 files	🔺 +269 B (+0.0%)	`█████████░` 87.0% of 9.25 MiB

🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx

Largest files eagerly shipped from src/index.tsx

Size	File
126.8 KiB	`../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js`
24.6 KiB	`../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js`
6.3 KiB	`../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js`
4.5 KiB	`../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js`
3.9 KiB	`../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js`
1.4 KiB	`../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js`
1.3 KiB	`src/RootErrorBoundary.tsx`
912 B	`../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js`
789 B	`src/scenes/ChunkLoadErrorBoundary.tsx`
668 B	`src/index.tsx`

Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx

Size	File
277.1 KiB	`../node_modules/.pnpm/posthog-js@1.398.2/node_modules/posthog-js/dist/rrweb.js`
266.9 KiB	`../node_modules/.pnpm/@posthog+icons@0.37.4_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js`
224.6 KiB	`src/taxonomy/core-filter-definitions-by-group.json`
213.6 KiB	`../node_modules/.pnpm/posthog-js@1.398.2/node_modules/posthog-js/dist/module.js`
160.9 KiB	`src/queries/validators.js`
154.0 KiB	`../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js`
126.8 KiB	`../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js`
106.1 KiB	`src/lib/api.ts`
93.3 KiB	`../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js`
90.5 KiB	`../node_modules/.pnpm/@tiptap+core@3.20.1_@tiptap+pm@3.20.1/node_modules/@tiptap/core/dist/index.js`

_{Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479}

⚠️

Dist folder size — 🔺 +8.0 KiB (+0.0%)

Total size of the built frontend/dist folder (all assets), compared against the base branch.

Total: 1423.09 MiB · 🔺 +8.0 KiB (+0.0%)

✅ Playwright — all passed

All tests passed.

View test results →

trunk-io · 2026-07-08T22:40:24Z

_{View Full Report ↗︎ ⋅ Docs}

posthog · 2026-07-08T22:50:29Z

👋 Visual changes detected for this PR.

Review and approve in PostHog Visual Review

If these changes are unexpected, they may be caused by a flaky test or a broken snapshot on master. Don't approve — rerun the job or wait for a fix.

posthog-local-dev · 2026-07-08T23:33:05Z

ReviewHog Alpha 🦔 If you find any issues helpful - please reply "valid", "invalid", etc., for evaluation purposes 🙏

posthog-local-dev

ReviewHog Report

Business logic

Files: products/warehouse_sources/backend/temporal/data_imports/sources/generated_configs.py, products/warehouse_sources/backend/temporal/data_imports/sources/_load_all.py, products/warehouse_sources/backend/temporal/data_imports/sources/SOURCES.md

What were the main changes

Adds empty placeholder *SourceConfig classes for all 25 new sources plus corresponding SOURCE_CONFIG_MAP entries in generated_configs.py
Registers the 25 new source modules in _load_all.py and lists them in SOURCES.md's scaffolded-sources list
Flagged regression: StripeAuthMethodConfig's stripe_secret_key/stripe_integration_id were unintentionally changed from optional (None-default) to required, which will break existing API-key or OAuth Stripe connections when parsing their nested auth config — appears to be unrelated drift pulled in by re-running the config generator, should be reverted

Feature

Files: products/warehouse_sources/backend/temporal/data_imports/sources/aikido_security/source.py, products/warehouse_sources/backend/temporal/data_imports/sources/alguna/source.py, products/warehouse_sources/backend/temporal/data_imports/sources/anthropic/source.py, products/warehouse_sources/backend/temporal/data_imports/sources/appwrite/source.py, products/warehouse_sources/backend/temporal/data_imports/sources/bland_ai/source.py, products/warehouse_sources/backend/temporal/data_imports/sources/browse_ai/source.py, products/warehouse_sources/backend/temporal/data_imports/sources/browser_use/source.py, products/warehouse_sources/backend/temporal/data_imports/sources/charthop/source.py, products/warehouse_sources/backend/temporal/data_imports/sources/cody/source.py, products/warehouse_sources/backend/temporal/data_imports/sources/cursor/source.py, products/warehouse_sources/backend/temporal/data_imports/sources/decagon/source.py, products/warehouse_sources/backend/temporal/data_imports/sources/deepgram/source.py, products/warehouse_sources/backend/temporal/data_imports/sources/elevenlabs/source.py

Issues: 1 issue

What were the main changes

Adds 13 stub source.py implementations (Aikido Security, Alguna, Anthropic, Appwrite, Bland AI, Browse AI, Browser Use, ChartHop, Cody, Cursor, Decagon, Deepgram, ElevenLabs) registered via SourceRegistry
Each source is unreleasedSource=True with empty fields, so nothing is user-visible yet — icons/config fields land in follow-up per-source PRs

…k on missing source icons The regenerated StripeAuthMethodConfig incorrectly made both stripe_secret_key and stripe_integration_id required, breaking Config.from_dict() for every existing Stripe source (API-key connections lack an integration id and OAuth connections lack a secret key). Restore both to optional to match the generator output on master. The 25 new scaffolded sources reference icon PNGs that aren't shipped yet, which would render as broken images in the "coming soon" source picker. Add an onError fallback in SourceIcon so a missing icon degrades to the placeholder hedgehog. Generated-By: PostHog Code Task-Id: 65c6cedb-aadf-4f89-914d-de1c069bbe3c

User-facing doc for the new bland_ai import source (PostHog/posthog#69517 scaffold batch). Follows the canonical source-doc template with SourceParameters and SourceTables. Generated-By: PostHog Code Task-Id: 1178b870-65b2-4a27-bd23-4c260aa64789

deployment-status-posthog · 2026-07-09T11:22:41Z

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-09 11:22 UTC	Run
prod-us	✅ Deployed	2026-07-09 11:51 UTC	Run
prod-eu	✅ Deployed	2026-07-09 11:55 UTC	Run

User-facing doc for the new bland_ai import source (PostHog/posthog#69517 scaffold batch). Follows the canonical source-doc template with SourceParameters and SourceTables. Generated-By: PostHog Code Task-Id: 1178b870-65b2-4a27-bd23-4c260aa64789

Gilbert09 self-assigned this Jul 8, 2026

github-actions Bot deployed to preview-pr-69517 July 8, 2026 22:19 View deployment

pr-assigner-resolver-posthog Bot requested a review from a team July 8, 2026 22:19

hex-security-app Bot reviewed Jul 8, 2026

View reviewed changes

Comment thread products/warehouse_sources/backend/temporal/data_imports/sources/generated_configs.py

hex-security-app Bot reviewed Jul 8, 2026

View reviewed changes

Comment thread products/warehouse_sources/backend/temporal/data_imports/sources/generated_configs.py Outdated

greptile-apps Bot reviewed Jul 8, 2026

View reviewed changes

Comment thread products/warehouse_sources/backend/temporal/data_imports/sources/generated_configs.py

posthog-local-dev Bot reviewed Jul 8, 2026

View reviewed changes

Comment thread products/warehouse_sources/backend/temporal/data_imports/sources/aikido_security/source.py

Gilbert09 mentioned this pull request Jul 9, 2026

feat(data-warehouse): scaffold 66 dev-infra and ai source stubs #69588

Merged

2 tasks

MarconLP approved these changes Jul 9, 2026

View reviewed changes

Gilbert09 added 2 commits July 9, 2026 11:02

feat(data-warehouse): scaffold 25 new saas/ai source stubs

42cf27c

Gilbert09 force-pushed the tom/scaffold-viral-saas-ai-sources branch from 7c9b3c1 to d549e7e Compare July 9, 2026 10:02

github-actions Bot deployed to preview-pr-69517 July 9, 2026 10:05 View deployment

Gilbert09 enabled auto-merge (squash) July 9, 2026 10:52

Gilbert09 mentioned this pull request Jul 9, 2026

feat(data-warehouse): implement harvey import source #69632

Merged

2 tasks

Gilbert09 merged commit 35c9e33 into master Jul 9, 2026
291 of 295 checks passed

Gilbert09 deleted the tom/scaffold-viral-saas-ai-sources branch July 9, 2026 10:55

Uh oh!

Conversation

Gilbert09 commented Jul 8, 2026

Problem

Changes

How did you test this code?

Automatic notifications

Docs update

Uh oh!

github-actions Bot commented Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦔 Hogbox preview · ✅ ready

▶ Open the preview

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Jul 8, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Migration SQL Changes

products/warehouse_sources/backend/migrations/0060_alter_externaldatasource_source_type_and_more.py

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Migration Risk Analysis

⚠️ Needs Review

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 CI report

Uh oh!

trunk-io Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

posthog Bot commented Jul 8, 2026

Uh oh!

posthog-local-dev Bot commented Jul 8, 2026

Uh oh!

posthog-local-dev Bot left a comment

Choose a reason for hiding this comment

ReviewHog Report

Business logic

Feature

Uh oh!

Uh oh!

Uh oh!

deployment-status-posthog Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploy status

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

`products/warehouse_sources/backend/migrations/0060_alter_externaldatasource_source_type_and_more.py`

github-actions Bot commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

trunk-io Bot commented Jul 8, 2026 •

edited

Loading

deployment-status-posthog Bot commented Jul 9, 2026 •

edited

Loading