feat(data-warehouse): implement unleash import source - #70002
Conversation
Fill in the scaffolded Unleash data warehouse source end to end: Admin API transport, endpoint catalog, credential validation, per-endpoint permission probing, canonical column descriptions, an SVG icon, and tests. Ships behind unreleasedSource=True at releaseStatus=alpha. Generated-By: PostHog Code Task-Id: e41299d2-ac3e-4079-bb38-d75add0ac8e4
The source now references the official unleash.svg logo, so the placeholder PNG is unused. Generated-By: PostHog Code Task-Id: e41299d2-ac3e-4079-bb38-d75add0ac8e4
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
|
Reviews (1): Last reviewed commit: "chore(data-warehouse): remove placeholde..." | Re-trigger Greptile |
🤖 CI report✅ Bundle size — no changeUncompressed size of every built Total: 70.08 MiB · no change No file changed by more than 1000 B. Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report ✅ Eager graph — within budgetHow much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy
🟢 Largest files eagerly shipped from
|
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 668 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 277.3 KiB | ../node_modules/.pnpm/posthog-js@1.399.1/node_modules/posthog-js/dist/rrweb.js |
| 266.9 KiB | ../node_modules/.pnpm/@posthog+icons@0.37.4_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 224.6 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 221.5 KiB | ../node_modules/.pnpm/posthog-js@1.399.1/node_modules/posthog-js/dist/module.js |
| 164.0 KiB | src/queries/validators.js |
| 154.0 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.9 KiB | src/lib/api.ts |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 90.6 KiB | ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Dist folder size — 🟢 -41.8 KiB (-0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1435.40 MiB · 🟢 -41.8 KiB (-0.0%)
estefaniarabadan
left a comment
There was a problem hiding this comment.
Small, isolated change to a single existing source — low-risk and self-contained, CI green. LGTM 👍
|
👋 Hey @Gilbert09! I analyzed this PR for documentation updates but noticed the Unleash connector is currently gated behind No documentation PR will be created at this time since the feature is not yet publicly available. Once the source is ready for general availability and these flags are removed, please ping me again and I'll create the documentation at This analysis was performed by the Inkeep Content Writer agent, which is maintained by the Docs and Wizard team. Please reach out on Slack for help if needed. |
Problem
PostHog can't yet pull Unleash feature-management configuration into the data warehouse. Teams that run Unleash want their flags, projects, environments, strategies, segments, and related config available alongside product analytics and experiments so they can join flag state against behavioral data.
Changes
This completes the
unleashdata warehouse import source and ships it behindunreleasedSource=TrueatreleaseStatus=alpha.The connector talks to the Unleash Admin REST API (
/api/admin) and follows the standard source split:settings.py— declarative endpoint catalog:features(flag search, paginated),projects,environments,strategies,segments,context_fields,tags,tag_types,feature_types,addons,users. Each declares its response selector and a table-wide-unique primary key (e.g.tagskeys on(type, value); flags onname).unleash.py— tracked-session transport (make_tracked_session), tenacity retry/backoff, SSRF guards on the customer-supplied instance URL (redirects refused, private hosts blocked on cloud, https enforced on cloud, parser-differential userinfo/backslash URLs rejected), credential validation, and per-endpoint permission probing.source.py— source config/form fields (instance URL + Admin API token sent as the rawAuthorizationheader, noBearerprefix), schema listing, non-retryable auth errors, and resumable-source wiring.canonical_descriptions.py— curated table/column descriptions from the official API docs for the AI agent and public docs.All endpoints sync in full refresh mode: the Admin API exposes no server-side
updated_after-style filter on these configuration list endpoints, so there is no reliable cursor to drive an incremental sync. Collections are configuration-sized, so a full pull each sync is cheap. The one paginated endpoint (features) is resumable via a persisted offset.Note
The transport, settings, credential validation, canonical descriptions, and both test modules were already present on
master. This change finalizes the connector: it replaces the placeholder service icon with the official Unleash logo (unleash.svg, pointsiconPathat it, drops the unused PNG) and removes the stale duplicateunleashentry from the scaffolded list inSOURCES.md(it already appears in the implemented table).How did you test this code?
Automated tests run locally (all passing):
products/warehouse_sources/backend/temporal/data_imports/sources/unleash/tests/— 74 tests covering URL normalization, SSRF/redirect guards, pagination + resume, row extraction, primary-key mapping, credential validation status mapping, and per-endpoint permission probing.products/warehouse_sources/backend/temporal/data_imports/sources/tests/test_source_categories.py— 1506 tests (confirmsunleashhas a valid category).ruff checkandruff format --checkclean on the changed files.I could not exercise the live Admin API (no Unleash credentials in this environment); endpoint behavior follows the current public API docs, and the pagination/sort assumptions are noted in code comments.
Docs update
The user-facing doc belongs in the separate
posthog.comrepo atcontents/docs/cdp/sources/unleash.md(matchingdocsUrl). Noposthog.comcheckout was available here, soaudit_source_docscould not be run. The doc has been drafted (canonical template,<SourceParameters />+<SourceTables />, alpha-release callout) and still needs to land inposthog.com. The source setslists_tables_without_credentials = True, so the Supported tables section will render.🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Authored with Claude Code (PostHog Code cloud task). Skills used:
/implementing-warehouse-sourcesand/documenting-warehouse-sources.On starting, the full source implementation was already merged on
masterand verified passing, so no re-implementation was possible or needed. The remaining gaps were the service icon (a red/blue placeholder glyph, not the Unleash brand) and a SOURCES.md inventory inconsistency. The official Unleash logo was pulled from the open-sourceUnleash/unleashrepo (logoPlain.svg, the icon mark) rather than a placeholder or a hardcoded Logo.dev key.unreleasedSource=True/releaseStatus=alphawere left in place per the connector's alpha state.Created with PostHog Code