feat(data-warehouse): implement pulumi_cloud import source by Gilbert09 · Pull Request #71203 · PostHog/posthog

Gilbert09 · 2026-07-15T16:38:26Z

Problem

The Pulumi Cloud source was scaffolded in #71137 as a hidden stub with no sync logic. Infra teams want their Pulumi data (stacks, update history, deployments, resources, audit logs) in the warehouse to compute DORA-style metrics for infrastructure changes (deployment frequency, change-failure rate, lead time) joined against product and incident data.

Why

Fills in the scaffolded pulumi_cloud stub so the connector actually syncs and is visible to users, as part of the eng/support sources batch.

Changes

Implements the source end-to-end on the ResumableSource base, following the source.py / settings.py / pulumi_cloud.py split:

Five tables: stacks, stack_updates (fan-out per stack), deployments (org-level listing), audit_logs, and resources (Pulumi Insights resource index).
Incremental sync where the API genuinely supports it:
- audit_logs uses the v2 endpoint's server-side startTime lower bound.
- stack_updates pages newest-first and stops client-side once a page predates the watermark, so incremental runs don't re-walk each stack's full history. Both re-pull a short trailing overlap that merge dedupes.
- stacks, deployments, and resources are current-state snapshots, so they ship full refresh.
Resumable state per pagination style (continuation token, page number, search cursor, completed-stack keys for the fan-out), saved after each yielded batch so a crash re-yields rather than skips.
All HTTP goes through make_tracked_session() with the token registered for redaction and sample capture disabled (responses carry arbitrary infra data). Non-retryable 401/403 mapped to actionable user messages.
Ships visible with releaseStatus=alpha (the scaffold's unreleasedSource=True is removed), plus canonical_descriptions.py, lists_tables_without_credentials, and the regenerated PulumiCloudSourceConfig (access token + organization).
SOURCES.md moved to the Implemented table; user-facing doc added.

Endpoint parameters, response shapes, and ordering were verified against the live OpenAPI spec Pulumi serves at api.pulumi.com/api/openapi/pulumi-spec.json (the docs pages are generated from it). Two behaviors could not be verified without an authenticated account and are handled conservatively with code comments: the resource-search cursor termination signal (we stop unless a next link is present) and the default ordering of the org deployments listing (full refresh, so ordering only affects sort_mode).

Note

The doc is committed in-tree as pulumi_cloud.posthog-doc.mdx (aviator-style) because no posthog.com checkout was available; it needs to land in the posthog.com repo at contents/docs/cdp/sources/pulumi-cloud.md, after which audit_source_docs should pass.

How did you test this code?

pytest products/warehouse_sources/.../pulumi_cloud/tests/ — 53 tests pass. They cover: pagination termination and resume for each of the four pagination styles (a broken terminator loops forever or drops pages), the client-side watermark stop on stack_updates (without it every incremental sync re-fetches full history), the server-side startTime bound on audit logs, update-row flattening including dropping the raw deployment state snapshot (which would bloat rows) and injecting the stack coordinates the composite primary key needs, fan-out resume via completed-stack keys, watermark coercion, 429/5xx retry vs 401 fail-fast, and per-endpoint sort mode / partition / primary keys.
Registry-wide source tests pass (test_source_categories, test_generated_configs — the two stripe failures there pre-exist on the base branch).
ruff check / ruff format clean; hogli ci:preflight --fix passes.
I could not run a live sync against a real Pulumi Cloud account (no credentials in this environment); unauthenticated probes confirmed the base URL, headers, and 401 error shape.

Automatic notifications

Publish to changelog?
Alert Sales and Marketing teams?

Docs update

Doc included in-tree (see note above) pending a posthog.com PR.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Written with PostHog Code (Claude). Skills invoked: implementing-warehouse-sources, documenting-warehouse-sources, writing-tests. Modeled on the aviator source (closest recent eng-category reference). Key decisions: used audit logs v2 instead of v1 because v2's startTime is a documented lower bound (v1's is an upper bound, unusable as an incremental cursor); used the org-level deployments listing to avoid a per-stack fan-out; kept stack_updates on a composite (orgName, projectName, stackName, version) key since version is stack-scoped; defaulted audit_logs off in the schema picker because the endpoint is tier-gated and would fail most first syncs; deferred webhook support (Pulumi org/stack webhooks are API-manageable, but payload shapes couldn't be verified against a live account, and webhook rows must match pull-path row shapes).

Created with PostHog Code

github-actions · 2026-07-15T16:42:13Z

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

github-actions · 2026-07-15T16:44:07Z

🤖 CI report

✅ Bundle size — no change

Uncompressed size of every built .js bundle, compared against the base branch.

Total: 64.82 MiB · no change

No file changed by more than 1000 B.

_{Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report}

✅ Eager graph — within budget

How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.

Root	Eager (shipped)	Δ vs base	Budget
entry (logged-out pages, app bootstrap) `src/index.tsx`	1.22 MiB · 22 files	no change	`███░░░░░░░` 28.4% of 4.29 MiB
authenticated shell (every logged-in page) `src/scenes/AuthenticatedShell.tsx`	8.15 MiB · 2,979 files	no change	`█████████░` 88.0% of 9.25 MiB

🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx

Largest files eagerly shipped from src/index.tsx

Size	File
126.8 KiB	`../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js`
24.6 KiB	`../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js`
6.3 KiB	`../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js`
4.5 KiB	`../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js`
3.9 KiB	`../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js`
1.4 KiB	`../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js`
1.3 KiB	`src/RootErrorBoundary.tsx`
912 B	`../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js`
789 B	`src/scenes/ChunkLoadErrorBoundary.tsx`
762 B	`src/index.tsx`

Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx

Size	File
281.1 KiB	`../node_modules/.pnpm/posthog-js@1.401.0/node_modules/posthog-js/dist/rrweb.js`
267.7 KiB	`../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js`
235.5 KiB	`src/taxonomy/core-filter-definitions-by-group.json`
222.7 KiB	`../node_modules/.pnpm/posthog-js@1.401.0/node_modules/posthog-js/dist/module.js`
164.0 KiB	`src/queries/validators.js`
154.3 KiB	`../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js`
126.8 KiB	`../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js`
106.1 KiB	`src/lib/api.ts`
93.3 KiB	`../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js`
92.7 KiB	`../packages/quill/packages/quill/dist/index.js`

_{Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479}

✅ Dist folder size — no change

Total size of the built frontend/dist folder (all assets), compared against the base branch.

Total: 1312.86 MiB · no change

⚠️

Backend coverage — 98.0% of changed backend lines covered — 11 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ████████████████████ 98.0% (611 / 622)

File	Patch	Uncovered changed lines
`products/warehouse_sources/backend/temporal/data_imports/sources/pulumi_cloud/pulumi_cloud.py`	95.8%	73, 159, 190, 250, 275, 307, 361, 409, 441, 491
`products/warehouse_sources/backend/temporal/data_imports/sources/pulumi_cloud/tests/test_pulumi_cloud.py`	99.6%	186

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29545077275 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
`demo`	`███████████░░░░░░░░░` 56.2%	1,497 / 2,663
`tasks`	`█████████████░░░░░░░` 67.0%	25,838 / 38,560
`signals`	`████████████████░░░░` 79.1%	19,033 / 24,073
`data_modeling`	`████████████████░░░░` 80.0%	4,834 / 6,045
`cdp`	`████████████████░░░░` 80.7%	3,118 / 3,864
`notebooks`	`█████████████████░░░` 84.8%	6,903 / 8,141
`cohorts`	`█████████████████░░░` 86.2%	4,065 / 4,717
`agent_platform`	`█████████████████░░░` 86.4%	3,807 / 4,405
`actions`	`█████████████████░░░` 86.6%	717 / 828
`product_tours`	`█████████████████░░░` 87.5%	1,266 / 1,447
`exports`	`██████████████████░░` 88.4%	6,904 / 7,813
`conversations`	`██████████████████░░` 89.0%	16,152 / 18,156
`mcp_analytics`	`██████████████████░░` 89.2%	2,514 / 2,819
`dashboards`	`██████████████████░░` 89.3%	5,841 / 6,540
`engineering_analytics`	`██████████████████░░` 89.3%	5,407 / 6,053
`error_tracking`	`██████████████████░░` 89.6%	9,790 / 10,925
`alerts`	`██████████████████░░` 90.0%	3,655 / 4,063
`streamlit_apps`	`██████████████████░░` 90.4%	2,499 / 2,764
`stamphog`	`██████████████████░░` 90.4%	3,503 / 3,874
`slack_app`	`██████████████████░░` 90.6%	9,511 / 10,503
`marketing_analytics`	`██████████████████░░` 90.8%	11,514 / 12,684
`product_analytics`	`██████████████████░░` 91.2%	5,666 / 6,212
`data_warehouse`	`██████████████████░░` 92.1%	18,216 / 19,776
`workflows`	`██████████████████░░` 92.4%	5,148 / 5,574
`web_analytics`	`███████████████████░` 92.7%	13,535 / 14,597
`ai_observability`	`███████████████████░` 92.8%	14,913 / 16,064
`surveys`	`███████████████████░` 92.9%	5,689 / 6,122
`posthog_ai`	`███████████████████░` 93.2%	1,322 / 1,418
`approvals`	`███████████████████░` 93.3%	3,395 / 3,640
`reminders`	`███████████████████░` 93.4%	468 / 501
`early_access_features`	`███████████████████░` 93.8%	848 / 904
`endpoints`	`███████████████████░` 94.1%	8,606 / 9,143
`revenue_analytics`	`███████████████████░` 94.5%	3,598 / 3,809
`skills`	`███████████████████░` 94.5%	2,881 / 3,049
`review_hog`	`███████████████████░` 94.6%	6,769 / 7,154
`logs`	`███████████████████░` 95.3%	9,561 / 10,032
`experiments`	`███████████████████░` 95.6%	24,407 / 25,517
`replay_vision`	`███████████████████░` 95.7%	13,425 / 14,026
`annotations`	`███████████████████░` 96.2%	732 / 761
`feature_flags`	`███████████████████░` 96.3%	16,141 / 16,763
`warehouse_sources`	`███████████████████░` 96.3%	246,791 / 256,178
`user_interviews`	`███████████████████░` 96.4%	2,242 / 2,325
`data_catalog`	`███████████████████░` 97.2%	2,117 / 2,179
`customer_analytics`	`███████████████████░` 97.2%	7,480 / 7,698
`pulse`	`████████████████████` 98.3%	1,582 / 1,609

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

✅ Django migration risk — no migrations to analyze

No Django migrations need risk analysis.

ℹ️ Docs preview — preview build triggered

Docs from this PR will be published at posthog.com.

Project	Preview	Updated (UTC)
posthog.com	Open preview	Jul 15, 2026, 9:20 PM

The preview should be ready in about 10 minutes. Open the preview at /handbook/engineering/.

github-actions · 2026-07-15T16:57:28Z

🦔 Hogbox preview · ✅ ready

▶ Open the preview


🔑 Login	`test@posthog.com` / `12345678` (demo data)
🧩 Running	this PR's backend and frontend, on the PostHog `:master` base
🔗 Link	stable across rebuilds — a re-push swaps the box underneath, the URL stays
🔒 Access	tailnet only (PostHog VPN)
🛠️ Admin	inspect & debug state in hogland
💤 Idle	sleeps after ~30 min idle (snapshot to S3, zero node cost) and wakes on your next visit in ~30s, behind a brief "waking up" screen

_{commit a0405d7 · box box-bfb7dba90009 · ready in 975s (push → usable) · build log · rebuilds on every push, torn down on close}

trunk-io · 2026-07-15T17:03:28Z

_{View Full Report ↗︎ ⋅ Docs}

danielcarletti

LGTM

veria-ai · 2026-07-15T17:35:17Z

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 5 · PR risk: 0/10

Implements the Pulumi Cloud warehouse source end-to-end: five endpoint tables (stacks, stack_updates, deployments, audit_logs, resources) with resumable pagination, incremental sync for audit logs (server-side startTime lower bound) and stack updates (newest-first client-side stop at the watermark), canonical table/column descriptions, tests, and the user-facing doc. Ships visible with releaseStatus=alpha. Generated-By: PostHog Code Task-Id: 1ed945b0-ca42-42e3-a40c-06d4eaca535e

Pulumi audit events carry no server-assigned id. Keying on the (timestamp, event, description) composite collapsed genuinely distinct events that differ only by actor, token, or source IP into one row on the merge upsert. Derive a stable synthetic `event_id` by hashing the full event payload instead, so distinct events stay apart while identical rows re-pulled inside the overlap window still dedupe. Generated-By: PostHog Code Task-Id: 455a165e-a890-4cd4-a2a3-a7ded18a2fba

`organization` selects which Pulumi tenant the stored access token queries. Without listing it in `connection_host_fields`, an editor could retarget the preserved token at another organization the token can access without re-entering the redacted secret. Override `connection_host_fields` to return `["organization"]`, matching other tenant-scoped sources (e.g. azure_devops). Generated-By: PostHog Code Task-Id: 455a165e-a890-4cd4-a2a3-a7ded18a2fba

Pulumi stack updates carry a `config` map whose secret entries return the encrypted value in `string` alongside `secret: true`. Copying that raw map into warehouse rows would let anyone with warehouse read access export the ciphertexts and attempt offline recovery against passphrase-protected stacks. Drop the `string` field for secret entries before yielding while keeping the metadata (that the key exists and is a secret) and all non-secret values. Generated-By: PostHog Code Task-Id: 455a165e-a890-4cd4-a2a3-a7ded18a2fba

Pulumi's UpdateInfo response includes `requestedByToken`, the access token used to request the update. Merging the outer update object into the row copied that live credential into the warehouse, where anyone with query access could retrieve it and act as the token owner. Drop the field before yielding; the `requestedBy` user object is retained. Generated-By: PostHog Code Task-Id: 455a165e-a890-4cd4-a2a3-a7ded18a2fba

… startTime In incremental mode the stack_updates paginator stops once a page reaches below the watermark, but that signal only exists when the page carries integer startTimes. A full page of updates without a usable startTime gave no stop signal and would keep paging to MAX_PAGES. Bail after such a page instead. Rows without a usable startTime are still yielded (not dropped), so an untimestamped update is never silently lost; merge dedupes the re-yielded rows on the primary key. Generated-By: PostHog Code Task-Id: 455a165e-a890-4cd4-a2a3-a7ded18a2fba

Pulumi's UpdateInfo `environment` map records the environment variables supplied to an update, which can include cloud credentials, CI tokens, or PULUMI_CONFIG_PASSPHRASE. Copying `info` wholesale made those values queryable by anyone with warehouse read access. Drop `environment` before yielding, alongside the existing deployment/requestedByToken/secret-config redaction. Generated-By: PostHog Code Task-Id: 455a165e-a890-4cd4-a2a3-a7ded18a2fba

Org deployment records embed the updates they ran, and each update's `environment` map can hold cloud credentials or PULUMI_CONFIG_PASSPHRASE. These rows skip `_flatten_update`, so strip the field on the yield path just like stack-update rows before they reach a queryable warehouse row. Generated-By: PostHog Code Task-Id: 55827811-5db7-41e3-ae19-940268a757e9

deployment-status-posthog · 2026-07-17T07:43:37Z

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-17 07:43 UTC	Run
prod-us	✅ Deployed	2026-07-17 07:57 UTC	Run
prod-eu	✅ Deployed	2026-07-17 07:57 UTC	Run

Gilbert09 self-assigned this Jul 15, 2026

pr-assigner-resolver-posthog Bot requested a review from a team July 15, 2026 16:39

github-actions Bot had a problem deploying to preview-pr-71203 July 15, 2026 16:57 Failure

danielcarletti approved these changes Jul 15, 2026

View reviewed changes

veria-ai Bot reviewed Jul 15, 2026

View reviewed changes

Comment thread products/warehouse_sources/backend/temporal/data_imports/sources/pulumi_cloud/settings.py Outdated

github-actions Bot deployed to preview-pr-71203 July 15, 2026 18:03 View deployment

veria-ai Bot reviewed Jul 15, 2026

View reviewed changes

Comment thread products/warehouse_sources/backend/temporal/data_imports/sources/pulumi_cloud/source.py

veria-ai Bot reviewed Jul 15, 2026

View reviewed changes