feat(data-warehouse): implement runpod import source

[Gilbert09]

Problem

RunPod is a fast-growing GPU cloud used heavily by AI startups for on-demand GPU capacity and autoscaling serverless inference. Teams running workloads there want their compute inventory and, above all, their billing history queryable next to product data in the PostHog data warehouse. The runpod source existed only as a scaffolded stub (unreleasedSource=True, no sync logic).

Why

Requested implementation of the scaffolded RunPod source so warehouse users can analyze GPU spend and infrastructure alongside their product data.

Changes

Implements the RunPod source end-to-end following the standard source.py / settings.py / runpod.py split, as a ResumableSource:

• 7 tables. Three billing histories (billing_pods, billing_endpoints, billing_network_volumes) and four inventory snapshots (pods, endpoints, templates, network_volumes), verified against RunPod's live OpenAPI spec at rest.runpod.io/v1.
• Incremental billing sync. The billing endpoints expose a genuine server-side startTime filter, so they support incremental syncs on the bucket time field with a 48h lookback re-read (open day buckets keep accumulating charges; merge dedupes on a surrogate id hashed from the bucket start + grouping dimensions, never the amounts). Inventory lists have no updated-since filter and ship full refresh only.
• Windowed, resumable fetching. RunPod's list responses are unpaginated, so billing history is walked in 90-day windows from a 2022 floor, sorted ascending per window (sort_mode="asc"), with resume state saved after each yielded window.
• Plumbing. All HTTP goes through make_tracked_session(), tenacity retries for 429/5xx, non-retryable 401/403 mapping, validate_credentials probe (403 accepted at create time for scoped keys), canonical table/column descriptions from the official API docs, lists_tables_without_credentials = True for public docs, and the official RunPod brand icon (SVG from their docs site).
• Kept behind unreleasedSource=True with releaseStatus=ReleaseStatus.ALPHA; regenerated RunPodSourceConfig; moved runpod into the Implemented table in SOURCES.md. Enum + migration already landed with the scaffold, so no new migration.

Note

I could not curl-verify the billing endpoints' filtering behavior with real credentials (no RunPod account available), so the time-window semantics follow the live OpenAPI spec conservatively: startTime is floored to UTC midnight to stay aligned with day-bucket boundaries, and merge-on-surrogate-id absorbs any boundary overlap. Noted in code comments.

The user-facing doc (contents/docs/cdp/sources/runpod.md) is written per the docs template and needs to land in the posthog.com repo; audit_source_docs reports no runpod issues against it.

How did you test this code?

• pytest products/warehouse_sources/backend/temporal/data_imports/sources/runpod/tests/ (41 passed). New coverage, per group:
  • Window walking: full refresh terminates at "now" with endTime omitted on the final window, resumes from saved state, and sorts rows ascending. Catches unbounded walks and watermark corruption.
  • Watermark handling: mid-bucket watermarks floor to UTC midnight, future-dated watermarks clamp to today. Catches bucket misalignment duplicates and wedged syncs.
  • Surrogate id: stable across amount restatements, None/empty-string dims can't collide. Catches merge-duplicate regressions.
  • Grouping params per endpoint, retry budget exhaustion on 429/5xx vs no-retry on 401, non-list response rejection, credential status mapping (200/403 valid, 401 invalid).
  • Source class: schema catalog incremental flags, config field shape, resume manager binding, partitioning, documented tables.
• pytest products/warehouse_sources/backend/temporal/data_imports/sources/tests/ + config generator tests (1566 passed) — registry-wide guards.
• ruff check --fix, ruff format, and hogli ci:preflight --fix (0 failures).
• Not done: a live sync against a real RunPod account (no credentials available in this environment).

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

Doc written for the posthog.com repo at contents/docs/cdp/sources/runpod.md (canonical template, <SourceParameters /> + <SourceTables />, alpha callout); it needs to land there since it lives outside this repo.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Built with PostHog Code following the /implementing-warehouse-sources, /documenting-warehouse-sources, and /writing-tests skills. Key decisions: modeled on the anthropic source (closest shape: API key + bucketed billing reports + entity snapshots); chose day buckets (API default, stable surrogate keys) with a 48h lookback over hour buckets; chose 90-day request windows to bound memory since the API doesn't paginate; skipped the containerregistryauth endpoint (id/name only, low warehouse value); kept the source unreleased/alpha as directed. The Stripe config drift the generator produced in this env (missing OAuth env vars) was reverted, keeping only the RunPod hunk in generated_configs.py.

---

Created with PostHog Code

[github-actions]

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

• Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
• Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

[github-actions]

🦔 Hogbox preview · ✅ ready

▶ Open the preview

🔑 Login	test@posthog.com / 12345678 (demo data)
🧩 Running	this PR's backend and frontend, on the PostHog :master base
🔗 Link	stable across rebuilds — a re-push swaps the box underneath, the URL stays
🔒 Access	tailnet only (PostHog VPN)
🛠️ Admin	inspect & debug state in hogland
💤 Idle	sleeps after ~30 min idle (snapshot to S3, zero node cost) and wakes on your next visit in ~30s, behind a brief "waking up" screen

_{commit 388ec81 · box box-d6c75c5b0651 · ready in 722s (push → usable) · build log · rebuilds on every push, torn down on close}

[github-actions]

🤖 CI report

✅ Bundle size — no change

Uncompressed size of every built .js bundle, compared against the base branch.

Total: 64.77 MiB · no change

No file changed by more than 1000 B.

_{Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report}

✅ Eager graph — within budget

How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.

Root	Eager (shipped)	Δ vs base	Budget
entry (logged-out pages, app bootstrap) src/index.tsx	1.22 MiB · 22 files	no change	███░░░░░░░ 28.4% of 4.29 MiB
authenticated shell (every logged-in page) src/scenes/AuthenticatedShell.tsx	8.13 MiB · 2,978 files	no change	█████████░ 87.9% of 9.25 MiB

🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx

Largest files eagerly shipped from src/index.tsx

Size	File
126.8 KiB	../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js
24.6 KiB	../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js
6.3 KiB	../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js
4.5 KiB	../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js
3.9 KiB	../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js
1.4 KiB	../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js
1.3 KiB	src/RootErrorBoundary.tsx
912 B	../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js
789 B	src/scenes/ChunkLoadErrorBoundary.tsx
762 B	src/index.tsx

Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx

Size	File
281.3 KiB	../node_modules/.pnpm/posthog-js@1.402.3/node_modules/posthog-js/dist/rrweb.js
267.7 KiB	../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js
235.5 KiB	src/taxonomy/core-filter-definitions-by-group.json
222.9 KiB	../node_modules/.pnpm/posthog-js@1.402.3/node_modules/posthog-js/dist/module.js
164.0 KiB	src/queries/validators.js
154.3 KiB	../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js
126.8 KiB	../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js
105.8 KiB	src/lib/api.ts
93.3 KiB	../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js
92.7 KiB	../packages/quill/packages/quill/dist/index.js

_{Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479}

✅ Dist folder size — 🔺 +3.0 KiB (+0.0%)

Total size of the built frontend/dist folder (all assets), compared against the base branch.

Total: 1312.62 MiB · 🔺 +3.0 KiB (+0.0%)

✅ Playwright — all passed

All tests passed.

View test results →

⚠️ Backend coverage — 98.0% of changed backend lines covered — 7 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ████████████████████ 98.0% (378 / 385)

File	Patch	Uncovered changed lines
products/warehouse_sources/backend/temporal/data_imports/sources/runpod/source.py	90.6%	114–115, 117
products/warehouse_sources/backend/temporal/data_imports/sources/runpod/runpod.py	96.6%	63, 104, 107, 224

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29463697062 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
demo	███████████░░░░░░░░░ 56.2%	1,497 / 2,663
tasks	█████████████░░░░░░░ 67.4%	25,560 / 37,895
signals	████████████████░░░░ 79.1%	19,035 / 24,073
data_modeling	████████████████░░░░ 80.0%	4,834 / 6,045
cdp	████████████████░░░░ 80.7%	3,118 / 3,864
notebooks	█████████████████░░░ 84.3%	6,343 / 7,520
agent_platform	█████████████████░░░ 84.7%	3,273 / 3,862
cohorts	█████████████████░░░ 86.1%	4,022 / 4,671
actions	█████████████████░░░ 86.6%	717 / 828
product_tours	█████████████████░░░ 87.5%	1,266 / 1,447
exports	██████████████████░░ 88.3%	6,891 / 7,800
conversations	██████████████████░░ 88.9%	16,129 / 18,133
dashboards	██████████████████░░ 89.1%	5,719 / 6,418
mcp_analytics	██████████████████░░ 89.1%	2,502 / 2,807
error_tracking	██████████████████░░ 89.6%	9,718 / 10,852
alerts	██████████████████░░ 89.9%	3,638 / 4,046
engineering_analytics	██████████████████░░ 90.1%	5,105 / 5,665
streamlit_apps	██████████████████░░ 90.4%	2,499 / 2,764
slack_app	██████████████████░░ 90.6%	9,511 / 10,503
marketing_analytics	██████████████████░░ 90.8%	11,514 / 12,684
product_analytics	██████████████████░░ 91.1%	5,599 / 6,143
data_warehouse	██████████████████░░ 92.1%	18,133 / 19,683
workflows	██████████████████░░ 92.4%	5,148 / 5,574
web_analytics	███████████████████░ 92.7%	13,624 / 14,691
ai_observability	███████████████████░ 92.8%	14,868 / 16,019
surveys	███████████████████░ 92.9%	5,687 / 6,120
posthog_ai	███████████████████░ 93.2%	1,322 / 1,418
approvals	███████████████████░ 93.3%	3,395 / 3,640
reminders	███████████████████░ 93.4%	468 / 501
early_access_features	███████████████████░ 93.8%	848 / 904
endpoints	███████████████████░ 94.1%	8,606 / 9,143
skills	███████████████████░ 94.4%	2,827 / 2,995
revenue_analytics	███████████████████░ 94.5%	3,598 / 3,809
review_hog	███████████████████░ 94.6%	6,532 / 6,905
logs	███████████████████░ 95.3%	9,528 / 9,994
experiments	███████████████████░ 95.6%	24,171 / 25,288
replay_vision	███████████████████░ 95.7%	13,354 / 13,952
annotations	███████████████████░ 96.2%	732 / 761
warehouse_sources	███████████████████░ 96.2%	232,919 / 241,997
feature_flags	███████████████████░ 96.3%	16,038 / 16,661
user_interviews	███████████████████░ 96.4%	2,242 / 2,325
data_catalog	███████████████████░ 97.1%	2,035 / 2,095
customer_analytics	███████████████████░ 97.2%	7,480 / 7,698

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

[trunk-io]

     

View Full Report ↗︎ ⋅ Docs

[danielcarletti]

LGTM

[Gilbert09]

The only red check is deploy preview (Hogbox preview environment), and it's an infrastructure failure unrelated to this change — not a code or test failure.

The provisioning backend fails before any of this PR's code runs, with errors like hog-exec dial failed: ... context deadline exceeded and placement failed: chunkfs mount: chunkfsd ready: not ready within 5s. The same failure appears across multiple runs (including on commits that predate this branch's changes), and a manual re-run reproduced it, so it's environment-wide rather than PR-specific. It's also not a required check — mergeStateStatus is UNSTABLE, i.e. all required checks are green and only this non-required preview job is red.

This is a backend-only data-warehouse source change with no frontend or deploy impact, so there's nothing to fix here in code. The preview job retries on each push and should recover once the preview infra is healthy.

All 168 required checks pass, including the warehouse-sources product tests and the Temporal Django tests that exercise this source.

_{🦉 via talyn.dev}

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[Gilbert09]

Addressed the startup-command finding in f930932: dockerStartCmd and dockerEntrypoint are now in sensitive_keys for pods, endpoints, and templates, so the existing recursive strip removes them (including when nested under an embedded template) and HTTP sample capture stays disabled for those endpoints. The stripping test now guards the full key set.

The failing Visual regression tests - chromium (12/16) shard is unrelated to this PR — it times out waiting for [data-attr="account-expansion"] in products/customer_analytics/.../AccountsTab.stories.tsx (RowExpandedUsagePopulated), a story this diff doesn't touch (backend warehouse source + one SVG icon). The new push re-runs it.

[posthog]

👋 Visual changes detected for this PR.

Review and approve in PostHog Visual Review

If these changes are unexpected, they may be caused by a flaky test or a broken snapshot on master. Don't approve — rerun the job or wait for a fix.

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-16 03:53 UTC	Run
prod-us	✅ Deployed	2026-07-16 04:06 UTC	Run
prod-eu	✅ Deployed	2026-07-16 04:06 UTC	Run