feat(data-warehouse): implement inngest import source - #70715
Conversation
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
🤖 CI report✅ Bundle size — no changeUncompressed size of every built Total: 64.77 MiB · no change No file changed by more than 1000 B. Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report ✅ Eager graph — within budgetHow much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy
🟢 Largest files eagerly shipped from
|
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 762 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 281.3 KiB | ../node_modules/.pnpm/posthog-js@1.402.3/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 235.5 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 222.9 KiB | ../node_modules/.pnpm/posthog-js@1.402.3/node_modules/posthog-js/dist/module.js |
| 164.0 KiB | src/queries/validators.js |
| 154.3 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.8 KiB | src/lib/api.ts |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 92.7 KiB | ../packages/quill/packages/quill/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Dist folder size — 🔺 +2.5 KiB (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1312.49 MiB · 🔺 +2.5 KiB (+0.0%)
⚠️ Backend coverage — 98.0% of changed backend lines covered — 7 uncovered
🧪 Backend test coverage
Patch coverage — changed backend lines (products + core): ████████████████████ 98.0% (522 / 529)
| File | Patch | Uncovered changed lines |
|---|---|---|
products/warehouse_sources/backend/temporal/data_imports/sources/inngest/inngest.py |
95.9% | 114, 118–120, 281, 293, 301 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29457855823 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
demo |
███████████░░░░░░░░░ 56.2% |
1,497 / 2,663 |
tasks |
█████████████░░░░░░░ 67.4% |
25,560 / 37,895 |
signals |
████████████████░░░░ 79.1% |
19,033 / 24,073 |
data_modeling |
████████████████░░░░ 80.0% |
4,834 / 6,045 |
cdp |
████████████████░░░░ 80.7% |
3,118 / 3,864 |
notebooks |
█████████████████░░░ 84.3% |
6,343 / 7,520 |
agent_platform |
█████████████████░░░ 84.7% |
3,273 / 3,862 |
cohorts |
█████████████████░░░ 86.1% |
4,022 / 4,671 |
actions |
█████████████████░░░ 86.6% |
717 / 828 |
product_tours |
█████████████████░░░ 87.5% |
1,266 / 1,447 |
exports |
██████████████████░░ 88.3% |
6,891 / 7,800 |
conversations |
██████████████████░░ 88.9% |
16,129 / 18,133 |
dashboards |
██████████████████░░ 89.1% |
5,719 / 6,418 |
mcp_analytics |
██████████████████░░ 89.1% |
2,502 / 2,807 |
error_tracking |
██████████████████░░ 89.6% |
9,718 / 10,852 |
alerts |
██████████████████░░ 89.9% |
3,638 / 4,046 |
engineering_analytics |
██████████████████░░ 90.1% |
5,105 / 5,665 |
streamlit_apps |
██████████████████░░ 90.4% |
2,499 / 2,764 |
slack_app |
██████████████████░░ 90.6% |
9,511 / 10,503 |
marketing_analytics |
██████████████████░░ 90.8% |
11,514 / 12,684 |
product_analytics |
██████████████████░░ 91.1% |
5,599 / 6,143 |
data_warehouse |
██████████████████░░ 92.1% |
18,133 / 19,683 |
workflows |
██████████████████░░ 92.4% |
5,148 / 5,574 |
web_analytics |
███████████████████░ 92.7% |
13,624 / 14,691 |
ai_observability |
███████████████████░ 92.8% |
14,868 / 16,019 |
surveys |
███████████████████░ 92.9% |
5,687 / 6,120 |
posthog_ai |
███████████████████░ 93.2% |
1,322 / 1,418 |
approvals |
███████████████████░ 93.3% |
3,395 / 3,640 |
reminders |
███████████████████░ 93.4% |
468 / 501 |
early_access_features |
███████████████████░ 93.8% |
848 / 904 |
endpoints |
███████████████████░ 94.1% |
8,606 / 9,143 |
skills |
███████████████████░ 94.4% |
2,827 / 2,995 |
revenue_analytics |
███████████████████░ 94.5% |
3,598 / 3,809 |
review_hog |
███████████████████░ 94.6% |
6,532 / 6,905 |
logs |
███████████████████░ 95.3% |
9,528 / 9,994 |
experiments |
███████████████████░ 95.6% |
24,171 / 25,288 |
replay_vision |
███████████████████░ 95.7% |
13,354 / 13,952 |
annotations |
███████████████████░ 96.2% |
732 / 761 |
warehouse_sources |
███████████████████░ 96.2% |
230,770 / 239,807 |
feature_flags |
███████████████████░ 96.3% |
16,002 / 16,625 |
user_interviews |
███████████████████░ 96.4% |
2,242 / 2,325 |
data_catalog |
███████████████████░ 97.1% |
2,035 / 2,095 |
customer_analytics |
███████████████████░ 97.2% |
7,480 / 7,698 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 2 · PR risk: 0/10 |
Implements the Inngest warehouse source end-to-end: events (append via server-side received_after windowing), function runs (merge, discovered by fanning out over the incremental events walk), plus cancellations, environments, webhooks, and event/signing key inventories as full-refresh tables. Resumable cursor walk with pinned time windows, tracked HTTP transport, canonical table descriptions, icon, docs, and tests. Generated-By: PostHog Code Task-Id: fe7d3805-9748-42dc-b991-77c43b21515a
Webhook intake URLs are capability-bearing: anyone holding one can submit events that trigger Inngest functions. Strip them before yielding, like the event/signing key secret material, and cover it with a test. Generated-By: PostHog Code Task-Id: fe7d3805-9748-42dc-b991-77c43b21515a
Remove unreleasedSource=True so the finished source is visible in the connector catalog, and drop any test asserting the hidden state.
Long stretches of run-less events yield no rows, making the per-event runs walk look stalled; emit a progress log every 1000 events walked. A per-sync request budget was considered and deliberately not added: the events walk ordering is unverified, so stopping early could persist a watermark past events never walked (silent data loss), and no global list-runs endpoint exists to replace the per-event lookups. Generated-By: PostHog Code Task-Id: fe7d3805-9748-42dc-b991-77c43b21515a
a1600f5 to
4f11758
Compare
2 participants
Problem
Inngest (durable workflow engine: event-triggered functions with steps, retries, and flow control) was scaffolded as a warehouse source but had no sync logic. Inngest's own event and run history is plan-gated (24 hours on free plans up to 90 days on enterprise), so users can't keep long-term history or join workflow data with their product data — a synced copy in the PostHog Data warehouse solves both.
Changes
Implements the
inngestsource end-to-end behindunreleasedSource=TruewithreleaseStatus=alpha:Authorization: Bearer signkey-...), which covers both the v1 and v2 REST APIs (dashboard API keys are v2-only and can't reach events/runs). Optionalenvironmentfield targets branch environments via theX-Inngest-Envheader.events– append-only sync driven by the server-sidereceived_after/received_beforewindow (always passed explicitly, since the server default is only 1 hour) withcursor+limit=100pagination. First sync backfills up to 90 days.function_runs– merge sync. Runs have no list endpoint, so they're discovered by walking the incremental events window and fetchingGET /v1/events/{internal_id}/runsper event. Each row carries an injectedevent_received_atwatermark column, batch runs are deduped byrun_idwithin a sync, and a 1-hour default incremental lookback re-reads recent runs so statuses settle.cancellations,environments,webhooks,event_keys,signing_keys– small full-refresh inventories. The rawkeysecret returned by the v2 key endpoints is stripped before rows are yielded, so key material never lands in the warehouse.ResumableSourcecontract: declarative endpoint catalog insettings.py, transport/pagination/normalization ininngest.py, registration/schemas/credential validation insource.py, pluscanonical_descriptions.pyfor the docs/AI table catalog. Resume state (cursor + pinned window) is saved after each yielded page. All HTTP goes throughmake_tracked_session()with the signing key value-redacted and sample capture disabled (event payloads are arbitrary customer data).sort_mode="desc"so the incremental watermark persists only at successful job end. The API's ordering within a window is undocumented and I couldn't curl-verify it without credentials (endpoint shapes were verified against the live API's OpenAPI specs and unauthenticated probes; noted in code comments). Same reason the paginator defensively drops a re-returned cursor event.inngest.posthog-doc.mdxin the source directory — a posthog.com checkout wasn't available, so it needs to be copied toposthog.com/contents/docs/cdp/sources/inngest.mdandaudit_source_docsrun once it lands.How did you test this code?
Automated tests only (no live Inngest credentials were available, so no manual end-to-end sync was run):
tests/test_inngest.py(transport): window computation guards the 1-hour server default and clamps future watermarks; cursor pagination carries the pinned window on every page, terminates on partial/empty pages, saves resume state after each yielded page and not on the final one, resumes from saved state without re-deriving the window, and drops a re-returned cursor event; runs fan-out injectsevent_received_at, JSON-encodes non-string outputs, and dedupes batch runs; v2 pagination followspage.hasMorewith a repeated-cursor loop guard and strips secretkeymaterial; 429/5xx retry vs 4xx fail-fast; sort mode/partition/primary keys per endpoint.tests/test_inngest_source.py(source class): schema catalog and per-endpoint incremental/append support, credential validation plumbing (including the env header), non-retryable 401/403 mapping, resumable manager binding, and pipeline argument plumbing (blank environment normalized to None, watermark omitted on full refresh).sources/tests/,common/test/test_source_config_generator.py, 1551 tests) plusruff check/ruff formatandhogli ci:preflight --fix(0 failures).Automatic notifications
Docs update
User-facing doc included as
inngest.posthog-doc.mdx(needs to land in the posthog.com repo, see above).🤖 Agent context
Autonomy: Fully autonomous
Implemented with Claude Code following the
/implementing-warehouse-sources,/documenting-warehouse-sources, and/writing-testsskills. Endpoint behavior was verified against Inngest's published v1/v2 OpenAPI specs and unauthenticated probes of the live API (auth error shapes, route existence); decisions that couldn't be verified without credentials were made conservatively and commented:sort_mode="desc"(unknown ordering), inclusive-cursor defense, nolimiton v2 lists (undocumented maxima), and dual-spelling normalization of the eventreceivedAt/received_atfield. Chose a hand-rolled paginator over the genericrest_source.RESTClientbecause the runs table needs a per-event fan-out driven off the same windowed cursor walk as events. Skipped afunctionstable: v2 has no list-apps endpoint, so app/function metadata isn't enumerable.Created with PostHog Code