feat(data-warehouse): implement koyeb import source by Gilbert09 · Pull Request #70718 · PostHog/posthog

Gilbert09 · 2026-07-14T13:07:44Z

Problem

The Koyeb data warehouse source existed only as a scaffolded stub (registered with unreleasedSource=True, no fields, no sync logic). Users can't pull their Koyeb deployment and infrastructure data into the warehouse.

Why: fill in the scaffolded Koyeb connector so engineering/DevOps data (apps, services, deployments, instances, event streams, usage) can be joined with product analytics in the warehouse.

Changes

Implements the Koyeb source end-to-end following the source.py / settings.py / koyeb.py architecture contract:

16 tables declared in settings.py: apps, services, deployments, regional_deployments, instances, domains, secrets (metadata only, values are never returned by Koyeb's list API), volumes, snapshots, organization_members, activities, the four per-entity event streams, and usage_details.
ResumableSource with offset+limit pagination (Koyeb's single pagination model), terminating on has_next=false or a short page for the replies that don't carry has_next. The offset checkpoints to Redis after each yielded page so interrupted syncs resume instead of restarting.
Incremental sync only where a genuine server-side filter exists: instances via the documented starting_time param on created_at. Everything else is full refresh, since Koyeb exposes no updated-since filters and re-paging an offset list costs the same as a full refresh. Endpoints that accept order are always requested asc so offsets stay stable mid-sync.
usage_details always sends the required starting_time/ending_time window (fixed floor to now, held constant across a run's pages) and uses a composite [instance_id, started_at] primary key since usage rows have no id.
All outbound HTTP rides make_tracked_session(), with tenacity retries on 429/5xx and get_non_retryable_errors() stopping syncs on 401/403.
canonical_descriptions.py documents every table/column from the Koyeb API reference, and lists_tables_without_credentials = True lets the public docs render the table catalog.
Datetime month partitioning on stable creation-time fields (created_at / when / started_at).
Icon at frontend/public/services/koyeb.svg, built from the official Koyeb logomark.
SOURCES.md updated (koyeb moved from Scaffolded to Implemented).
Stays behind unreleasedSource=True with releaseStatus=alpha until exercised against a live account.

Note

I had no Koyeb API credentials in this environment, so authenticated behaviors that the implementing-warehouse-sources skill asks to curl-verify (whether order=asc and starting_time are honored rather than silently ignored) are taken from the public swagger spec (api.prod.koyeb.com/public.swagger.json) plus unauthenticated probes of the live API. The conservative consequences are documented in code comments: only instances is incremental, and the paginator never relies on the time filter for termination. Worth a smoke test against a real Koyeb org before de-flagging.

How did you test this code?

pytest products/warehouse_sources/backend/temporal/data_imports/sources/koyeb/tests/ (55 tests) plus the registry-wide suites in sources/tests/ (1606 passed): source categories, generated configs.
tests/test_koyeb.py (transport): paginator termination across the three reply shapes (has_next, short page, exact-multiple + empty page) — catches the runaway/premature-stop regressions; resume-from-saved-offset and checkpoint-after-yield ordering — catches state saved before a page is yielded (data loss on resume); incremental instances sending starting_time and full-refresh omitting it; RFC 3339 formatting of naive/aware/offset datetimes — Koyeb rejects non-RFC 3339 values; credential status mapping.
tests/test_koyeb_source.py (source class): endpoint catalog and incremental flags, non-retryable 401/403 matching vs retryable 429/5xx/timeout, resumable manager binding, and source_for_pipeline plumbing (including dropping a stale watermark when incremental is off).
hogli ci:preflight --fix passes; ruff check / ruff format clean. I did not run a live sync (no credentials).

Automatic notifications

Publish to changelog?
Alert Sales and Marketing teams?

Docs update

The posthog.com doc (contents/docs/cdp/sources/koyeb.md) needs to land in the posthog.com repo — no checkout was available in this environment, so audit_source_docs couldn't run. docsUrl is already set to https://posthog.com/docs/cdp/sources/koyeb to match. Ready-to-commit doc content:

contents/docs/cdp/sources/koyeb.md

---
title: Linking Koyeb as a source
sidebar: Docs
showTitle: true
availability: { free: full, selfServe: full, enterprise: full }
sourceId: Koyeb
beta: true
---

import SourceSetupIntro from "../_snippets/source-setup-intro.mdx"
import SyncModes from "../_snippets/sync-modes.mdx"
import TroubleshootingLink from "../_snippets/dw-troubleshooting-link.mdx"
import AlphaRelease from "../_snippets/alpha-release.mdx"

<AlphaRelease />

The Koyeb connector syncs your Koyeb apps, services, deployments, instances, event streams, and usage details into the PostHog Data warehouse, so you can join deployment and infrastructure data with your product analytics.

## Prerequisites

A Koyeb account with access to the organization you want to sync. Any member who can create an API token can connect the source.

## Adding a data source

<SourceSetupIntro />

You need a Koyeb API token:

1. In the [Koyeb console](https://app.koyeb.com), open [API settings](https://app.koyeb.com/user/settings/api).
2. Click **Create API token**, give it a name, and copy the token.

Tokens are scoped to the organization they were created in, so create the token in the organization you want to sync.

## Sync modes

<SyncModes />

Most Koyeb tables are small configuration entities and sync as full refresh. The `instances` table supports incremental sync on `created_at`.

## Configuration

<SourceParameters />

## Supported tables

<SourceTables />

## Troubleshooting

If a sync fails with an authorization error, the API token was revoked or belongs to a different organization. Create a new token in the Koyeb console and update the source credentials.

<TroubleshootingLink />

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Authored with Claude Code (PostHog Code cloud task). Skills invoked: implementing-warehouse-sources, documenting-warehouse-sources, writing-tests.
Endpoint behavior was verified against the live public swagger spec and unauthenticated probes (401 shapes, base host); Vercel (closest-shape source) was used as the structural reference.
Decisions: shipped usage_details as full refresh rather than incremental because its rows mutate while an instance is running (duration_seconds accrues) and row identity under windowed queries couldn't be verified without credentials; skipped a partition key on organization_members because joined_at can move on re-invite; reverted an unrelated Stripe drift that generate:source-configs produced (the committed generated file predates current generator output for Stripe's auth block — left untouched for a separate fix).
Note: an earlier abandoned attempt exists on the posthog-code/koyeb-warehouse-source branch (no PR); it can be deleted once this lands.

Created with PostHog Code

github-actions · 2026-07-14T13:07:55Z

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

github-actions · 2026-07-14T13:08:16Z

🦔 Hogbox preview · ✅ ready

▶ Open the preview


🔑 Login	`test@posthog.com` / `12345678` (demo data)
🧩 Running	this PR's backend and frontend, on the PostHog `:master` base
🔗 Link	stable across rebuilds — a re-push swaps the box underneath, the URL stays
🔒 Access	tailnet only (PostHog VPN)
🛠️ Admin	inspect & debug state in hogland
💤 Idle	sleeps after ~30 min idle (snapshot to S3, zero node cost) and wakes on your next visit in ~30s, behind a brief "waking up" screen

_{commit 565bd31 · box box-db817e6e1d75 · ready in 1901s (push → usable) · build log · rebuilds on every push, torn down on close}

github-actions · 2026-07-14T13:15:13Z

🤖 CI report

✅ Bundle size — no change

Uncompressed size of every built .js bundle, compared against the base branch.

Total: 64.77 MiB · no change

No file changed by more than 1000 B.

_{Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report}

✅ Eager graph — within budget

How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.

Root	Eager (shipped)	Δ vs base	Budget
entry (logged-out pages, app bootstrap) `src/index.tsx`	1.22 MiB · 22 files	no change	`███░░░░░░░` 28.4% of 4.29 MiB
authenticated shell (every logged-in page) `src/scenes/AuthenticatedShell.tsx`	8.13 MiB · 2,978 files	no change	`█████████░` 87.9% of 9.25 MiB

🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx

Largest files eagerly shipped from src/index.tsx

Size	File
126.8 KiB	`../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js`
24.6 KiB	`../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js`
6.3 KiB	`../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js`
4.5 KiB	`../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js`
3.9 KiB	`../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js`
1.4 KiB	`../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js`
1.3 KiB	`src/RootErrorBoundary.tsx`
912 B	`../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js`
789 B	`src/scenes/ChunkLoadErrorBoundary.tsx`
762 B	`src/index.tsx`

Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx

Size	File
281.3 KiB	`../node_modules/.pnpm/posthog-js@1.402.2/node_modules/posthog-js/dist/rrweb.js`
267.7 KiB	`../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js`
235.5 KiB	`src/taxonomy/core-filter-definitions-by-group.json`
222.9 KiB	`../node_modules/.pnpm/posthog-js@1.402.2/node_modules/posthog-js/dist/module.js`
164.0 KiB	`src/queries/validators.js`
154.3 KiB	`../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js`
126.8 KiB	`../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js`
105.8 KiB	`src/lib/api.ts`
93.3 KiB	`../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js`
92.7 KiB	`../packages/quill/packages/quill/dist/index.js`

_{Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479}

✅ Dist folder size — 🔺 +330 B (+0.0%)

Total size of the built frontend/dist folder (all assets), compared against the base branch.

Total: 1312.45 MiB · 🔺 +330 B (+0.0%)

✅ Playwright — all passed

All tests passed.

View test results →

⚠️

Backend coverage — 96.0% of changed backend lines covered — 12 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ███████████████████░ 96.0% (378 / 390)

File	Patch	Uncovered changed lines
`products/warehouse_sources/backend/temporal/data_imports/sources/koyeb/koyeb.py`	91.1%	61, 129, 134–135, 137–139, 141, 219–220
`products/warehouse_sources/backend/temporal/data_imports/sources/koyeb/source.py`	92.6%	76, 80

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29452350474 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
`demo`	`███████████░░░░░░░░░` 56.2%	1,497 / 2,663
`tasks`	`█████████████░░░░░░░` 67.4%	25,560 / 37,895
`signals`	`████████████████░░░░` 79.1%	19,037 / 24,073
`data_modeling`	`████████████████░░░░` 80.0%	4,834 / 6,045
`cdp`	`████████████████░░░░` 80.7%	3,118 / 3,864
`notebooks`	`█████████████████░░░` 84.3%	6,343 / 7,520
`agent_platform`	`█████████████████░░░` 84.7%	3,273 / 3,862
`cohorts`	`█████████████████░░░` 86.1%	4,022 / 4,671
`actions`	`█████████████████░░░` 86.6%	717 / 828
`product_tours`	`█████████████████░░░` 87.5%	1,266 / 1,447
`exports`	`██████████████████░░` 88.3%	6,891 / 7,800
`conversations`	`██████████████████░░` 88.9%	16,129 / 18,133
`dashboards`	`██████████████████░░` 89.1%	5,719 / 6,418
`mcp_analytics`	`██████████████████░░` 89.1%	2,502 / 2,807
`error_tracking`	`██████████████████░░` 89.6%	9,718 / 10,852
`alerts`	`██████████████████░░` 89.9%	3,638 / 4,046
`engineering_analytics`	`██████████████████░░` 90.1%	5,105 / 5,665
`streamlit_apps`	`██████████████████░░` 90.4%	2,499 / 2,764
`slack_app`	`██████████████████░░` 90.6%	9,511 / 10,503
`marketing_analytics`	`██████████████████░░` 90.8%	11,514 / 12,684
`product_analytics`	`██████████████████░░` 91.1%	5,599 / 6,143
`data_warehouse`	`██████████████████░░` 92.1%	18,133 / 19,683
`workflows`	`██████████████████░░` 92.4%	5,148 / 5,574
`web_analytics`	`███████████████████░` 92.7%	13,624 / 14,691
`ai_observability`	`███████████████████░` 92.8%	14,868 / 16,019
`surveys`	`███████████████████░` 92.9%	5,660 / 6,094
`posthog_ai`	`███████████████████░` 93.2%	1,322 / 1,418
`approvals`	`███████████████████░` 93.3%	3,395 / 3,640
`reminders`	`███████████████████░` 93.4%	468 / 501
`early_access_features`	`███████████████████░` 93.8%	848 / 904
`endpoints`	`███████████████████░` 94.1%	8,606 / 9,143
`skills`	`███████████████████░` 94.4%	2,827 / 2,995
`revenue_analytics`	`███████████████████░` 94.5%	3,598 / 3,809
`review_hog`	`███████████████████░` 94.6%	6,529 / 6,902
`logs`	`███████████████████░` 95.3%	9,528 / 9,994
`experiments`	`███████████████████░` 95.6%	24,139 / 25,256
`replay_vision`	`███████████████████░` 95.7%	13,337 / 13,935
`annotations`	`███████████████████░` 96.2%	732 / 761
`warehouse_sources`	`███████████████████░` 96.2%	225,964 / 234,879
`feature_flags`	`███████████████████░` 96.3%	16,002 / 16,625
`user_interviews`	`███████████████████░` 96.4%	2,242 / 2,325
`data_catalog`	`███████████████████░` 97.1%	2,034 / 2,095
`customer_analytics`	`███████████████████░` 97.2%	7,480 / 7,698

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

veria-ai · 2026-07-14T13:27:29Z

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

trunk-io · 2026-07-14T13:28:21Z

_{View Full Report ↗︎ ⋅ Docs}

danielcarletti

LGTM

Gilbert09 · 2026-07-14T17:38:45Z

Heads-up on the one red check: deploy preview is failing on preview-environment infrastructure, not on anything in this PR. Two consecutive runs failed while bringing up the ephemeral preview sandbox — first an httpx.ReadTimeout, then hog-exec dial failed: ... connect: no route to host — before any of this branch's code runs. It's a non-required check (the PR's merge state is UNSTABLE, not BLOCKED), so it doesn't gate merge. All required checks — backend Django matrix, warehouse-sources product tests, Jest, and the full visual-regression suite — are green.

_{🦉 via talyn.dev}

Implements the Koyeb warehouse source end-to-end: declarative endpoint catalog (16 tables), offset-based resumable pagination via the tracked HTTP transport, incremental sync for instances via the server-side starting_time filter, canonical table/column descriptions for docs enrichment, and full source + transport test suites. Generated-By: PostHog Code Task-Id: 4de80210-ae0f-4387-a2ad-9ef0b05cf6e6

Generated-By: PostHog Code Task-Id: 4de80210-ae0f-4387-a2ad-9ef0b05cf6e6

Deployment rows embed a `definition` whose `env[].value` and `config_files[].content` carry plaintext application secrets. Yielding them unchanged would let anyone with warehouse-query access read credentials they cannot see in Koyeb itself. Redact those values before persisting while keeping the surrounding structure (env keys, secret references, file paths) so the rows stay useful. Generated-By: PostHog Code Task-Id: a084024d-9dcd-4163-b9f2-7cf96735e008

Remove unreleasedSource=True so the finished source is visible in the connector catalog, and drop any test asserting the hidden state.

…le capture The definition scrub runs after _fetch_page returns, but opt-in HTTP sample capture stores the raw response body at the adapter layer, and its name-based scrubbers can't recognise plaintext env values or config-file content. Create the session with capture=False for secret-scrubbed endpoints (the documented seam, same as the OAuth token-exchange path) so those bodies never land in captured samples; requests stay metered and logged. Generated-By: PostHog Code Task-Id: 4de80210-ae0f-4387-a2ad-9ef0b05cf6e6

deployment-status-posthog · 2026-07-15T23:30:51Z

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-15 23:30 UTC	Run
prod-us	✅ Deployed	2026-07-15 23:48 UTC	Run
prod-eu	✅ Deployed	2026-07-15 23:48 UTC	Run

Gilbert09 self-assigned this Jul 14, 2026

pr-assigner-resolver-posthog Bot requested a review from a team July 14, 2026 13:08

github-actions Bot had a problem deploying to preview-pr-70718 July 14, 2026 13:08 Failure

github-actions Bot had a problem deploying to preview-pr-70718 July 14, 2026 13:26 Failure

veria-ai Bot reviewed Jul 14, 2026

View reviewed changes

Comment thread products/warehouse_sources/backend/temporal/data_imports/sources/koyeb/koyeb.py

github-actions Bot requested a deployment to preview-pr-70718 July 14, 2026 16:20 In progress

github-actions Bot had a problem deploying to preview-pr-70718 July 14, 2026 16:21 Failure

hex-security-app Bot reviewed Jul 14, 2026

View reviewed changes

Comment thread products/warehouse_sources/backend/temporal/data_imports/sources/koyeb/koyeb.py

github-actions Bot requested a deployment to preview-pr-70718 July 14, 2026 16:32 In progress

github-actions Bot deployed to preview-pr-70718 July 14, 2026 16:36 View deployment

danielcarletti approved these changes Jul 14, 2026

View reviewed changes

Gilbert09 added 5 commits July 15, 2026 22:33

fix(data-warehouse): annotate heterogeneous test fixture for mypy

d225714

Generated-By: PostHog Code Task-Id: 4de80210-ae0f-4387-a2ad-9ef0b05cf6e6

fix(data-warehouse): ship koyeb source released

e1f378b

Remove unreleasedSource=True so the finished source is visible in the connector catalog, and drop any test asserting the hidden state.

Gilbert09 force-pushed the posthog-code/koyeb-import-source branch from 582df77 to 565bd31 Compare July 15, 2026 21:33

github-actions Bot deployed to preview-pr-70718 July 15, 2026 21:46 View deployment

talyn-app Bot merged commit 2aa50ff into master Jul 15, 2026
257 checks passed

talyn-app Bot deleted the posthog-code/koyeb-import-source branch July 15, 2026 22:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(data-warehouse): implement koyeb import source - #70718

feat(data-warehouse): implement koyeb import source#70718
talyn-app[bot] merged 5 commits into
masterfrom
posthog-code/koyeb-import-source

Gilbert09 commented Jul 14, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 14, 2026 •

edited

Loading

🧪 Backend test coverage

Uh oh!

Uh oh!

veria-ai Bot commented Jul 14, 2026 •

edited

Loading

Uh oh!

trunk-io Bot commented Jul 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

danielcarletti left a comment

Uh oh!

Gilbert09 commented Jul 14, 2026

Uh oh!

Uh oh!

deployment-status-posthog Bot commented Jul 15, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Gilbert09 commented Jul 14, 2026

Problem

Changes

How did you test this code?

Automatic notifications

Docs update

🤖 Agent context

Uh oh!

github-actions Bot commented Jul 14, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦔 Hogbox preview · ✅ ready

▶ Open the preview

Uh oh!

github-actions Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 CI report

🧪 Backend test coverage

Uh oh!

Uh oh!

veria-ai Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR overview

Security review

Uh oh!

trunk-io Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

danielcarletti left a comment

Choose a reason for hiding this comment

Uh oh!

Gilbert09 commented Jul 14, 2026

Uh oh!

Uh oh!

deployment-status-posthog Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploy status

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jul 14, 2026 •

edited

Loading

github-actions Bot commented Jul 14, 2026 •

edited

Loading

veria-ai Bot commented Jul 14, 2026 •

edited

Loading

trunk-io Bot commented Jul 14, 2026 •

edited

Loading

deployment-status-posthog Bot commented Jul 15, 2026 •

edited

Loading