feat(data-warehouse): implement clickhouse_cloud import source

[Gilbert09]

Problem

The clickhouse_cloud warehouse source was scaffolded in #71137 with unreleasedSource=True and no sync logic. FinOps and platform teams want ClickHouse Cloud cost and service data in the warehouse for chargeback, cost-trend, and capacity dashboards.

Changes

Implements the source end-to-end and ships it visible with releaseStatus=ALPHA (the unreleasedSource flag is removed).

Architecture (per the source.py / settings.py / transport split):

• settings.py – declarative endpoint catalog: organizations, services, usage_cost, api_keys, members, activities, backups, with per-endpoint primary keys, stable partition keys (date, createdAt, startedAt), and incremental fields.
• clickhouse_cloud.py – transport over make_tracked_session() with HTTP Basic auth (API key ID/secret). The API returns complete unpaginated result arrays, and scopes each key to exactly one organization.
• usage_cost is the flagship stream: the API caps from_date/to_date at 31 days per request, so the source walks contiguous non-overlapping windows forward from the org's createdAt (full refresh) or the watermark (incremental), yielding rows sorted ascending so per-batch watermark checkpointing stays correct. Nested metrics (storageCHC, computeCHC, ...) are flattened into columns. Recent records are restated until ClickHouse locks them, so the schema declares a 31-day incremental lookback and is merge-only.
• ResumableSource: the usage cost walk bookmarks the next window start in Redis after each yielded window, so Temporal retries resume mid-backfill instead of restarting.
• 429s honor Retry-After (the API allows 10 requests per 10s); 401/403 are registered as non-retryable with actionable user messages.
• canonical_descriptions.py documents every table/column from the official OpenAPI spec, and lists_tables_without_credentials=True publishes the catalog to the public docs.

Note

Endpoint shapes were verified against the live public OpenAPI spec at api.clickhouse.cloud (fetched during implementation) plus unauthenticated probes for error shapes. I couldn't run authenticated calls (no ClickHouse Cloud credentials), so two behaviors are spec-verified but not live-verified: the activities from_date filter (kept merge-only so a silently-ignored filter can't duplicate rows) and the exact lock cadence of usage cost records (covered by the 31-day re-pull window).

Docs: PostHog/posthog.com#18554 adds contents/docs/cdp/sources/clickhouse-cloud.md; docsUrl matches the slug and audit_source_docs reports no issues for this source.

How did you test this code?

• pytest products/warehouse_sources/backend/temporal/data_imports/sources/clickhouse_cloud/tests/ – 52 tests pass.
  • Transport tests catch: >31-day or overlapping usage cost windows (API 400s / duplicate rows within a sync), a future-dated watermark producing from_date > to_date, resume state saved before yield (data loss on crash), missing organizationId/serviceId stamping (breaks composite primary keys), activities arriving unsorted (corrupts the ascending watermark), credential-status mapping, and retry/non-retryable error classification.
  • Source tests catch: incremental advertised on endpoints without a server-side filter, the restatement lookback being dropped, wrong primary keys/partitioning per endpoint, and the source regressing to hidden (unreleasedSource).
• Registry-wide suites (sources/tests/, common/test/) pass except pre-existing MinIO-dependent failures that also fail on the base branch.
• ruff check / ruff format clean. generate:source-configs regenerated ClickhouseCloudSourceConfig.

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

PostHog/posthog.com#18554 adds the source doc.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

• Authored with Claude Code following the /implementing-warehouse-sources, /documenting-warehouse-sources, and /writing-tests skills.
• Modeled on the Anthropic source (closest shape: org-level cost reporting with date windows), but yields plain list[dict] batches instead of a source-level Batcher per current skill guidance.
• Chose a bespoke transport over rest_source.RESTClient because the API has no pagination and needs date-window chunking; skipped the Prometheus metrics endpoint (point-in-time text-format scrapes don't fit warehouse tables well) – can be added later if requested.

---

Created with PostHog Code

[github-actions]

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

• Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
• Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

[github-actions]

🤖 CI report

⚠️ Backend coverage — 99.0% of changed backend lines covered — 10 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ████████████████████ 99.0% (2,232 / 2,242)

File	Patch	Uncovered changed lines
products/warehouse_sources/backend/temporal/data_imports/sources/clickhouse_cloud/clickhouse_cloud.py	96.4%	46–47, 108, 140–141, 143
products/warehouse_sources/backend/temporal/data_imports/sources/clickhouse_cloud/tests/test_clickhouse_cloud.py	97.9%	140–142, 238

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29435165022 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
demo	███████████░░░░░░░░░ 56.2%	1,497 / 2,663
tasks	█████████████░░░░░░░ 67.4%	25,473 / 37,804
signals	████████████████░░░░ 79.1%	19,072 / 24,111
data_modeling	████████████████░░░░ 80.0%	4,834 / 6,045
cdp	████████████████░░░░ 80.7%	3,118 / 3,864
agent_platform	█████████████████░░░ 84.2%	3,112 / 3,695
notebooks	█████████████████░░░ 84.3%	6,343 / 7,520
cohorts	█████████████████░░░ 86.1%	4,022 / 4,671
actions	█████████████████░░░ 86.6%	717 / 828
product_tours	█████████████████░░░ 87.5%	1,266 / 1,447
exports	██████████████████░░ 88.3%	6,891 / 7,800
conversations	██████████████████░░ 88.9%	16,129 / 18,133
dashboards	██████████████████░░ 89.0%	5,647 / 6,345
mcp_analytics	██████████████████░░ 89.1%	2,502 / 2,807
error_tracking	██████████████████░░ 89.6%	9,724 / 10,852
alerts	██████████████████░░ 89.9%	3,638 / 4,046
engineering_analytics	██████████████████░░ 90.1%	5,105 / 5,665
streamlit_apps	██████████████████░░ 90.5%	2,502 / 2,764
slack_app	██████████████████░░ 90.6%	9,511 / 10,503
marketing_analytics	██████████████████░░ 90.8%	11,514 / 12,684
product_analytics	██████████████████░░ 91.1%	5,527 / 6,068
data_warehouse	██████████████████░░ 92.4%	18,005 / 19,489
workflows	██████████████████░░ 92.4%	5,150 / 5,574
web_analytics	███████████████████░ 92.7%	13,624 / 14,691
ai_observability	███████████████████░ 92.8%	14,863 / 16,014
surveys	███████████████████░ 92.9%	5,660 / 6,094
posthog_ai	███████████████████░ 93.2%	1,322 / 1,418
approvals	███████████████████░ 93.3%	3,395 / 3,640
reminders	███████████████████░ 93.4%	468 / 501
early_access_features	███████████████████░ 93.8%	848 / 904
skills	███████████████████░ 94.4%	2,827 / 2,995
revenue_analytics	███████████████████░ 94.5%	3,598 / 3,809
review_hog	███████████████████░ 94.6%	6,500 / 6,873
endpoints	███████████████████░ 94.6%	8,647 / 9,143
logs	███████████████████░ 95.3%	9,528 / 9,994
replay_vision	███████████████████░ 95.7%	13,300 / 13,896
experiments	███████████████████░ 95.7%	24,171 / 25,248
feature_flags	███████████████████░ 96.1%	14,757 / 15,354
annotations	███████████████████░ 96.2%	732 / 761
warehouse_sources	███████████████████░ 96.2%	224,038 / 232,910
user_interviews	███████████████████░ 96.4%	2,242 / 2,325
data_catalog	███████████████████░ 97.1%	2,035 / 2,095
customer_analytics	███████████████████░ 97.2%	7,480 / 7,698

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

✅ Django migration risk — no migrations to analyze

No Django migrations need risk analysis.

[Gilbert09]

CI triage: the failing Python code quality (mypy) and Product tests (warehouse-sources) checks all traced to one issue inherited from the base branch — the scaffold branch's regeneration of generated_configs.py turned StripeAuthMethodConfig's option sub-fields required (stripe_secret_key: str, stripe_integration_id: int), which breaks Stripe test collection and mypy. Only the selected auth option's fields exist in a config dict, so they must stay optional.

Pushed 0176148 restoring the exact class shape master has. Note for the base branch (#71137): anyone re-running generate:source-configs will reintroduce this, since Stripe's select-option sub-fields are marked required=True in stripe/source.py and the generator honors that for the dataclass — the generator likely needs to force select-option children optional.

[trunk-io]

     

View Full Report ↗︎ ⋅ Docs

[danielcarletti]

LGTM