feat(data-warehouse): implement dbt import source

[Gilbert09]

Problem

The dbt source existed only as a scaffolded stub (unreleasedSource=True, no fields, no sync logic), so users couldn't pull their dbt platform (dbt Cloud) metadata into the Data warehouse.

Why: teams orchestrating transformations on the dbt platform want run history, job definitions, and project/environment metadata next to their product data to analyze pipeline reliability, run durations, and failure rates.

Changes

Implements the source end-to-end following the implementing-warehouse-sources architecture contract:

• settings.py – declarative endpoint catalog: accounts, projects, environments, jobs, users, runs (v3 for account-scoped metadata, v2 for jobs/runs, matching the API's current coverage). All primary keys are id; runs partitions on the stable created_at.
• dbt.py – transport: tracked session, Token bearer auth, limit/offset pagination (100-row API cap) using the response envelope's extra.pagination, tenacity retries on 429/5xx, and credential validation against the account endpoint.
• source.py – ResumableSource registration: region select (US / EMEA / APAC hosts) plus an optional custom base URL for cell-based and single-tenant deployments, per-endpoint permission probes for the schema picker, non-retryable 401/403 mapping, and canonical table/column descriptions from the official API docs.

Design notes:

• Incremental runs sync without a server-side time filter. The runs list supports order_by=-created_at but no created_after param, so incremental syncs walk newest-first and stop once a page dips below the watermark – only new rows are fetched (sort_mode="desc", same pattern as the GitHub source). A 24h lookback re-pulls recently created runs so late status changes land; merge dedupes on id. All other endpoints are small and full refresh.
• Resumability saves the page offset after each yielded batch. Offset resume is safe against inserts mid-run because the walk is newest-first (rows shift deeper, so a resume re-reads rather than skips).
• Credential retargeting: region and custom_base_url are declared as connection_host_fields, and custom URLs go through the _is_host_safe SSRF check at create and sync time, with redirects blocked.
• Ships visible with releaseStatus=ReleaseStatus.ALPHA (the unreleasedSource=True stub flag is removed), and lists_tables_without_credentials=True so the public docs render the table catalog.
• SOURCES.md updated (scaffolded → implemented).

The user-facing doc is in PostHog/posthog.com#18548; docsUrl points at /docs/cdp/sources/dbt and manage.py audit_source_docs reports no issues for this source against that checkout.

Note

Endpoint behavior was verified against the live API where possible without credentials: path validity (valid paths return the 403 auth JSON, /api/v3/.../jobs/ 404s – hence jobs stays on v2), auth error shape (bogus token → 401 with the documented envelope), and regional hosts. Pagination envelope handling and order_by semantics follow the current API reference and could not be exercised against a live account; the ALPHA label reflects that.

How did you test this code?

Automated tests only (no live dbt account available):

• tests/test_dbt_source.py – guards the source contract: the source can't be re-hidden (unreleasedSource is None), the token field stays secret, runs is the only incremental schema and everything merges (append would duplicate lookback re-pulls), users stays deselected by default (permission-gated), and credential/pipeline plumbing passes the config through intact.
• tests/test_dbt.py – guards the transport: pagination terminates on both short-page and total_count signals, incremental walks stop requesting once a page predates the watermark (the API-cost property that makes this incremental), lookback-window and missing-cursor rows are kept, resume starts from the saved offset and state is saved only after a yield, region/custom-URL resolution rejects non-https hosts, and validate maps 200/401/403/404 and redirects correctly.

Ran the full sources/common + test_source_categories suites (1899 passed; 10 pre-existing errors in test_webhook_s3.py from MinIO being unreachable in the sandbox, unrelated). ruff check/format and hogli ci:preflight --fix pass with no failures.

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

Doc added in PostHog/posthog.com#18548.

🤖 Agent context

Autonomy: Fully autonomous

Authored with Claude Code (PostHog Code cloud task). Skills invoked: implementing-warehouse-sources, documenting-warehouse-sources, writing-tests. Decisions: chose ResumableSource over SimpleSource because offset pagination gives a deterministic resume point; kept runs on a desc walk with client-side watermark stop instead of claiming server-side incremental (the API has no time filter); regenerated generated_configs.py but kept only the dbt hunk because a full regeneration in this environment flipped unrelated Stripe fields (their optionality depends on OAuth env vars not set here).

---

Created with PostHog Code

[github-actions]

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

• Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
• Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

[trunk-io]

     

View Full Report ↗︎ ⋅ Docs

[danielcarletti]

LGTM

[github-actions]

🤖 CI report

⚠️ Backend coverage — 95.0% of changed backend lines covered — 21 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ███████████████████░ 95.0% (487 / 508)

File	Patch	Uncovered changed lines
products/warehouse_sources/backend/temporal/data_imports/sources/dbt/dbt.py	89.3%	61, 78–79, 101–103, 110–111, 131, 149–150, 164–165, 215, 218, 221–222, 256
products/warehouse_sources/backend/temporal/data_imports/sources/dbt/source.py	91.7%	113, 117, 170

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29456123751 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
demo	███████████░░░░░░░░░ 56.2%	1,497 / 2,663
tasks	█████████████░░░░░░░ 67.4%	25,560 / 37,895
signals	████████████████░░░░ 79.1%	19,035 / 24,073
data_modeling	████████████████░░░░ 80.0%	4,834 / 6,045
cdp	████████████████░░░░ 80.7%	3,118 / 3,864
notebooks	█████████████████░░░ 84.3%	6,343 / 7,520
agent_platform	█████████████████░░░ 84.7%	3,273 / 3,862
cohorts	█████████████████░░░ 86.1%	4,022 / 4,671
actions	█████████████████░░░ 86.6%	717 / 828
product_tours	█████████████████░░░ 87.5%	1,266 / 1,447
exports	██████████████████░░ 88.3%	6,891 / 7,800
conversations	██████████████████░░ 88.9%	16,129 / 18,133
dashboards	██████████████████░░ 89.1%	5,719 / 6,418
mcp_analytics	██████████████████░░ 89.1%	2,502 / 2,807
error_tracking	██████████████████░░ 89.6%	9,718 / 10,852
alerts	██████████████████░░ 89.9%	3,638 / 4,046
engineering_analytics	██████████████████░░ 90.1%	5,105 / 5,665
streamlit_apps	██████████████████░░ 90.4%	2,499 / 2,764
slack_app	██████████████████░░ 90.6%	9,511 / 10,503
marketing_analytics	██████████████████░░ 90.8%	11,514 / 12,684
product_analytics	██████████████████░░ 91.1%	5,599 / 6,143
data_warehouse	██████████████████░░ 92.1%	18,133 / 19,683
workflows	██████████████████░░ 92.4%	5,148 / 5,574
web_analytics	███████████████████░ 92.7%	13,624 / 14,691
ai_observability	███████████████████░ 92.8%	14,868 / 16,019
surveys	███████████████████░ 92.9%	5,660 / 6,094
posthog_ai	███████████████████░ 93.2%	1,322 / 1,418
approvals	███████████████████░ 93.3%	3,395 / 3,640
reminders	███████████████████░ 93.4%	468 / 501
early_access_features	███████████████████░ 93.8%	848 / 904
endpoints	███████████████████░ 94.1%	8,606 / 9,143
skills	███████████████████░ 94.4%	2,827 / 2,995
revenue_analytics	███████████████████░ 94.5%	3,598 / 3,809
review_hog	███████████████████░ 94.6%	6,529 / 6,902
logs	███████████████████░ 95.3%	9,528 / 9,994
experiments	███████████████████░ 95.6%	24,140 / 25,257
replay_vision	███████████████████░ 95.7%	13,354 / 13,952
annotations	███████████████████░ 96.2%	732 / 761
warehouse_sources	███████████████████░ 96.2%	229,975 / 239,011
feature_flags	███████████████████░ 96.3%	16,002 / 16,625
user_interviews	███████████████████░ 96.4%	2,242 / 2,325
data_catalog	███████████████████░ 97.1%	2,034 / 2,095
customer_analytics	███████████████████░ 97.2%	7,480 / 7,698

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-16 02:51 UTC	Run
prod-us	✅ Deployed	2026-07-16 03:02 UTC	Run
prod-eu	✅ Deployed	2026-07-16 03:06 UTC	Run