Skip to content

feat(data-warehouse): implement deno_deploy import source - #70030

Merged
talyn-app[bot] merged 3 commits into
masterfrom
posthog-code/deno-deploy-warehouse-source
Jul 13, 2026
Merged

feat(data-warehouse): implement deno_deploy import source#70030
talyn-app[bot] merged 3 commits into
masterfrom
posthog-code/deno-deploy-warehouse-source

Conversation

@Gilbert09

Copy link
Copy Markdown
Member

Problem

We had a scaffolded but empty deno_deploy warehouse source (stub source.py, enum, and generated config already merged). This fills it in so teams on Deno Deploy can pull their edge-hosting deploy activity and usage into the PostHog data warehouse and analyze it next to product data.

Deno Deploy is a well-known serverless JS/TS edge platform from the Deno company, and its v2 REST API cleanly exposes the resources warehouse users care about: apps, deployment revisions, custom domains, per-app usage analytics, and runtime logs.

Changes

Implemented the source against the Deno Deploy v2 API only (api.deno.com/v2 — the legacy v1 Subhosting API sunsets 2026-07-20, so nothing targets it). Standard source.py / settings.py / deno_deploy.py split.

Tables:

Table Grain Sync
apps one row per app in the org full refresh
revisions deployment/build history, fanned out per app full refresh
domains custom domains for the org full refresh
analytics per-app usage metrics in 15-minute buckets, fanned out per app incremental (since/until on bucket time)
logs runtime log lines, fanned out per app incremental (start/end on timestamp)

Design notes:

  • ResumableSource with cursor + limit pagination. List endpoints follow the Link: <...>; rel="next" header; the logs endpoint pages on the body next_cursor. Fan-out endpoints checkpoint a stable app-id bookmark so a heartbeat-timeout retry resumes mid-fan-out instead of restarting.
  • Incremental only where the API genuinely filters server-side. Only analytics (since/until) and logs (start/end) expose real time filters, so only they are incremental. end/until is always sent (omitting end on logs switches the endpoint to real-time streaming). The other three are small catalogs with no updated-since filter, so full refresh. Partition keys are all stable created_at-style fields, never updated_at.
  • Columnar analytics are reshaped from the {fields, values} payload into one row per bucket, keyed by field name.
  • Log lines carry no natural id, so the transport synthesizes a stable content-hash id. Merging on it makes the re-pulled boundary window idempotent rather than duplicating rows. Because every run fetches each app up to now, consecutive [start, now] windows overlap and leave no gap even though the fan-out shares one watermark.
  • All outbound HTTP goes through make_tracked_session(); get_non_retryable_errors() covers 401/403; validate_credentials does one cheap probe against /v2/apps.
  • Kept behind unreleasedSource=True with releaseStatus=alpha as requested.

Note

Endpoint paths, pagination, and auth were verified against the live Deno Deploy OpenAPI spec (api.deno.com/v2/openapi.json) and unauthenticated probes. The server-side time-filter behavior of since/until and start/end could not be smoke-tested with a real token (no credentials available), so that assumption, and the assumption that logs return oldest-first within a window, are called out in code comments.

Missing icon: no logo exists at frontend/public/services/deno_deploy.*. Fetching one needs a Logo.dev API key I don't have, so I left iconPath pointing at deno_deploy.png and did not commit a placeholder. The icon needs to be added separately.

Docs: no posthog.com checkout was available here, so audit_source_docs was not run. The user-facing doc (below) still needs to land in the posthog.com repo at contents/docs/cdp/sources/deno-deploy.md (slug matches docsUrl).

Drafted posthog.com doc — contents/docs/cdp/sources/deno-deploy.md
---
title: Linking Deno Deploy as a source
sidebar: Docs
showTitle: true
availability: { free: full, selfServe: full, enterprise: full }
sourceId: DenoDeploy
beta: true
---

import SourceSetupIntro from "../_snippets/source-setup-intro.mdx"
import SyncModes from "../_snippets/sync-modes.mdx"
import TroubleshootingLink from "../_snippets/dw-troubleshooting-link.mdx"
import AlphaRelease from "../_snippets/alpha-release.mdx"

<AlphaRelease />

Sync your Deno Deploy apps, deployment revisions, custom domains, usage analytics, and runtime logs into the PostHog data warehouse to analyze deploy activity and edge usage alongside your product data.

## Prerequisites

You need an organization access token for the Deno Deploy organization you want to sync. The token is scoped to a single organization, so one connection syncs one organization.

## Adding a data source

<SourceSetupIntro />

You need a **Deno Deploy organization access token**. Create one in your [Deno Deploy dashboard](https://app.deno.com) under **Settings > Access Tokens**. The token starts with `ddo_`.

## Sync modes

<SyncModes />

The `apps`, `revisions`, and `domains` tables are small catalogs with no updated-since filter on the API, so they full refresh each sync. The `analytics` and `logs` tables sync incrementally using the API's server-side time windows. Logs are high volume, so that table is deselected by default; enable it only if you want log lines in the warehouse.

## Configuration

<SourceParameters />

## Supported tables

<SourceTables />

## Troubleshooting

If the connection fails with an invalid-token error, create a fresh organization access token in the Deno Deploy dashboard and reconnect. Tokens are organization scoped, so make sure the token belongs to the organization whose data you want to sync.

<TroubleshootingLink />

How did you test this code?

Automated tests I (Claude) ran locally, all green:

  • tests/test_deno_deploy.py (33 tests) — Link-header + body-cursor pagination, fan-out with parent-app injection and mid-fan-out resume, analytics reshaping, synthesized log id stability, incremental time-window math (first-sync lookback, watermark minus lookback, future-clamp), credential status mapping, and SourceResponse shape per endpoint.
  • tests/test_deno_deploy_source.py (16 tests) — source config metadata, per-endpoint schema flags, credential delegation, resumable-manager binding, and source_for_pipeline plumbing.
  • The shared test_source_categories.py and test_source_config_generator.py suites (1521 tests) still pass.

ruff check and ruff format are clean. I could not run a real end-to-end sync (no Deno Deploy credentials in this environment).

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

🤖 Agent context

Autonomy: Fully autonomous

Built with Claude Code. Invoked the /implementing-warehouse-sources and /documenting-warehouse-sources skills and followed the klaviyo/github sources as references.

Decisions worth flagging for review:

  • Endpoint set was chosen from the v2 OpenAPI spec (there are no Airbyte/Fivetran connectors for Deno Deploy to cross-reference). Picked the five that map to durable warehouse tables and skipped write/management endpoints and one-off resources like layers and database instances.
  • Incremental scope was kept deliberately narrow: only analytics and logs have real server-side time filters, so only they are incremental. I resisted marking revisions incremental on a client-side created_at cutoff, since that still pages the whole history each run.
  • Log primary key: Deno runtime logs have no id, which risks duplicate/OOM-prone merges. I synthesized a content-hash id so merge is idempotent across the overlapping incremental boundary, and documented the rare tradeoff (two byte-identical lines collapse).
  • Generated config: re-running the generator surfaced unrelated pre-existing drift (a stale StripeAuthMethodConfig, plus formatter line-wrapping differences), so I applied only the one-line access_token change the generator produces for this source and left the rest untouched to keep the diff minimal.

Agent-authored, so it needs a human review before merge.


Created with PostHog Code

@github-actions

Copy link
Copy Markdown
Contributor

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

  • Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
  • Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

@greptile-apps

greptile-apps Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Reviews (1): Last reviewed commit: "feat(data-warehouse): implement deno_dep..." | Re-trigger Greptile

@veria-ai

veria-ai Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

@trunk-io

trunk-io Bot commented Jul 10, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

@estefaniarabadan estefaniarabadan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isolated new data-warehouse source scaffold — self-contained under its own source directory with only the expected shared-registry additions (SOURCES.md, generated_configs.py, icon), tests included, CI green. Reviewed as part of a batch. LGTM 👍

Fill in the scaffolded Deno Deploy warehouse source end to end: apps, revisions, domains, analytics, and runtime logs, built against the Deno Deploy v2 REST API (api.deno.com/v2).

- ResumableSource with cursor/Link-header pagination and per-app fan-out resume state.
- Full refresh for apps/revisions/domains; incremental for analytics (since/until) and logs (start/end), the only endpoints with genuine server-side time filters.
- Reshapes the columnar analytics payload into rows and synthesizes a stable content-hash id for log lines so merge dedup is idempotent.
- All outbound HTTP goes through make_tracked_session().

Generated-By: PostHog Code
Task-Id: 841f0ce5-8b0c-4374-9ddf-a3e56cd8a493
Address review feedback and CI:

- Redact the bearer token from tracked HTTP telemetry and disable redirects
  by routing every session through a shared _make_session helper.
- Add an SSRF guard so only HTTPS api.deno.com URLs (built, Link-header, logs
  cursor, or persisted resume URLs) can carry the token.
- Fix logs cursor double-encoding by decoding existing query values with
  parse_qsl before re-encoding.
- Fix mypy type errors in the source and its tests.

Generated-By: PostHog Code
Task-Id: 9537d5a3-7c66-41ec-854b-1012eab23f1b
Run the credential-validation probe through the same _require_deno_deploy_url
host allowlist as the sync path, for defense in depth and consistency.

Generated-By: PostHog Code
Task-Id: 9537d5a3-7c66-41ec-854b-1012eab23f1b
@Gilbert09
Gilbert09 force-pushed the posthog-code/deno-deploy-warehouse-source branch from ccc7b12 to 060681a Compare July 13, 2026 18:01
@github-actions

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Backend coverage — 97.0% of changed backend lines covered — 14 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ███████████████████░ 97.0% (510 / 524)

File Patch Uncovered changed lines
products/warehouse_sources/backend/temporal/data_imports/sources/deno_deploy/deno_deploy.py 93.9% 38–39, 87, 131, 134–135, 155–159, 272
products/warehouse_sources/backend/temporal/data_imports/sources/deno_deploy/tests/test_deno_deploy.py 98.9% 50–51

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29272764375 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)
Product Coverage Lines
warehouse_sources ███████████████████░ 96.1% 209,173 / 217,751

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

@talyn-app
talyn-app Bot merged commit 635db37 into master Jul 13, 2026
212 checks passed
@talyn-app
talyn-app Bot deleted the posthog-code/deno-deploy-warehouse-source branch July 13, 2026 18:35
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 13, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-13 19:16 UTC Run
prod-us ✅ Deployed 2026-07-13 19:28 UTC Run
prod-eu ✅ Deployed 2026-07-13 19:31 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants