feat(data-warehouse): implement deno_deploy import source - #70030
Conversation
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
|
Reviews (1): Last reviewed commit: "feat(data-warehouse): implement deno_dep..." | Re-trigger Greptile |
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 1 · PR risk: 0/10 |
estefaniarabadan
left a comment
There was a problem hiding this comment.
Isolated new data-warehouse source scaffold — self-contained under its own source directory with only the expected shared-registry additions (SOURCES.md, generated_configs.py, icon), tests included, CI green. Reviewed as part of a batch. LGTM 👍
Fill in the scaffolded Deno Deploy warehouse source end to end: apps, revisions, domains, analytics, and runtime logs, built against the Deno Deploy v2 REST API (api.deno.com/v2). - ResumableSource with cursor/Link-header pagination and per-app fan-out resume state. - Full refresh for apps/revisions/domains; incremental for analytics (since/until) and logs (start/end), the only endpoints with genuine server-side time filters. - Reshapes the columnar analytics payload into rows and synthesizes a stable content-hash id for log lines so merge dedup is idempotent. - All outbound HTTP goes through make_tracked_session(). Generated-By: PostHog Code Task-Id: 841f0ce5-8b0c-4374-9ddf-a3e56cd8a493
Address review feedback and CI: - Redact the bearer token from tracked HTTP telemetry and disable redirects by routing every session through a shared _make_session helper. - Add an SSRF guard so only HTTPS api.deno.com URLs (built, Link-header, logs cursor, or persisted resume URLs) can carry the token. - Fix logs cursor double-encoding by decoding existing query values with parse_qsl before re-encoding. - Fix mypy type errors in the source and its tests. Generated-By: PostHog Code Task-Id: 9537d5a3-7c66-41ec-854b-1012eab23f1b
Run the credential-validation probe through the same _require_deno_deploy_url host allowlist as the sync path, for defense in depth and consistency. Generated-By: PostHog Code Task-Id: 9537d5a3-7c66-41ec-854b-1012eab23f1b
ccc7b12 to
060681a
Compare
🤖 CI report
|
| File | Patch | Uncovered changed lines |
|---|---|---|
products/warehouse_sources/backend/temporal/data_imports/sources/deno_deploy/deno_deploy.py |
93.9% | 38–39, 87, 131, 134–135, 155–159, 272 |
products/warehouse_sources/backend/temporal/data_imports/sources/deno_deploy/tests/test_deno_deploy.py |
98.9% | 50–51 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29272764375 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
warehouse_sources |
███████████████████░ 96.1% |
209,173 / 217,751 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
Problem
We had a scaffolded but empty
deno_deploywarehouse source (stubsource.py, enum, and generated config already merged). This fills it in so teams on Deno Deploy can pull their edge-hosting deploy activity and usage into the PostHog data warehouse and analyze it next to product data.Deno Deploy is a well-known serverless JS/TS edge platform from the Deno company, and its v2 REST API cleanly exposes the resources warehouse users care about: apps, deployment revisions, custom domains, per-app usage analytics, and runtime logs.
Changes
Implemented the source against the Deno Deploy v2 API only (
api.deno.com/v2— the legacy v1 Subhosting API sunsets 2026-07-20, so nothing targets it). Standardsource.py/settings.py/deno_deploy.pysplit.Tables:
appsrevisionsdomainsanalyticssince/untilon buckettime)logsstart/endontimestamp)Design notes:
ResumableSourcewith cursor +limitpagination. List endpoints follow theLink: <...>; rel="next"header; the logs endpoint pages on the bodynext_cursor. Fan-out endpoints checkpoint a stable app-id bookmark so a heartbeat-timeout retry resumes mid-fan-out instead of restarting.analytics(since/until) andlogs(start/end) expose real time filters, so only they are incremental.end/untilis always sent (omittingendon logs switches the endpoint to real-time streaming). The other three are small catalogs with no updated-since filter, so full refresh. Partition keys are all stablecreated_at-style fields, neverupdated_at.{fields, values}payload into one row per bucket, keyed by field name.id. Merging on it makes the re-pulled boundary window idempotent rather than duplicating rows. Because every run fetches each app up tonow, consecutive[start, now]windows overlap and leave no gap even though the fan-out shares one watermark.make_tracked_session();get_non_retryable_errors()covers 401/403;validate_credentialsdoes one cheap probe against/v2/apps.unreleasedSource=TruewithreleaseStatus=alphaas requested.Note
Endpoint paths, pagination, and auth were verified against the live Deno Deploy OpenAPI spec (
api.deno.com/v2/openapi.json) and unauthenticated probes. The server-side time-filter behavior ofsince/untilandstart/endcould not be smoke-tested with a real token (no credentials available), so that assumption, and the assumption that logs return oldest-first within a window, are called out in code comments.Missing icon: no logo exists at
frontend/public/services/deno_deploy.*. Fetching one needs a Logo.dev API key I don't have, so I lefticonPathpointing atdeno_deploy.pngand did not commit a placeholder. The icon needs to be added separately.Docs: no
posthog.comcheckout was available here, soaudit_source_docswas not run. The user-facing doc (below) still needs to land in the posthog.com repo atcontents/docs/cdp/sources/deno-deploy.md(slug matchesdocsUrl).Drafted posthog.com doc —
contents/docs/cdp/sources/deno-deploy.mdHow did you test this code?
Automated tests I (Claude) ran locally, all green:
tests/test_deno_deploy.py(33 tests) — Link-header + body-cursor pagination, fan-out with parent-app injection and mid-fan-out resume, analytics reshaping, synthesized log id stability, incremental time-window math (first-sync lookback, watermark minus lookback, future-clamp), credential status mapping, andSourceResponseshape per endpoint.tests/test_deno_deploy_source.py(16 tests) — source config metadata, per-endpoint schema flags, credential delegation, resumable-manager binding, andsource_for_pipelineplumbing.test_source_categories.pyandtest_source_config_generator.pysuites (1521 tests) still pass.ruff checkandruff formatare clean. I could not run a real end-to-end sync (no Deno Deploy credentials in this environment).Automatic notifications
🤖 Agent context
Autonomy: Fully autonomous
Built with Claude Code. Invoked the
/implementing-warehouse-sourcesand/documenting-warehouse-sourcesskills and followed the klaviyo/github sources as references.Decisions worth flagging for review:
analyticsandlogshave real server-side time filters, so only they are incremental. I resisted markingrevisionsincremental on a client-sidecreated_atcutoff, since that still pages the whole history each run.idso merge is idempotent across the overlapping incremental boundary, and documented the rare tradeoff (two byte-identical lines collapse).StripeAuthMethodConfig, plus formatter line-wrapping differences), so I applied only the one-lineaccess_tokenchange the generator produces for this source and left the rest untouched to keep the diff minimal.Agent-authored, so it needs a human review before merge.
Created with PostHog Code