feat(data-warehouse): add resend source#55935
Merged
Merged
Conversation
Adds Resend (https://resend.com) as a beta Data warehouse source covering audiences, broadcasts, domains, emails (cursor-paginated), and contacts (fan-out from audiences). Built on ResumableSource so /emails backfills can resume mid-sync. Generated-By: PostHog Code Task-Id: 9c3a0cc6-cc75-4def-9a85-fe8da4927b4f
Contributor
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
|
Size Change: 0 B Total Size: 132 MB ℹ️ View Unchanged
|
MarconLP
approved these changes
Apr 23, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new Resend (transactional email platform) Data Warehouse source, including backend source implementation + resumable sync logic, schema/enum registration across backend/shared schema/frontend, and an icon for the UI.
Changes:
- Introduces
ResendSource(resumable) with 5 endpoints (audiences,broadcasts,domains,emails,contacts) and associated settings + transport logic. - Registers
Resendin enums/choices/schema lists and wires the source into the import sources registry. - Adds unit tests covering credential validation, pagination/resume, fan-out behavior, and retry behavior.
Reviewed changes
Copilot reviewed 12 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| products/data_warehouse/backend/types.py | Adds RESEND to backend ExternalDataSourceType choices. |
| products/data_warehouse/backend/migrations/max_migration.txt | Bumps recorded latest migration to 0048. |
| products/data_warehouse/backend/migrations/0048_alter_externaldatasource_source_type.py | Adds "Resend" to ExternalDataSource.source_type Django choices. |
| posthog/temporal/data_imports/sources/resend/source.py | Implements ResendSource registry entry, UI config, schemas, and pipeline wiring. |
| posthog/temporal/data_imports/sources/resend/settings.py | Defines endpoint metadata and incremental field definitions. |
| posthog/temporal/data_imports/sources/resend/resend.py | Implements HTTP fetching, retry policy, pagination, resumable state, and fan-out contacts behavior. |
| posthog/temporal/data_imports/sources/resend/tests/test_resend.py | Transport-level tests for pagination/resume/fan-out and retry behavior. |
| posthog/temporal/data_imports/sources/resend/tests/test_resend_source.py | Source-class tests for config, schemas, validation, and pipeline wiring. |
| posthog/temporal/data_imports/sources/generated_configs.py | Adds ResendSourceConfig type used for parsing job inputs. |
| posthog/temporal/data_imports/sources/init.py | Exports/imports ResendSource so it is registered with the sources package. |
| posthog/schema.py | Adds RESEND to shared schema enum. |
| frontend/src/queries/schema/schema-general.ts | Adds Resend to the frontend external data source type list. |
| frontend/public/services/resend.png | Adds Resend service icon for the UI. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
Migration SQL ChangesHey 👋, we've detected some migrations on this PR. Here's the SQL output for each migration, make sure they make sense:
|
Contributor
🔍 Migration Risk AnalysisWe've analyzed your migrations for potential risks. Summary: 0 Safe | 1 Needs Review | 0 Blocked
|
Contributor
|
⏭️ Skipped snapshot commit because branch advanced to The new commit will trigger its own snapshot update workflow. If you expected this workflow to succeed: This can happen due to concurrent commits. To get a fresh workflow run, either:
|
- Drop the `requests.HTTPError(...)` construction in a test mock (mypy required a `response` kwarg) — the transport code only needs any exception to propagate, so use a plain `Exception`. - Use direct key access on `mock.call_args_list[i].kwargs["params"]` so mypy doesn't treat the indexed value as `Any | None`. - Simplify `get_schemas` to compute `has_incremental` once per endpoint instead of double-looking up `INCREMENTAL_FIELDS`, and drop the unnecessary `list(ENDPOINTS)` (per greptile review). Generated-By: PostHog Code Task-Id: 9c3a0cc6-cc75-4def-9a85-fe8da4927b4f
CI was failing the `schema.json and validators.js up to date` and `schema.py up to date` gates because `schema.json` wasn't regenerated when `Resend` was added to `externalDataSources` in `schema-general.ts`. Regenerated via `pnpm run schema:build:json`; `schema.py` was already in sync with the expected output once `schema.json` caught up. Generated-By: PostHog Code Task-Id: 9c3a0cc6-cc75-4def-9a85-fe8da4927b4f
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
Contributor
Query snapshots: Backend query snapshots updatedChanges: 2 snapshots (2 modified, 0 added, 0 deleted) What this means:
Next steps:
|
Replaces the deprecated `betaSource=True` flag with `releaseStatus="beta"` to match the new release-status field added in #55934 (alpha/beta/ga). Generated-By: PostHog Code Task-Id: 9c3a0cc6-cc75-4def-9a85-fe8da4927b4f
Contributor
|
⏭️ Skipped snapshot commit because branch advanced to The new commit will trigger its own snapshot update workflow. If you expected this workflow to succeed: This can happen due to concurrent commits. To get a fresh workflow run, either:
|
- Register ResendSourceConfig in generated_configs.get_config_for_source mapping. Without this entry ResendSource.parse_config() would raise KeyError at runtime when the pipeline tries to resolve the config class for an ExternalDataSource row of type Resend. - Drop advertised incremental / append support. Resend's list endpoints don't expose server-side filtering on created_at, and source_for_pipeline didn't forward `should_use_incremental_field` / `db_incremental_field_last_value` / `incremental_field` to the source iterator — so every schema was in practice syncing as full_refresh. Aligning the advertisement to match reality; incremental can be added in a follow-up if/when the API supports it. - Fix contacts-fan-out resume logic: if the previously-completed audience has been deleted between syncs the old code kept `skipping=True` forever, silently dropping every remaining audience on every subsequent sync. Now we locate the resume index up front and fall back to a fresh resync with a warning if the audience is gone. Use direct key access for the audience primary key so missing ids fail loudly instead of silently. - Patch `tenacity.nap.time.sleep` in the 429 retry test to avoid real exponential-backoff sleeps during the test run. Generated-By: PostHog Code Task-Id: 9c3a0cc6-cc75-4def-9a85-fe8da4927b4f
Contributor
Query snapshots: Backend query snapshots updatedChanges: 1 snapshots (1 modified, 0 added, 0 deleted) What this means:
Next steps:
|
Per codebase convention, primary-key-like fields should raise loudly when
missing rather than silently skipping. Drop the `.get("id")` + warning
fallback in the emails pagination so an item missing an `id` surfaces as
a KeyError immediately instead of producing a silent data gap.
Generated-By: PostHog Code
Task-Id: 4b2e65f0-fa0e-4492-915d-b94ef6763ce2
…stomer_io secret - Resend pagination now surfaces an empty-page-with-has-more=True response as a hard ValueError instead of silently terminating, so an API contract violation produces a data gap that's actually visible. - Customer.io webhook signing key field was missing secret=True; without it Pydantic validation fails for SourceFieldInputConfig and breaks every source-registry-iterating test (and the new-source page). Generated-By: PostHog Code Task-Id: 8a086398-abdf-4c00-ace1-a257789c9fa4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Resend (https://resend.com) is a transactional email platform. PostHog customers that send email through Resend have no first-class way to pull audiences, broadcasts, domains, emails, or contacts into the Data warehouse for analytics alongside their product data.
Changes
Adds Resend as a new Data warehouse source, shipping as beta:
ResendSourceatposthog/temporal/data_imports/sources/resend/(source.py/settings.py/resend.pysplit)ResumableSource—/emailsis cursor-paginated (limit+after,has_more) and persists its cursor after each page;/contactsfan-out persists the last completed parent audience id so interrupted backfills don't restart from the topaudiences,broadcasts,domains,emails,contacts(fan-out from audiences with_audience_idinjected onto each row). All partitioned bycreated_atwith monthly partitioningapi_key(password) field; validation probesGET /domainsget_non_retryable_errorsmaps 401/403 to actionable messages so bad keys fail fast instead of retry-stormingproducts/data_warehouse/backend/types.py,posthog/schema.py,frontend/src/queries/schema/schema-general.ts,sources/__init__.py, andgenerated_configs.py(the latter two are normally regenerated bypnpm run generate:source-configs/pnpm run schema:build)0048_alter_externaldatasource_source_type.pyadds"Resend"toExternalDataSource.source_typechoicesfrontend/public/services/resend.png(sourced via Logo.dev)How did you test this code?
I am an LLM agent; I verified via unit tests only and did not test the end-to-end sync in the UI or against a real Resend account.
test_resend.py, 21 cases):validate_credentialsstatus-code matrix, flat-endpoint single-batch behavior,/emailscursor pagination + state save, resume-from-saved-cursor,/contactsfan-out row enrichment and parent-id resumption, per-endpointSourceResponseshape, 429 retry, 401 non-retry.test_resend_source.py, 10 cases):source_type, config shape and fields,get_schemas(all + filtered by names), credential validation success/failure, resumable manager binding,source_for_pipelineplumbing.31 passed in 2.91s.ruff check/ruff formatclean across all modified files.ResendSource.get_source_configresolves, reportsLabel: Resend, Beta: True, andget_schemasreturns the expected five endpoint names.Reviewers should:
pnpm run generate:source-configsandpnpm run schema:buildlocally and confirm the generated-file diffs are empty (i.e. what I wrote by hand matches the generator output).audiencesordomainsfor speed,emailsfor pagination coverage).Publish to changelog?
yes — new Data warehouse source
Docs update
Need docs page at
posthog.com/docs/cdp/sources/resend(referenced bydocsUrl). Flagged separately.🤖 LLM context
Authored by PostHog Code (task
9c3a0cc6-cc75-4def-9a85-fe8da4927b4f) using theimplementing-warehouse-sourcesskill. Modeled afterposthog/temporal/data_imports/sources/klaviyo/as the reference ResumableSource. Plan was reviewed and approved by the human before implementation.