Skip to content

feat(data-warehouse): implement retently import source - #69393

Merged
Gilbert09 merged 3 commits into
masterfrom
tom/dwh-retently
Jul 9, 2026
Merged

feat(data-warehouse): implement retently import source#69393
Gilbert09 merged 3 commits into
masterfrom
tom/dwh-retently

Conversation

@Gilbert09

@Gilbert09 Gilbert09 commented Jul 8, 2026

Copy link
Copy Markdown
Member

Problem

Retently (NPS/CSAT/CES surveys) was scaffolded as a data warehouse source but had no sync logic, so users can't pull their customer feedback data into PostHog.

Changes

Implements the Retently source end to end against the v2 REST API (https://app.retently.com/api/v2, X-Api-Key auth):

  • Tables: customers, companies, feedback, outbox, campaigns, templates, reports – matching the stream list of community connectors for Retently.
  • Architecture: ResumableSource with the standard source.py / settings.py / retently.py split. Page-number pagination checkpoints the next page to Redis after each yielded batch, and all HTTP goes through make_tracked_session() with the key redacted. 429/5xx retry with bounded exponential backoff (Retently rate-limits at ~150 req/min).
  • Incremental sync is enabled only on feedback, where the documented startDate query param filters server-side by response creation date. Customers also documents startDate/endDate but doesn't say which field it filters and customer records are mutable, so it stays full refresh (as do companies, which have no date filter). Outbox rows carry no consistently documented unique id, so it ships full refresh with no merge key.
  • Envelope handling covers the three documented response shapes (records under data.<key>, a bare list under data for reports, top-level arrays for campaigns/templates), and pagination follows the pages metadata whether it appears inside data or at the top level – the docs are inconsistent between endpoints.
  • canonical_descriptions.py documents every table/column from the official API docs, and lists_tables_without_credentials = True lets the public docs render the table catalog.
  • Kept behind unreleasedSource=True with releaseStatus="alpha".
  • SOURCES.md moved from Scaffolded to Implemented; generated_configs.py regenerated (Retently class only).

Note

I couldn't curl-verify success-path behavior against a live account (no Retently credentials available), so a few choices are deliberately conservative and commented in code: the incremental endpoint requests sort=createdDate ascending but declares sort_mode="desc" so the watermark only persists after a fully successful sync, and unverifiable incremental candidates ship as full refresh. Verifying against a live account is the remaining step before releasing the source.

The user-facing doc (contents/docs/cdp/sources/retently.md) was written per the docs template and lands via PostHog/posthog.com#18294; audit_source_docs passes for Retently against that doc.

How did you test this code?

  • pytest products/warehouse_sources/backend/temporal/data_imports/sources/retently/tests/ (57 tests pass): envelope extraction per documented response shape, pagination via pages metadata in both documented locations (catches a silent short-page termination if the API caps limit), incremental startDate plumbing and omission, resume-from-saved-page and save-after-yield semantics, retry on 429/5xx without real sleeps, credential validation status mapping, and per-endpoint SourceResponse shape (primary keys, partitioning, sort mode).
  • pytest products/warehouse_sources/backend/temporal/data_imports/sources/tests/ (registry-wide checks incl. source categories, 1353 pass).
  • ruff check / ruff format and hogli ci:preflight --fix pass. No manual testing against a live Retently account was possible.

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Docs update

Doc written for posthog.com at contents/docs/cdp/sources/retently.md (PostHog/posthog.com#18294).

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

  • Authored with Claude Code following the /implementing-warehouse-sources, /writing-tests, and /documenting-warehouse-sources skills.
  • API behavior was verified from the official Retently API docs (including the embedded response examples) plus unauthenticated probes of the live API for error envelopes; secoda/ruddr/huntr/aircall/beamer were used as reference implementations.
  • Considered incremental sync on outbox (documented startDate filter) but rejected it: example rows lack a consistent unique id, so there's no safe merge key.

Created with PostHog Code

Implements the Retently source: customers, companies, feedback, outbox,
campaigns, templates, and reports endpoints via the v2 REST API with
X-Api-Key auth, page-number pagination, resumable syncs, and incremental
sync on feedback via the server-side startDate filter.

Generated-By: PostHog Code
Task-Id: 023c3eab-e55c-429f-b546-084ca0b13ced
@Gilbert09 Gilbert09 self-assigned this Jul 8, 2026
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

  • Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
  • Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

Gilbert09 added a commit to PostHog/posthog.com that referenced this pull request Jul 8, 2026
Companion to PostHog/posthog#69393, which implements the Retently import source.

Generated-By: PostHog Code
Task-Id: 023c3eab-e55c-429f-b546-084ca0b13ced
@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team July 8, 2026 16:24
@greptile-apps

greptile-apps Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Comments Outside Diff (1)

  1. products/warehouse_sources/backend/temporal/data_imports/sources/retently/tests/test_retently.py, line 843-855 (link)

    P2 Loop-based multi-case test should use @parameterized.expand

    test_unpaginated_endpoints_make_one_request_without_page_params iterates over three endpoint cases (campaigns, templates, reports) with a for loop. Each case exercises a meaningfully different _extract_items code path (top-level array vs. bare list under data), so this is a good candidate for parameterization per the repo convention. A failure in one case silently prevents the remaining cases from running, and the test output doesn't identify which endpoint failed without inspecting the assertion message.

    test_every_declared_endpoint_builds_a_response (line 1011) has the same pattern — it loops over all declared endpoints without parameterization.

    Rule Used: Use @parameterized.expand from the `parameterize... (source)

    Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Reviews (1): Last reviewed commit: "feat(data-warehouse): implement retently..." | Re-trigger Greptile

Moves response extraction inside the retried fetch scope so a transiently
malformed payload retries the single request instead of failing the sync,
and parameterizes the loop-based test cases.

Generated-By: PostHog Code
Task-Id: 023c3eab-e55c-429f-b546-084ca0b13ced
@trunk-io

trunk-io Bot commented Jul 8, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

Generated-By: PostHog Code
Task-Id: 81b3dd34-5f9c-4013-a52c-ed16fb84ca7d
@Gilbert09
Gilbert09 merged commit 362ede9 into master Jul 9, 2026
222 checks passed
@Gilbert09
Gilbert09 deleted the tom/dwh-retently branch July 9, 2026 10:51
Gilbert09 added a commit to PostHog/posthog.com that referenced this pull request Jul 9, 2026
Companion to PostHog/posthog#69393, which implements the Retently import source.

Generated-By: PostHog Code
Task-Id: 023c3eab-e55c-429f-b546-084ca0b13ced
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 9, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-09 11:22 UTC Run
prod-us ✅ Deployed 2026-07-09 11:51 UTC Run
prod-eu ✅ Deployed 2026-07-09 11:55 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants