feat(data-warehouse): implement retently import source - #69393
Conversation
Implements the Retently source: customers, companies, feedback, outbox, campaigns, templates, and reports endpoints via the v2 REST API with X-Api-Key auth, page-number pagination, resumable syncs, and incremental sync on feedback via the server-side startDate filter. Generated-By: PostHog Code Task-Id: 023c3eab-e55c-429f-b546-084ca0b13ced
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
Companion to PostHog/posthog#69393, which implements the Retently import source. Generated-By: PostHog Code Task-Id: 023c3eab-e55c-429f-b546-084ca0b13ced
|
Moves response extraction inside the retried fetch scope so a transiently malformed payload retries the single request instead of failing the sync, and parameterizes the loop-based test cases. Generated-By: PostHog Code Task-Id: 023c3eab-e55c-429f-b546-084ca0b13ced
Generated-By: PostHog Code Task-Id: 81b3dd34-5f9c-4013-a52c-ed16fb84ca7d
Companion to PostHog/posthog#69393, which implements the Retently import source. Generated-By: PostHog Code Task-Id: 023c3eab-e55c-429f-b546-084ca0b13ced
Problem
Retently (NPS/CSAT/CES surveys) was scaffolded as a data warehouse source but had no sync logic, so users can't pull their customer feedback data into PostHog.
Changes
Implements the Retently source end to end against the v2 REST API (
https://app.retently.com/api/v2,X-Api-Keyauth):customers,companies,feedback,outbox,campaigns,templates,reports– matching the stream list of community connectors for Retently.ResumableSourcewith the standardsource.py/settings.py/retently.pysplit. Page-number pagination checkpoints the next page to Redis after each yielded batch, and all HTTP goes throughmake_tracked_session()with the key redacted. 429/5xx retry with bounded exponential backoff (Retently rate-limits at ~150 req/min).feedback, where the documentedstartDatequery param filters server-side by response creation date. Customers also documentsstartDate/endDatebut doesn't say which field it filters and customer records are mutable, so it stays full refresh (as do companies, which have no date filter). Outbox rows carry no consistently documented unique id, so it ships full refresh with no merge key.data.<key>, a bare list underdatafor reports, top-level arrays for campaigns/templates), and pagination follows thepagesmetadata whether it appears insidedataor at the top level – the docs are inconsistent between endpoints.canonical_descriptions.pydocuments every table/column from the official API docs, andlists_tables_without_credentials = Truelets the public docs render the table catalog.unreleasedSource=TruewithreleaseStatus="alpha".SOURCES.mdmoved from Scaffolded to Implemented;generated_configs.pyregenerated (Retently class only).Note
I couldn't curl-verify success-path behavior against a live account (no Retently credentials available), so a few choices are deliberately conservative and commented in code: the incremental endpoint requests
sort=createdDateascending but declaressort_mode="desc"so the watermark only persists after a fully successful sync, and unverifiable incremental candidates ship as full refresh. Verifying against a live account is the remaining step before releasing the source.The user-facing doc (
contents/docs/cdp/sources/retently.md) was written per the docs template and lands via PostHog/posthog.com#18294;audit_source_docspasses for Retently against that doc.How did you test this code?
pytest products/warehouse_sources/backend/temporal/data_imports/sources/retently/tests/(57 tests pass): envelope extraction per documented response shape, pagination viapagesmetadata in both documented locations (catches a silent short-page termination if the API capslimit), incrementalstartDateplumbing and omission, resume-from-saved-page and save-after-yield semantics, retry on 429/5xx without real sleeps, credential validation status mapping, and per-endpointSourceResponseshape (primary keys, partitioning, sort mode).pytest products/warehouse_sources/backend/temporal/data_imports/sources/tests/(registry-wide checks incl. source categories, 1353 pass).ruff check/ruff formatandhogli ci:preflight --fixpass. No manual testing against a live Retently account was possible.Automatic notifications
Docs update
Doc written for posthog.com at
contents/docs/cdp/sources/retently.md(PostHog/posthog.com#18294).🤖 Agent context
Autonomy: Human-driven (agent-assisted)
/implementing-warehouse-sources,/writing-tests, and/documenting-warehouse-sourcesskills.secoda/ruddr/huntr/aircall/beamerwere used as reference implementations.outbox(documentedstartDatefilter) but rejected it: example rows lack a consistent unique id, so there's no safe merge key.Created with PostHog Code