feat(data-warehouse): implement workable import source#65657
Conversation
Fills in the scaffolded Workable (ATS / recruiting) data warehouse source end to end. Implements a ResumableSource over Workable's SPI v3 REST API: jobs, candidates, members, recruiters and stages. Jobs and candidates sync incrementally via the server-side updated_after / created_after filters; the rest are full refresh. Pagination follows the paging.next cursor, with resume state saved after each page. Generated-By: PostHog Code Task-Id: 31740fe2-f367-4984-8fa8-a913aa1d6c13
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
|
Reviews (1): Last reviewed commit: "feat(data-warehouse): implement workable..." | Re-trigger Greptile |
Fix the mypy errors flagged by CI (all in the test modules: loosen the page-dict type, annotate an empty dict, pass response= to HTTPError, narrow the field type before reading .secret, type the patched return as Any). Also tighten the subdomain validation regex to reject a trailing hyphen, which is not a valid DNS label and would otherwise fail at the HTTP layer with a confusing network error rather than a clear validation message. Generated-By: PostHog Code Task-Id: 31740fe2-f367-4984-8fa8-a913aa1d6c13
Pass redact_values=(api_token,) to the tracked sessions in get_rows and validate_credentials so the Workable bearer token is scrubbed from any captured HTTP request/response samples, consistent with other connectors. Generated-By: PostHog Code Task-Id: 42bb41c5-6cc6-4fd1-9654-af539b030e73
Generated-By: PostHog Code Task-Id: 42bb41c5-6cc6-4fd1-9654-af539b030e73
CI status updateThe mypy / Python code-quality checks that were failing are now green, and the two security review threads (Workable token redaction in tracked HTTP sessions) have been addressed in The branch is up to date with Two checks remain red, but neither is caused by this PR:
No further code changes are warranted here; the migration-job failure needs to clear at the infra level (or a fresh re-run once master CI is healthy). |
Generated-By: PostHog Code Task-Id: 42bb41c5-6cc6-4fd1-9654-af539b030e73
Problem
Workable (a cloud applicant tracking / recruiting platform) was registered as a scaffolded data warehouse source — visible in the registry but with no sync logic. Customers running recruiting on Workable couldn't pull their jobs and candidates into PostHog for analysis.
Changes
Implements the Workable source end to end against the SPI v3 REST API, following the architecture contract from the
implementing-warehouse-sourcesskill (source.py/settings.py/workable.pysplit):jobs,candidates,members,recruiters,stages— the canonical Workable connector stream set.ResumableSource. Pagination follows thepaging.nextcursor URL; resume state (the next URL) is persisted after each page so a heartbeat-timeout can pick up where it left off rather than restarting.jobsandcandidatesexpose genuine server-side time filters (updated_after/created_after), so they support incremental + append onupdated_at(default) orcreated_at.members,recruitersandstageshave no update timestamp (and stages/recruiters aren't even paginated) so they ship full refresh.connection_host_fieldsso retargeting it re-requires the token (prevents credential exfiltration to an attacker-controlled host).make_tracked_session(). Tenacity owns retries (the adapter's own status retries are disabled to avoid double-retrying) and backs off on 429 / 5xx to stay clear of Workable's ~10 req/10s limit. 401/403/invalid-subdomain are mapped as non-retryable.canonical_descriptions.pyfor all five tables from the official docs.unreleasedSource=TruewithreleaseStatus=ALPHA.A note on sort order
Workable paginates by
since_id(ascending id, which tracks ascendingcreated_at) and has no way to sort byupdated_at. Sosort_modeis chosen per run:ascwhen the cursor field iscreated_at(arrival order matches the cursor, watermark advances per batch),descwhen it'supdated_at(arrival order is unrelated to the cursor, so the watermark commit is deferred to sync completion — avoiding advancing past unsynced rows on a partial failure). Merge dedupes on the primary key in all cases.How did you test this code?
I'm an agent. I verified the live API shape with
curland the public docs (response wrappers,paging.next, theupdated_after/created_afterparams, candidate/job field presence) before finalizing endpoint logic — an unauthenticated probe confirmed thehttps://<subdomain>.workable.com/spi/v3/host and a 401 on missing auth. I could not curl the authenticated endpoints (no token), so the incremental filter behavior is taken from the documented params rather than a future-date smoke test; this is called out in code comments.Automated tests I actually ran (all passing):
posthog/temporal/data_imports/sources/workable/tests/test_workable.py— transport: subdomain validation (incl. host-injection attempts), datetime formatting, URL/filter construction, sort-mode selection, cursor pagination + resume + state-save-after-yield, retryable vs non-retryable status handling,SourceResponseshape per endpoint, credential status mapping.posthog/temporal/data_imports/sources/workable/tests/test_workable_source.py— source class: config/fields, per-endpoint incremental support, schema filtering, credential 401/403 (create vs schema) mapping, canonical descriptions, argument plumbing.posthog/temporal/data_imports/sources/tests/test_source_categories.py(CI-enforced category coverage).ruff checkandruff formaton the changed files.Docs update
A docs page at
posthog.com/docs/cdp/sources/workable(referenced bydocsUrl) does not exist yet and would need to be added separately.🤖 Agent context
Autonomy: Fully autonomous
implementing-warehouse-sourcesskill and followed its end-to-end workflow and conventions; used the repo'sgenerate_source_configsmanagement command to regenerateWorkableSourceConfig.klaviyosource (also a cursor-paginatedResumableSource).events/activities; (2)membersrelies onpaging.next+limit=100and would truncate accounts with >100 members (the docs show no paging object for it) — acceptable for alpha, noted in code; (3) thesort_modereasoning above; (4) webhooks are out of scope for this PR despite Workable supporting subscription management — the pull source stands on its own and webhooks can be layered on later.