feat(data-warehouse): implement hubplanner import source - #67818
Merged
Conversation
Contributor
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
Contributor
|
Reviews (1): Last reviewed commit: "feat(data-warehouse): implement hubplann..." | Re-trigger Greptile |
Contributor
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 1 · PR risk: 0/10 |
MarconLP
approved these changes
Jul 2, 2026
Fill in the scaffolded Hub Planner source end-to-end as a ResumableSource: declarative endpoint catalog in settings.py, requests-based paginated transport through the tracked session in hubplanner.py, and source wiring (validate_credentials, get_schemas, source_for_pipeline, non-retryable auth errors, canonical descriptions) in source.py. Incremental sync is enabled only on bookings and time_entries, the two areas whose POST /<area>/search exposes a searchable updatedDate filter; every other endpoint ships full refresh. Page-number pagination is persisted as a resume cursor. Kept behind unreleasedSource=True with releaseStatus=alpha. Generated-By: PostHog Code Task-Id: 5fd67b9c-0543-4880-8122-2c2f91efe2ec
The ty type checker can't narrow the optional incremental_search_field to str through the incremental_active bool, so the search-filter dict key inferred as str | None. Add an explicit None check so the key narrows to str. Generated-By: PostHog Code Task-Id: 5fd67b9c-0543-4880-8122-2c2f91efe2ec
Move the get_rows arg-type ignore onto the resumable_source_manager argument line (ruff had reflowed the call so the ignore sat on the closing paren), and narrow the source config field to SourceFieldInputConfig before asserting on its required/secret attributes. Generated-By: PostHog Code Task-Id: 5fd67b9c-0543-4880-8122-2c2f91efe2ec
Register the Hub Planner API key for value-based redaction on both the credential probe and the sync session so it can't leak into captured HTTP samples, and remove the unused `incremental_field` parameter that was threaded through `get_rows`/`hubplanner_source` but never read. Generated-By: PostHog Code Task-Id: 27926718-a0d3-459c-a99e-8cbccac634b9
Gilbert09
force-pushed
the
posthog-code/hubplanner-source
branch
from
July 2, 2026 23:58
5050929 to
1a82d10
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Hub Planner (resource scheduling, project planning, time tracking) had a scaffolded, hidden stub in the data warehouse sources registry but no working sync. Teams that run their delivery ops in Hub Planner couldn't pull projects, bookings, or time entries into PostHog to join against product and revenue data.
Changes
Implements the
hubplannersource end-to-end, following theimplementing-warehouse-sourcesskill'ssource.py/settings.py/hubplanner.pysplit.hubplanner.py): requests-based client throughmake_tracked_session(), page-number pagination (0-indexed,limit=1000, terminate on a short page), tenacity retries on 429/5xx, and aResumableSourcepage cursor persisted after each full page.settings.py): 13 endpoints (projects, resources, bookings, time entries, events, clients, milestones, project/resource groups, billing rates, holidays, vacations, project managers). Primary key_id, partition keycreatedDate(stable) where present.bookingsandtime_entries— the two areas whosePOST /<area>/searchexposes a searchableupdatedDatefilter. Everything else is full refresh, because Hub Planner only filters searchable fields and the other areas' search omitsupdatedDate(projects/resources have anupdatedDatefield but it isn't searchable). Incremental syncs POST to/<area>/searchwith{"updatedDate": {"$gte": ...}}, sorted ascending so the watermark advances (sort_mode="asc").source.py):validate_credentials(one/projectprobe), staticget_schemas,get_non_retryable_errors(403/401 auth failures),canonical_descriptions.pyfrom the official API docs,lists_tables_without_credentials = Trueso the docs render the table catalog. Kept behindunreleasedSource=TruewithreleaseStatus=ALPHA.generated_configs.py(HubplannerSourceConfig.api_key), movedhubplannerinto the Implemented table inSOURCES.md. Icon already present atfrontend/public/services/hubplanner.png.A few behaviors were verified against the live API without a token (the docs are the primary source, curl-confirmed where possible): both missing and invalid keys return 403 (not 401), and the 403 body echoes the supplied key back — so the transport never logs response bodies, to avoid leaking the credential. Auth is a raw API key in the
Authorizationheader with noBearerprefix, despite Hub Planner's docs calling it "OAuth 2.0 Bearer Token".How did you test this code?
I (Claude) added two test modules and ran them locally:
tests/test_hubplanner.py(transport, 29 cases): request-plan routing (GET vs POST-search, incremental filter body), pagination termination + resume-from-saved-page,_fetch_pagestatus mapping (429/5xx retry, bare-list vs non-list), the credential-not-logged guard, andvalidate_credentialsstatus mapping.tests/test_hubplanner_source.py(source class, 20 cases): schema/incremental gating, secret field config, non-retryable auth errors,source_for_pipelineplumbing, canonical-description coverage.All 49 pass. I also ran the shared
test_source_categories(1312) andtest_source_config_generator(15) suites to confirm registration and config generation stay green, plusruff check/ruff format. I could not run a real end-to-end sync (no Hub Planner API key), so the live incremental-filter and search-sort behaviors are asserted from the docs and unit tests, not a live pull.Regressions these tests catch that nothing else did: incremental routing sending the wrong method/path/filter; the paginator looping forever or losing the last page on resume; a future refactor logging the API-key-bearing 403 body; and the source silently advertising incremental on a full-refresh-only endpoint.
Docs update
The user-facing doc belongs in the posthog.com repo, which isn't checked out here, so it isn't in this PR. It needs to land at
contents/docs/cdp/sources/hubplanner.md(matchingdocsUrl).audit_source_docscouldn't be run for the same reason. Draft content:hubplanner.md
🤖 Agent context
Autonomy: Fully autonomous
Authored by Claude (Claude Code) executing a delegated task to implement the Hub Planner source. Invoked the
/implementing-warehouse-sources,/documenting-warehouse-sources, and/writing-testsskills.Key decisions:
ResumableSourceoverSimpleSource+WebhookSource. Page-number pagination gives a clean resume cursor, so resumable is the right base. I deferred webhook support: I couldn't verify thePOST /subscriptiondelivery payload shape or thehubplanner-tokensecret handling against a live account, and shipping unverified webhook transforms is riskier than a solid pull-only source. Left as a follow-up.updatedDateis searchable only for bookings and time entries; projects/resources expose the field but not as a search filter, so marking them incremental would fetch every page anyway (full-refresh cost) while pretending to be cheap.sortparam on full-refresh GETs. The docs markcreatedDatesortable on the core resources, but I couldn't curl-confirm it for every endpoint, and a rejected sort field 400s the whole sync. Since full refresh replaces the table each run, unsorted paging self-heals, so no-sort is the safe default. Incremental search keeps its ascending sort because the watermark needs it.POST /milestone/searchwith an empty body.No API credentials were available, so no live sync was run — the live incremental/search behaviors are documented uncertainties noted in code comments.