Skip to content

feat(data-warehouse): implement hubplanner import source - #67818

Merged
talyn-app[bot] merged 4 commits into
masterfrom
posthog-code/hubplanner-source
Jul 3, 2026
Merged

feat(data-warehouse): implement hubplanner import source#67818
talyn-app[bot] merged 4 commits into
masterfrom
posthog-code/hubplanner-source

Conversation

@Gilbert09

Copy link
Copy Markdown
Member

Problem

Hub Planner (resource scheduling, project planning, time tracking) had a scaffolded, hidden stub in the data warehouse sources registry but no working sync. Teams that run their delivery ops in Hub Planner couldn't pull projects, bookings, or time entries into PostHog to join against product and revenue data.

Changes

Implements the hubplanner source end-to-end, following the implementing-warehouse-sources skill's source.py / settings.py / hubplanner.py split.

  • Transport (hubplanner.py): requests-based client through make_tracked_session(), page-number pagination (0-indexed, limit=1000, terminate on a short page), tenacity retries on 429/5xx, and a ResumableSource page cursor persisted after each full page.
  • Endpoint catalog (settings.py): 13 endpoints (projects, resources, bookings, time entries, events, clients, milestones, project/resource groups, billing rates, holidays, vacations, project managers). Primary key _id, partition key createdDate (stable) where present.
  • Incremental: enabled only on bookings and time_entries — the two areas whose POST /<area>/search exposes a searchable updatedDate filter. Everything else is full refresh, because Hub Planner only filters searchable fields and the other areas' search omits updatedDate (projects/resources have an updatedDate field but it isn't searchable). Incremental syncs POST to /<area>/search with {"updatedDate": {"$gte": ...}}, sorted ascending so the watermark advances (sort_mode="asc").
  • Source wiring (source.py): validate_credentials (one /project probe), static get_schemas, get_non_retryable_errors (403/401 auth failures), canonical_descriptions.py from the official API docs, lists_tables_without_credentials = True so the docs render the table catalog. Kept behind unreleasedSource=True with releaseStatus=ALPHA.
  • Regenerated generated_configs.py (HubplannerSourceConfig.api_key), moved hubplanner into the Implemented table in SOURCES.md. Icon already present at frontend/public/services/hubplanner.png.

A few behaviors were verified against the live API without a token (the docs are the primary source, curl-confirmed where possible): both missing and invalid keys return 403 (not 401), and the 403 body echoes the supplied key back — so the transport never logs response bodies, to avoid leaking the credential. Auth is a raw API key in the Authorization header with no Bearer prefix, despite Hub Planner's docs calling it "OAuth 2.0 Bearer Token".

How did you test this code?

I (Claude) added two test modules and ran them locally:

  • tests/test_hubplanner.py (transport, 29 cases): request-plan routing (GET vs POST-search, incremental filter body), pagination termination + resume-from-saved-page, _fetch_page status mapping (429/5xx retry, bare-list vs non-list), the credential-not-logged guard, and validate_credentials status mapping.
  • tests/test_hubplanner_source.py (source class, 20 cases): schema/incremental gating, secret field config, non-retryable auth errors, source_for_pipeline plumbing, canonical-description coverage.

All 49 pass. I also ran the shared test_source_categories (1312) and test_source_config_generator (15) suites to confirm registration and config generation stay green, plus ruff check/ruff format. I could not run a real end-to-end sync (no Hub Planner API key), so the live incremental-filter and search-sort behaviors are asserted from the docs and unit tests, not a live pull.

Regressions these tests catch that nothing else did: incremental routing sending the wrong method/path/filter; the paginator looping forever or losing the last page on resume; a future refactor logging the API-key-bearing 403 body; and the source silently advertising incremental on a full-refresh-only endpoint.

Docs update

The user-facing doc belongs in the posthog.com repo, which isn't checked out here, so it isn't in this PR. It needs to land at contents/docs/cdp/sources/hubplanner.md (matching docsUrl). audit_source_docs couldn't be run for the same reason. Draft content:

hubplanner.md
---
title: Linking Hub Planner as a source
sidebar: Docs
showTitle: true
availability: { free: full, selfServe: full, enterprise: full }
sourceId: Hubplanner
---

import SourceSetupIntro from "../_snippets/source-setup-intro.mdx"
import SyncModes from "../_snippets/sync-modes.mdx"
import TroubleshootingLink from "../_snippets/dw-troubleshooting-link.mdx"
import AlphaRelease from "../_snippets/alpha-release.mdx"

<AlphaRelease />

[Hub Planner](https://hubplanner.com) is a resource-scheduling, project-planning, and time-tracking tool. Linking it as a source pulls your projects, resources, bookings, time entries, and related scheduling data into the PostHog data warehouse, where you can join it with your product and revenue data.

## Prerequisites

Connecting Hub Planner requires an **admin** account, which is needed to generate an API key. A **Read Only** key is sufficient for syncing.

## Adding a data source

<SourceSetupIntro />

You need a Hub Planner API key:

1. In Hub Planner, go to **Settings → API** (admin access required).
2. Generate a new **Read Only** API key.
3. Copy the key and paste it into the **API key** field when setting up the source in PostHog.

## Sync modes

<SyncModes />

Most Hub Planner tables are full-refresh only, because the API only exposes a server-side date filter on the areas that support search. **Bookings** and **Time entries** additionally support incremental sync via their `updatedDate` field — prefer incremental for those to keep syncs cheap.

## Configuration

<SourceParameters />

## Supported tables

<SourceTables />

## Troubleshooting

- **Invalid API key**: Hub Planner returns a `403` for both an invalid key and a key without the required permissions. Regenerate a **Read Only** key under **Settings → API** and reconnect.
- **Slow or throttled syncs**: Hub Planner enforces a daily cap (6,000 calls/day per account) and a burst limit (50 requests / 5 seconds). Large initial backfills page through data automatically and back off on rate limits, so they may take a while to complete.

<TroubleshootingLink />

🤖 Agent context

Autonomy: Fully autonomous

Authored by Claude (Claude Code) executing a delegated task to implement the Hub Planner source. Invoked the /implementing-warehouse-sources, /documenting-warehouse-sources, and /writing-tests skills.

Key decisions:

  • Chose ResumableSource over SimpleSource+WebhookSource. Page-number pagination gives a clean resume cursor, so resumable is the right base. I deferred webhook support: I couldn't verify the POST /subscription delivery payload shape or the hubplanner-token secret handling against a live account, and shipping unverified webhook transforms is riskier than a solid pull-only source. Left as a follow-up.
  • Enabled incremental only where the API genuinely filters server-side. Cross-referencing every area's "Searchable Properties" table showed updatedDate is searchable only for bookings and time entries; projects/resources expose the field but not as a search filter, so marking them incremental would fetch every page anyway (full-refresh cost) while pretending to be cheap.
  • Dropped the sort param on full-refresh GETs. The docs mark createdDate sortable on the core resources, but I couldn't curl-confirm it for every endpoint, and a rejected sort field 400s the whole sync. Since full refresh replaces the table each run, unsorted paging self-heals, so no-sort is the safe default. Incremental search keeps its ascending sort because the watermark needs it.
  • Milestones have no GET-all endpoint, so they list via POST /milestone/search with an empty body.

No API credentials were available, so no live sync was run — the live incremental/search behaviors are documented uncertainties noted in code comments.

@Gilbert09 Gilbert09 added skip-agent-review Save $$$, skip auto agent reviews (Greptile) — use for trivial or chore PRs and removed skip-agent-review Save $$$, skip auto agent reviews (Greptile) — use for trivial or chore PRs labels Jul 2, 2026
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

  • Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
  • Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

@greptile-apps

greptile-apps Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Reviews (1): Last reviewed commit: "feat(data-warehouse): implement hubplann..." | Re-trigger Greptile

@Gilbert09
Gilbert09 marked this pull request as ready for review July 2, 2026 13:39
Copilot AI review requested due to automatic review settings July 2, 2026 13:39
@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team July 2, 2026 13:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@veria-ai

veria-ai Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

Gilbert09 added 4 commits July 3, 2026 00:58
Fill in the scaffolded Hub Planner source end-to-end as a ResumableSource:
declarative endpoint catalog in settings.py, requests-based paginated transport
through the tracked session in hubplanner.py, and source wiring
(validate_credentials, get_schemas, source_for_pipeline, non-retryable auth
errors, canonical descriptions) in source.py.

Incremental sync is enabled only on bookings and time_entries, the two areas
whose POST /<area>/search exposes a searchable updatedDate filter; every other
endpoint ships full refresh. Page-number pagination is persisted as a resume
cursor. Kept behind unreleasedSource=True with releaseStatus=alpha.

Generated-By: PostHog Code
Task-Id: 5fd67b9c-0543-4880-8122-2c2f91efe2ec
The ty type checker can't narrow the optional incremental_search_field to str
through the incremental_active bool, so the search-filter dict key inferred as
str | None. Add an explicit None check so the key narrows to str.

Generated-By: PostHog Code
Task-Id: 5fd67b9c-0543-4880-8122-2c2f91efe2ec
Move the get_rows arg-type ignore onto the resumable_source_manager argument
line (ruff had reflowed the call so the ignore sat on the closing paren), and
narrow the source config field to SourceFieldInputConfig before asserting on its
required/secret attributes.

Generated-By: PostHog Code
Task-Id: 5fd67b9c-0543-4880-8122-2c2f91efe2ec
Register the Hub Planner API key for value-based redaction on both the
credential probe and the sync session so it can't leak into captured HTTP
samples, and remove the unused `incremental_field` parameter that was
threaded through `get_rows`/`hubplanner_source` but never read.

Generated-By: PostHog Code
Task-Id: 27926718-a0d3-459c-a99e-8cbccac634b9
@Gilbert09
Gilbert09 force-pushed the posthog-code/hubplanner-source branch from 5050929 to 1a82d10 Compare July 2, 2026 23:58
@talyn-app
talyn-app Bot merged commit 2e0a821 into master Jul 3, 2026
212 checks passed
@talyn-app
talyn-app Bot deleted the posthog-code/hubplanner-source branch July 3, 2026 00:29
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 3, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-03 00:59 UTC Run
prod-us ✅ Deployed 2026-07-03 01:12 UTC Run
prod-eu ✅ Deployed 2026-07-03 01:13 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants