feat(data-warehouse): implement hyperspell import source

[Gilbert09]

Problem

Hyperspell (an API-based memory layer for AI apps and agents) was scaffolded as a data warehouse source in #69517 but had no sync logic: empty fields, no schemas, no pipeline wiring. Users can't yet pull their agent-memory corpus into PostHog to analyze it alongside product data.

Why: part of the batch effort to turn the scaffolded SaaS/AI source stubs into working connectors, so this PR targets the scaffold branch and will retarget to master when #69517 merges.

Changes

Implements the source end to end following the source.py / settings.py / hyperspell.py architecture:

• Six tables, chosen from Hyperspell's published OpenAPI spec: memories, connections, integrations, entities, queries (query log), context_documents. All full refresh - no Hyperspell list endpoint exposes a server-side updated-since filter (memories only filter on source/collection/status/metadata), so no endpoint claims supports_incremental.
• ResumableSource using Hyperspell's cursor/next_cursor pagination. The cursor of the page most recently yielded is saved after each batch, so a resume re-fetches that page and merge-dedupes rather than risking skipped rows. connections and integrations are single unpaginated responses.
• Region config (US default, EU option): Hyperspell runs two isolated regions with separate base URLs and region-locked API keys.
• Optional user ID: memories are per-user scoped, so an app-level key plus the X-As-User header syncs one user's data; blank syncs app-scoped data only.
• All HTTP goes through make_tracked_session() with key redaction, bounded tenacity retries on 429/5xx, and non-retryable 401/403 mappings.
• Stable partition keys only (ingested_at, created_at, time) - never moving fields like last_modified_at. Composite primary key [source, resource_id] on memories since resource_id is only unique within its provider.
• memories lowers the per-chunk byte cap to 100 MiB since rows embed the full nested document payload.
• Canonical table/column descriptions sourced from the official API reference, lists_tables_without_credentials = True so the docs table list renders, generated HyperspellSourceConfig, SOURCES.md entry, and the official Hyperspell SVG icon (their favicon, with light/dark support).
• Kept behind unreleasedSource=True with releaseStatus="alpha" per the batch rollout plan.

How did you test this code?

• pytest products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/tests/ - 58 tests pass. They cover: cursor pagination follows next_cursor and terminates, resume state is saved per yielded page and honored on restart, unpaginated endpoints make exactly one request, per-endpoint page-size param names (size vs limit), region base URL selection, X-As-User header wiring, credential validation status mapping, retryable vs non-retryable error classification, and the SourceResponse shape (primary keys, partitioning, sort mode, chunk byte cap) per endpoint.
• pytest .../sources/common/test/ - 362 pass (10 pre-existing errors in MinIO-backed webhook tests that need a live MinIO container, unrelated).
• Ran the real SourceConfigGenerator against the registry to confirm the committed HyperspellSourceConfig matches generator output exactly.
• ruff check / ruff format clean; hogli ci:preflight --fix reports 0 failures.
• Verified live API behavior anonymously with curl: fetched the OpenAPI spec from api.hyperspell.com, confirmed pagination/response shapes against it, and confirmed 401 (invalid key) / 403 (missing auth) behavior on both the US and EU hosts. I don't have Hyperspell credentials, so an authenticated end-to-end sync wasn't run - endpoint logic follows the published spec, and the absence of incremental filters means no future-date filter smoke test was applicable (nothing claims server-side filtering).

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

The user-facing doc lives in the posthog.com repo, which isn't available in this environment. docsUrl is set to https://posthog.com/docs/cdp/sources/hyperspell; the doc below needs to land at contents/docs/cdp/sources/hyperspell.md (and audit_source_docs run) as a follow-up.

Drafted doc content for posthog.com

---
title: Linking Hyperspell as a source
sidebar: Docs
showTitle: true
availability: { free: full, selfServe: full, enterprise: full }
sourceId: Hyperspell
beta: true
---

import SourceSetupIntro from "../_snippets/source-setup-intro.mdx"
import SyncModes from "../_snippets/sync-modes.mdx"
import TroubleshootingLink from "../_snippets/dw-troubleshooting-link.mdx"
import AlphaRelease from "../_snippets/alpha-release.mdx"

<AlphaRelease />

Hyperspell is a memory layer for AI apps and agents. This connector syncs your indexed memories, data-source connections, integrations, extracted entities, query logs and generated context documents into PostHog, so you can analyze your agent's memory corpus alongside your product data.

## Prerequisites

- A Hyperspell account with an API key, created in the [Hyperspell dashboard](https://dashboard.hyperspell.com).
- Know which region your API key was created in (US or EU). Keys are only valid in their own region.

## Adding a data source

<SourceSetupIntro />

You'll need:

- **API key**: created in the Hyperspell dashboard.
- **Region**: the region the key was created in. US (api.hyperspell.com) is the default; EU keys must select EU (api.eu.hyperspell.com).
- **User ID** (optional): Hyperspell memories are scoped per user. Set a user ID to sync that user's memories. Leave blank to sync app-scoped data only.

## Sync modes

<SyncModes />

Hyperspell's API doesn't expose server-side timestamp filters, so all tables sync as full refresh.

## Configuration

<SourceParameters />

## Supported tables

<SourceTables />

## Troubleshooting

- **Invalid API key**: check that the key is correct and that the selected region matches the region the key was created in. A US key fails with 401 against the EU endpoint and vice versa.

<TroubleshootingLink />

🤖 Agent context

Autonomy: Fully autonomous

• Built with Claude Code (PostHog Code cloud task). Skills invoked: implementing-warehouse-sources, documenting-warehouse-sources, writing-tests.
• Modeled on the instatus source (plain requests + tracked session + ResumableSourceManager) rather than the declarative rest_source.RESTClient path - Hyperspell's per-endpoint quirks (different page-size param names, different response wrapper keys, two unpaginated endpoints) fit a small endpoint-config dataclass better than the generic REST config.
• Considered client-side high-watermark filtering on ingested_at for pseudo-incremental sync of memories; rejected it since the API documents no ordering guarantee for /memories/list, so a watermark could silently skip rows. Full refresh is the honest mode.
• The environment lacked a flox setup, so HyperspellSourceConfig in generated_configs.py was first hand-written, then verified byte-for-byte by running SourceConfigGenerator directly in a uv sync venv (the management command itself needs a DB connection this environment doesn't have).
• No Hyperspell credentials were available, so live verification was limited to anonymous auth-behavior probes and the published OpenAPI spec (noted in the testing section).

---

Created with PostHog Code

[github-actions]

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

• Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
• Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

[github-actions]

🦔 Hogbox preview · ✅ ready

▶ Open the preview

🔑 Login	test@posthog.com / 12345678 (demo data)
🧩 Running	this PR's backend and frontend, on the PostHog :master base
🔗 Link	stable across rebuilds — a re-push swaps the box underneath, the URL stays
🔒 Access	tailnet only (PostHog VPN)
🛠️ Admin	inspect & debug state in hogland
💤 Idle	sleeps after ~30 min idle (snapshot to S3, zero node cost) and wakes on your next visit in ~30s, behind a brief "waking up" screen

_{commit 19f2da3 · box box-0dad96e456a3 · ready in 741s (push → usable) · build log · rebuilds on every push, torn down on close}

[greptile-apps]

Comments Outside Diff (1)

1. products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/tests/test_hyperspell.py, line 677-686 (link)

   Prefer @parameterized.expand for sync tests in new files

   The repo-wide convention (per CLAUDE.md) is @parameterized.expand from the parameterized library for parameterised sync tests; @pytest.mark.parametrize is the exception reserved for async test functions where parameterized.expand cannot inject. Both test_hyperspell.py and test_hyperspell_source.py are new files with only sync tests, so @parameterized.expand is the expected choice. The pattern recurs on several test methods across both files (e.g. test_fetch_page_retryable_statuses_raise, test_page_size_param_per_endpoint, test_status_mapping, test_non_retryable_errors, and others).

   Rule Used: Use @parameterized.expand from the `parameterize... (source)

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

_{Reviews (1): Last reviewed commit: "feat(data-warehouse): implement hyperspe..." | Re-trigger Greptile}

[github-actions]

🤖 CI report

✅ Bundle size — no change

Uncompressed size of every built .js bundle, compared against the base branch.

Total: 64.76 MiB · no change

No file changed by more than 1000 B.

_{Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report}

✅ Eager graph — within budget

How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.

Root	Eager (shipped)	Δ vs base	Budget
entry (logged-out pages, app bootstrap) src/index.tsx	1.22 MiB · 22 files	no change	███░░░░░░░ 28.4% of 4.29 MiB
authenticated shell (every logged-in page) src/scenes/AuthenticatedShell.tsx	8.14 MiB · 2,977 files	no change	█████████░ 88.0% of 9.25 MiB

🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx

Largest files eagerly shipped from src/index.tsx

Size	File
126.8 KiB	../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js
24.6 KiB	../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js
6.3 KiB	../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js
4.5 KiB	../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js
3.9 KiB	../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js
1.4 KiB	../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js
1.3 KiB	src/RootErrorBoundary.tsx
912 B	../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js
789 B	src/scenes/ChunkLoadErrorBoundary.tsx
762 B	src/index.tsx

Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx

Size	File
280.3 KiB	../node_modules/.pnpm/posthog-js@1.400.1/node_modules/posthog-js/dist/rrweb.js
267.7 KiB	../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js
235.5 KiB	src/taxonomy/core-filter-definitions-by-group.json
222.7 KiB	../node_modules/.pnpm/posthog-js@1.400.1/node_modules/posthog-js/dist/module.js
164.0 KiB	src/queries/validators.js
154.3 KiB	../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js
126.8 KiB	../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js
105.9 KiB	src/lib/api.ts
93.3 KiB	../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js
92.7 KiB	../packages/quill/packages/quill/dist/index.js

_{Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479}

✅ Dist folder size — 🔺 +1.6 KiB (+0.0%)

Total size of the built frontend/dist folder (all assets), compared against the base branch.

Total: 1307.79 MiB · 🔺 +1.6 KiB (+0.0%)

✅ Playwright — all passed

All tests passed.

View test results →

⚠️ Backend coverage — 99.0% of changed backend lines covered — 3 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ████████████████████ 99.0% (376 / 379)

File	Patch	Uncovered changed lines
products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/hyperspell.py	96.8%	158–159, 174

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29405511751 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
demo	███████████░░░░░░░░░ 56.2%	1,497 / 2,663
tasks	█████████████░░░░░░░ 67.4%	25,425 / 37,703
signals	████████████████░░░░ 79.0%	19,009 / 24,055
data_modeling	████████████████░░░░ 80.0%	4,834 / 6,045
cdp	████████████████░░░░ 80.7%	3,118 / 3,864
notebooks	█████████████████░░░ 83.8%	6,086 / 7,259
agent_platform	█████████████████░░░ 84.2%	3,112 / 3,695
cohorts	█████████████████░░░ 86.0%	3,989 / 4,639
actions	█████████████████░░░ 86.6%	717 / 828
product_tours	█████████████████░░░ 87.5%	1,266 / 1,447
exports	██████████████████░░ 88.3%	6,861 / 7,769
conversations	██████████████████░░ 88.9%	16,028 / 18,034
dashboards	██████████████████░░ 89.1%	5,648 / 6,337
mcp_analytics	██████████████████░░ 89.1%	2,502 / 2,807
error_tracking	██████████████████░░ 89.6%	9,749 / 10,883
engineering_analytics	██████████████████░░ 89.8%	4,861 / 5,414
streamlit_apps	██████████████████░░ 90.4%	2,499 / 2,764
slack_app	██████████████████░░ 90.6%	9,511 / 10,503
marketing_analytics	██████████████████░░ 90.8%	11,514 / 12,684
alerts	██████████████████░░ 90.9%	3,416 / 3,760
product_analytics	██████████████████░░ 91.1%	5,527 / 6,068
data_warehouse	██████████████████░░ 92.0%	17,807 / 19,349
workflows	██████████████████░░ 92.0%	4,795 / 5,210
ai_observability	███████████████████░ 92.7%	14,670 / 15,822
web_analytics	███████████████████░ 92.7%	13,607 / 14,674
surveys	███████████████████░ 92.9%	5,660 / 6,094
posthog_ai	███████████████████░ 93.2%	1,312 / 1,408
approvals	███████████████████░ 93.3%	3,395 / 3,640
reminders	███████████████████░ 93.4%	468 / 501
early_access_features	███████████████████░ 93.8%	848 / 904
endpoints	███████████████████░ 94.1%	8,606 / 9,143
skills	███████████████████░ 94.4%	2,827 / 2,995
revenue_analytics	███████████████████░ 94.5%	3,598 / 3,809
review_hog	███████████████████░ 94.6%	6,475 / 6,848
logs	███████████████████░ 95.3%	9,476 / 9,942
experiments	███████████████████░ 95.6%	24,005 / 25,112
replay_vision	███████████████████░ 95.7%	13,300 / 13,896
feature_flags	███████████████████░ 96.1%	14,721 / 15,325
warehouse_sources	███████████████████░ 96.1%	217,424 / 226,136
annotations	███████████████████░ 96.2%	732 / 761
user_interviews	███████████████████░ 96.4%	2,242 / 2,325
data_catalog	███████████████████░ 97.2%	1,642 / 1,689
customer_analytics	███████████████████░ 97.3%	7,442 / 7,648

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

[trunk-io]

     

View Full Report ↗︎ ⋅ Docs

[Gilbert09]

Re: the @parameterized.expand vs @pytest.mark.parametrize suggestion (greptile) — I'm keeping @pytest.mark.parametrize here. It's the established convention for the data warehouse source tests in this exact directory (products/warehouse_sources/backend/temporal/data_imports/sources/*/tests/), where sync test methods use @pytest.mark.parametrize throughout (e.g. active_campaign, adroll, agilecrm). Switching just hyperspell to parameterized.expand would make it inconsistent with every sibling source test. Consistency with the surrounding code wins here.

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 2 · PR risk: 0/10

[Gilbert09]

Note: the Visual regression tests - chromium (2/16) shard has intermittently failed on products/customer_analytics/frontend/components/Accounts/AccountsTab.stories.tsx (a page.waitForSelector timeout on [data-attr="account-expansion"] / .DataVisualization canvas). This story is not touched by this PR (it's Customer analytics, not the data-warehouse source) and is a known pre-existing flake — the most recent change to that file on master is #69464, fix(test): wait for DataVisualization canvas in play fn to prevent flaky timeout. It passes on re-run; re-triggering the shard.

[Gilbert09]

CI status update

All checks owned by this PR are green:

• ✅ All 5 Product tests (warehouse-sources) shards pass
• ✅ Hyperspell backend tests (59) pass locally
• ✅ Merge conflict resolved; branch is up to date with master and mergeable

The earlier Django tests – Temporal and Jest test (FOSS - 1) failures were transient GitHub infra errors (HTTP 504 Gateway Time-out in the quarantine-gate artifact download step, not test failures) and pass on re-run.

The one remaining red check, Python code quality checks, is a master-wide breakage unrelated to this PR — master tip is itself red on this exact check. It fails in tools/hogli-commands/hogli_commands/tests/test_devbox.py::TestDevboxCommands (devbox CLI tooling): devbox:setup now calls mutagen.ensure_mutagen_installed() / ensure_daemon_with_shim() / ensure_user_mutagen_config() (devbox/cli.py:1131-1137), but test_devbox_setup_runs_explicit_setup_steps (and siblings) don't monkeypatch those, so the command shells out to a missing mutagen binary and raises FileNotFoundError. This PR touches nothing under tools/hogli-commands/, so the failure is inherited from master and will clear once the devex owners fix it there (I'll re-merge master to pick up the fix).

[danielcarletti]

LGTM

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-15 10:50 UTC	Run
prod-us	✅ Deployed	2026-07-15 11:03 UTC	Run
prod-eu	✅ Deployed	2026-07-15 11:09 UTC	Run