feat(data-warehouse): implement mem0 import source - #70716
Conversation
|
Companion posthog.com doc — ready to land at contents/docs/cdp/sources/mem0.md---
title: Linking Mem0 as a source
sidebar: Docs
showTitle: true
availability:
free: full
selfServe: full
enterprise: full
sourceId: Mem0
---
import SourceSetupIntro from "../_snippets/source-setup-intro.mdx"
import SyncModes from "../_snippets/sync-modes.mdx"
import TroubleshootingLink from "../_snippets/dw-troubleshooting-link.mdx"
import AlphaRelease from "../_snippets/alpha-release.mdx"
<AlphaRelease />
The Mem0 connector syncs your AI agent memory data into the PostHog Data warehouse, so you can analyze what your agents remember about users alongside your product data.
## Prerequisites
You need a Mem0 platform account with an API key. Note that Mem0's free tier has monthly usage limits, and syncs count towards them.
## Adding a data source
<SourceSetupIntro />
When linking Mem0, you'll need:
- **API key** – find it in the [Mem0 dashboard](https://app.mem0.ai/dashboard/api-keys) under **API keys**.
- **Organization ID** and **Project ID** (optional) – scope the entities table to a specific Mem0 organization or project.
## Sync modes
<SyncModes />
The memories table supports incremental syncs on `updated_at` (recommended) or `created_at`. Entities and events don't expose a server-side timestamp filter, so they sync as full refreshes.
## Configuration
<SourceParameters />
## Supported tables
<SourceTables />
## Troubleshooting
- If you see an authentication error, your API key is invalid or has been revoked. Create a new key in the Mem0 dashboard, then reconnect.
- If syncs fail with rate-limit errors, you may have hit your Mem0 plan's monthly usage limits. Reduce sync frequency or upgrade your Mem0 plan.
<TroubleshootingLink /> |
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
![hex-security-app[bot]](https://avatars.githubusercontent.com/in/2665233?s=60&v=4){kind=small}
🤖 CI report✅ Bundle size — no changeUncompressed size of every built Total: 64.77 MiB · no change No file changed by more than 1000 B. Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report ✅ Eager graph — within budgetHow much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy
🟢 Largest files eagerly shipped from
|
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 762 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 281.3 KiB | ../node_modules/.pnpm/posthog-js@1.402.3/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 235.5 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 222.9 KiB | ../node_modules/.pnpm/posthog-js@1.402.3/node_modules/posthog-js/dist/module.js |
| 164.0 KiB | src/queries/validators.js |
| 154.3 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.8 KiB | src/lib/api.ts |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 92.7 KiB | ../packages/quill/packages/quill/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Dist folder size — 🔺 +5.3 KiB (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1312.49 MiB · 🔺 +5.3 KiB (+0.0%)
⚠️ Backend coverage — 99.0% of changed backend lines covered — 4 uncovered
🧪 Backend test coverage
Patch coverage — changed backend lines (products + core): ████████████████████ 99.0% (439 / 443)
| File | Patch | Uncovered changed lines |
|---|---|---|
products/warehouse_sources/backend/temporal/data_imports/sources/mem0/tests/test_mem0.py |
98.2% | 257, 272, 299 |
products/warehouse_sources/backend/temporal/data_imports/sources/mem0/mem0.py |
99.1% | 266 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29457753842 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
demo |
███████████░░░░░░░░░ 56.2% |
1,497 / 2,663 |
tasks |
█████████████░░░░░░░ 67.4% |
25,560 / 37,895 |
signals |
████████████████░░░░ 79.1% |
19,037 / 24,073 |
data_modeling |
████████████████░░░░ 80.0% |
4,834 / 6,045 |
cdp |
████████████████░░░░ 80.7% |
3,118 / 3,864 |
notebooks |
█████████████████░░░ 84.3% |
6,343 / 7,520 |
agent_platform |
█████████████████░░░ 84.7% |
3,273 / 3,862 |
cohorts |
█████████████████░░░ 86.1% |
4,022 / 4,671 |
actions |
█████████████████░░░ 86.6% |
717 / 828 |
product_tours |
█████████████████░░░ 87.5% |
1,266 / 1,447 |
exports |
██████████████████░░ 88.3% |
6,891 / 7,800 |
conversations |
██████████████████░░ 88.9% |
16,129 / 18,133 |
dashboards |
██████████████████░░ 89.1% |
5,719 / 6,418 |
mcp_analytics |
██████████████████░░ 89.1% |
2,502 / 2,807 |
error_tracking |
██████████████████░░ 89.6% |
9,718 / 10,852 |
alerts |
██████████████████░░ 89.9% |
3,638 / 4,046 |
engineering_analytics |
██████████████████░░ 90.1% |
5,105 / 5,665 |
streamlit_apps |
██████████████████░░ 90.4% |
2,499 / 2,764 |
slack_app |
██████████████████░░ 90.6% |
9,511 / 10,503 |
marketing_analytics |
██████████████████░░ 90.8% |
11,514 / 12,684 |
product_analytics |
██████████████████░░ 91.1% |
5,599 / 6,143 |
data_warehouse |
██████████████████░░ 92.1% |
18,133 / 19,683 |
workflows |
██████████████████░░ 92.4% |
5,148 / 5,574 |
web_analytics |
███████████████████░ 92.7% |
13,624 / 14,691 |
ai_observability |
███████████████████░ 92.8% |
14,868 / 16,019 |
surveys |
███████████████████░ 92.9% |
5,687 / 6,120 |
posthog_ai |
███████████████████░ 93.2% |
1,322 / 1,418 |
approvals |
███████████████████░ 93.3% |
3,395 / 3,640 |
reminders |
███████████████████░ 93.4% |
468 / 501 |
early_access_features |
███████████████████░ 93.8% |
848 / 904 |
endpoints |
███████████████████░ 94.1% |
8,606 / 9,143 |
skills |
███████████████████░ 94.4% |
2,827 / 2,995 |
revenue_analytics |
███████████████████░ 94.5% |
3,598 / 3,809 |
review_hog |
███████████████████░ 94.6% |
6,532 / 6,905 |
logs |
███████████████████░ 95.3% |
9,528 / 9,994 |
experiments |
███████████████████░ 95.6% |
24,171 / 25,288 |
replay_vision |
███████████████████░ 95.7% |
13,354 / 13,952 |
annotations |
███████████████████░ 96.2% |
732 / 761 |
warehouse_sources |
███████████████████░ 96.2% |
230,687 / 239,721 |
feature_flags |
███████████████████░ 96.3% |
16,002 / 16,625 |
user_interviews |
███████████████████░ 96.4% |
2,242 / 2,325 |
data_catalog |
███████████████████░ 97.1% |
2,034 / 2,095 |
customer_analytics |
███████████████████░ 97.2% |
7,480 / 7,698 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 1 · PR risk: 0/10 |
|
Addressed the security-bot finding in 345d2ba: event pagination and resume URLs are now validated against the exact |
|
CI status: every required check passes on |
Implements the Mem0 warehouse source as a ResumableSource pulling three tables: memories (incremental via the v3 filter DSL's server-side created_at/updated_at gte filters), entities, and events (full refresh — no server-side timestamp filters). All HTTP goes through the tracked session. Kept behind unreleasedSource with releaseStatus alpha. Generated-By: PostHog Code Task-Id: 728de8ca-e311-47fc-b79f-75c16df5840b
The events iterator followed the envelope's `next` URL (and any persisted resume URL) with the credentialed session, so a tampered response or poisoned resume state pointing off-origin would leak the API key. Both paths now require the exact https://api.mem0.ai origin before any request is made or state is saved. Flagged by security review bots on the PR. Generated-By: PostHog Code Task-Id: 728de8ca-e311-47fc-b79f-75c16df5840b
Remove unreleasedSource=True so the finished source is visible in the connector catalog, and drop any test asserting the hidden state.
977fcbb to
26fe143
Compare
2 participants
Problem
Mem0 is an AI agent memory platform: it stores extracted facts per user/agent/app/run. Teams building AI products on it want that memory data queryable next to their product analytics. The
mem0source existed only as a scaffolded stub (unreleasedSource=True, no fields, no sync logic).Why
Fill in the scaffolded Mem0 connector end-to-end so Mem0 memory data can be synced into the Data warehouse.
Changes
Implements the Mem0 source end-to-end following the
source.py/settings.py/mem0.pyarchitecture contract:memories(POST/v3/memories/, incremental),entities(GET/v1/entities/, full refresh), andevents(GET/v1/events/, full refresh, opt-in by default since it's a potentially high-volume audit stream).ResumableSource— memories resume by page number (with the incremental cutoff pinned in the resume state so a resumed run paginates the same server-side result set), events resume by the envelope'snextURL. State is saved after each yielded batch.memoriesgetssupports_incremental=True— the v3 filter DSL exposes genuine server-sidecreated_at/updated_atcomparison operators (gte). The filter body rides every page request, so pagination terminates at the watermark naturally.sort_mode="desc"because the endpoint has no sort parameter (undefined ordering), which defers the watermark commit to successful end of run.filtersbody, and a bare{"user_id": "*"}would miss agent/app/run-scoped memories, so the request ORs the wildcard across all four entity ids.requeststhroughmake_tracked_session(),tenacityretries on 429/5xx/transient network errors,get_non_retryable_errorsfor 401/403.canonical_descriptions.pysourced from the official API docs,lists_tables_without_credentials = True(static catalog) so the public docs render the table list, official Mem0 SVG icon,SOURCES.mdmoved to the Implemented table.unreleasedSource=TruewithreleaseStatus=ReleaseStatus.ALPHAfor now.Note
Mem0 offers
memory:add/update/delete/categorizewebhooks, but I deliberately did not wire aWebhookSource: deliveries carry only{event_details: {id, data.memory, event}}(no categories, metadata, timestamps, or entity ids) and Mem0 documents no signing secret. Since webhook mode replaces the poll once active, the memories table would permanently degrade to sparse text-only rows. The server-side incremental filter already makes scheduled pulls cheap; a code comment insource.pyrecords this decision.Verified against the live API where possible without credentials:
/v1/ping/,/v3/memories/,/v1/entities/, and/v1/events/all exist and return structured 401s for bad keys (/v1/ping/is the same probe the officialmem0aiSDK uses for key validation). Filter/pagination semantics come from the current API reference; the incremental cutoff is conservatively sent as a date string (the only format the docs demonstrate), which can only over-fetch — never skip — rows, and the merge dedupes onid.Docs: the companion posthog.com doc is written (attached in a PR comment below) but could not be pushed from this session — the credentials only cover this repo. It needs to land as
contents/docs/cdp/sources/mem0.mdin posthog.com.docsUrlmatches themem0slug andaudit_source_docsrun against a local posthog.com checkout with the doc in place reports no mem0 mismatch.How did you test this code?
pytest products/warehouse_sources/backend/temporal/data_imports/sources/mem0/tests/— 41 tests pass. Transport tests catch: dropping agent/app/run-scoped memories from the filter, saving resume state before yield (skipped rows on crash), replaying one endpoint's cursor against another, recomputing the cutoff on resume (pagination misalignment), retrying auth errors / not retrying rate limits, and forwarding the incremental flag to full-refresh endpoints. Source tests cover schema catalog behavior, credential validation plumbing, resume manager binding, and the credential-free documented-tables path that feeds public docs.pytest .../sources/tests/test_source_categories.py .../sources/tests/test_generated_configs.py— 1550 pass (registry/category/config-generation consistency).ruff check/ruff formatclean;hogli ci:preflight --fixreports 0 failures.Automatic notifications
Docs update
Companion doc for posthog.com (
contents/docs/cdp/sources/mem0.md) is written and attached in a PR comment below — it needs a separate PR in the posthog.com repo (this session's credentials are scoped to this repo only).🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Authored with PostHog Code (Claude). Skills invoked:
/implementing-warehouse-sources,/documenting-warehouse-sources,/writing-tests. Key decisions: chose pull-onlyResumableSourceover the initially suggestedResumableSource+WebhookSourcebecause Mem0's webhook payloads are too sparse to feed the merge path without degrading the table (details in the note above); modeled the operation-event stream as the pulledeventstable instead, which has real ids and timestamps. The generatedMem0SourceConfigcame fromgenerate:source-configs; an unrelated environment-dependent Stripe hunk the generator produced locally was excluded from the commit.Created with PostHog Code