feat(data-warehouse): implement mistral_ai import source - #70036
Conversation
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
![hex-security-app[bot]](https://avatars.githubusercontent.com/in/2665233?s=60&v=4){kind=small}
|
Reviews (1): Last reviewed commit: "feat(data-warehouse): implement mistral_..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
🤖 CI report
|
| File | Size | Δ vs base |
|---|---|---|
posthog-app/src/scenes/inbox/InboxScene.js |
649.3 KiB | 🔺 +13.8 KiB (+2.2%) |
render-query/src/render-query/render-query.js |
21.20 MiB | 🟢 -9.9 KiB (-0.0%) |
Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report
✅ Eager graph — within budget
How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.
| Root | Eager (shipped) | Δ vs base | Budget |
|---|---|---|---|
entry (logged-out pages, app bootstrap)src/index.tsx |
1.21 MiB · 22 files | no change | ███░░░░░░░ 28.1% of 4.29 MiB |
authenticated shell (every logged-in page)src/scenes/AuthenticatedShell.tsx |
8.10 MiB · 2,972 files | 🔺 +313 B (+0.0%) | █████████░ 87.5% of 9.25 MiB |
🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
Largest files eagerly shipped from src/index.tsx
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 668 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 278.6 KiB | ../node_modules/.pnpm/posthog-js@1.399.2/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 234.9 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 221.5 KiB | ../node_modules/.pnpm/posthog-js@1.399.2/node_modules/posthog-js/dist/module.js |
| 164.0 KiB | src/queries/validators.js |
| 154.3 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.9 KiB | src/lib/api.ts |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 90.6 KiB | ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Dist folder size — 🟢 -320.7 KiB (-0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1279.92 MiB · 🟢 -320.7 KiB (-0.0%)
⚠️ Backend coverage — 98.0% of changed backend lines covered — 7 uncovered
🧪 Backend test coverage
Patch coverage — changed backend lines (products + core): ████████████████████ 98.0% (358 / 365)
| File | Patch | Uncovered changed lines |
|---|---|---|
products/warehouse_sources/backend/temporal/data_imports/sources/mistral_ai/mistral_ai.py |
93.4% | 189, 229–230, 235–237 |
products/warehouse_sources/backend/temporal/data_imports/sources/mistral_ai/settings.py |
96.6% | 46 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29252103782 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
warehouse_sources |
███████████████████░ 96.0% |
202,314 / 210,745 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
Fills in the scaffolded Mistral AI source: a resumable REST source syncing models, files, fine-tuning jobs, batch jobs, and the beta agents/conversations/libraries tables. Fine-tuning and batch jobs sync incrementally via the API's server-side created_after filter; everything else is full refresh (no creation-time filter in the API). All HTTP goes through the tracked session. Ships behind unreleasedSource with releaseStatus alpha. Generated-By: PostHog Code Task-Id: 01e5fe68-4ffb-4d5b-8e69-b453863b96b1
Address review feedback and fix CI: - Redact the API key in tracked HTTP telemetry (pass redact_values so a sampled Bearer request can't persist the raw credential). - Fail loudly on an unexpected 2xx response shape instead of returning [] (which get_rows treats as end-of-pagination, silently finishing a sync with missing rows). - Accept a bare JSON array for any endpoint, so a wrapped-vs-bare mismatch on models/libraries still syncs rows rather than dropping them. - Persist the incremental watermark with "desc" semantics for fine_tuning_jobs (no sort param, so page order isn't guaranteed) while keeping "asc" for batch_jobs, which forces ascending order via order_by=created. Also fixes pre-existing mypy errors in the tests and reformats the source doc so the oxfmt markdown check passes. Generated-By: PostHog Code Task-Id: a4e42a28-c388-4c18-8ff9-e07a74073f35
e81592b to
b602ca2
Compare
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
estefaniarabadan
left a comment
estefaniarabadan
left a comment
There was a problem hiding this comment.
Isolated new data-warehouse source scaffold — self-contained under its own source directory with only the expected shared-registry additions (SOURCES.md, generated_configs.py, icon), tests included, CI green. Reviewed as part of a batch. LGTM 👍
|
Rebased onto latest master to clear the merge conflict (the only conflict was an additive one in CI note: one unrelated flaky test tripped on the first run — 🦉 via talyn.dev |
bool is an int subclass, so a stray boolean watermark would be read as a Unix timestamp instead of failing. Guard it explicitly and raise rather than emit an invalid created_after value. Generated-By: PostHog Code Task-Id: 1ca40e68-7cfe-497c-a95b-ca363228555d
2 participants
Problem
Mistral AI is a widely used LLM platform, and many PostHog customers hold API keys for it. Beyond
inference, its API exposes queryable asset and job history (fine-tuning jobs, batch jobs, files, models,
and beta agents/conversations/libraries) worth pulling into the Data warehouse to analyze alongside
product data. The source was scaffolded as a stub; this fills it in end to end.
Changes
Implements the
mistral_aiData warehouse source following theimplementing-warehouse-sourcesskill,with the standard
source.py/settings.py/mistral_ai.pysplit.ResumableSourcewith offset-page pagination. Resume state is a page cursor savedafter each yielded page.
models(unpaginated),files,fine_tuning_jobs,batch_jobs, and the betaagents,conversations,libraries(off by default). Response shapes, params, and field types weretaken from Mistral's published OpenAPI spec.
fine_tuning_jobsandbatch_jobs, which expose a genuine server-sidecreated_afterdate-time filter.batch_jobsalso requestsorder_by=createdto force ascendingorder for a monotonic watermark. Everything else ships full refresh (the API has no creation-time
filter for those), partitioned on the stable
created/created_atfield.make_tracked_session(). Bearer API-key auth.tenacityretrieson 429/5xx/transient; 401/403 are terminal via
get_non_retryable_errors.canonical_descriptions.pyfrom the API docs,lists_tables_without_credentials = Trueso thepublic docs render the table catalog, icon at
frontend/public/services/mistral_ai.svg, and theuser-facing doc.
Note
Kept behind
unreleasedSource=TruewithreleaseStatus=alphaas requested, so it is not yet visibleto users.
Caveats worth a reviewer's eye
the OpenAPI spec and an unauthenticated probe (confirms base URL + auth), not a live smoke test. The
incremental
created_afterformat (ISO 8601) and its filtering behavior come from the spec.fine_tuning_jobsordering. That endpoint exposes no sort parameter, so arrival order is notguaranteed. The dataset is small enough to sync in a single batch, keeping the max-
created_atwatermark correct regardless of order; noted in a code comment.
mistral-ai.mdlives beside the source only because this PR has no posthog.comcheckout. It needs to move to
posthog.comatcontents/docs/cdp/sources/mistral-ai.md; I could notrun
audit_source_docshere for the same reason.How did you test this code?
Automated tests only (I am an agent and did not run a live sync against Mistral):
tests/test_mistral_ai.py(transport): watermark formatting (unix int to ISO Z), incremental vsfull-refresh param building, response-shape extraction (wrapped
datavs bare array), pagination(empty-page termination, save-after-yield, resume-from-page), retry classification (429/5xx retried,
401/403 not), and SourceResponse partition/keys.
tests/test_mistral_ai_source.py(source class): schemas advertise incremental only where the serverfilter exists, beta tables off by default, credential validation mapping, non-retryable errors,
documented-tables rendering, and pipeline plumbing.
Ran locally: the 56 new tests pass, plus
test_source_categories.pyand the source-config generatorsnapshot suite (1521 passed).
ruff checkandruff formatare clean on the changed files. Each testgroup guards a concrete regression (dropping the server filter silently reverting to full refresh, a
malformed
created_afterwedging the sync, pagination looping or skipping, a credential error spinningthe retry budget).
Automatic notifications
Docs update
Added
mistral-ai.md(needs to land in the posthog.com repo, see caveat above).🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Authored by Claude (Claude Code). Skills invoked:
/implementing-warehouse-sources,/documenting-warehouse-sources, and/writing-tests.Key decisions: I chose a custom
requests-based transport (like the klaviyo source) over the declarativerest_source.RESTClientpath because Mistral mixes an unpaginated endpoint, offset-page endpoints, andtwo response shapes (wrapped
datavs bare array), which are cleaner to handle and test with explicitpagination. I pulled endpoint params, response shapes, and field types from Mistral's OpenAPI spec rather
than trusting the initial research notes. On incremental sync I was deliberately conservative: only the
two endpoints with a documented server-side
created_afterfilter enable it, and I leaned on the task'sexplicit "ship full refresh when you can't curl-verify" guidance for the rest. The icon came from
PostHog's own Cloudinary asset already referenced elsewhere in the repo, so no Logo.dev key was needed.
Created with PostHog Code