feat(data-warehouse): implement fireworks_ai import source - #70717
Conversation
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
![hex-security-app[bot]](https://avatars.githubusercontent.com/in/2665233?s=60&v=4){kind=small}
🤖 CI report✅ Bundle size — no changeUncompressed size of every built Total: 64.77 MiB · no change No file changed by more than 1000 B. Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report ✅ Eager graph — within budgetHow much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy
🟢 Largest files eagerly shipped from
|
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 762 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 281.3 KiB | ../node_modules/.pnpm/posthog-js@1.402.2/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 235.5 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 222.9 KiB | ../node_modules/.pnpm/posthog-js@1.402.2/node_modules/posthog-js/dist/module.js |
| 164.0 KiB | src/queries/validators.js |
| 154.3 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.8 KiB | src/lib/api.ts |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 92.7 KiB | ../packages/quill/packages/quill/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Dist folder size — 🔺 +805 B (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1312.45 MiB · 🔺 +805 B (+0.0%)
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 1 · PR risk: 0/10 |
Implements the Fireworks AI warehouse source as a ResumableSource over the Fireworks control-plane REST API (Google AIP-style, pageToken pagination), syncing models, datasets, deployments, deployed models, fine-tuning jobs, batch inference jobs, evaluation jobs, evaluators, and users. Full refresh only; page-token resume state saved after each yielded page. Includes canonical descriptions, icon, tests, SOURCES.md entry, and the user-facing doc (pending move to posthog.com). Generated-By: PostHog Code Task-Id: b73336b1-98a5-4bac-9551-6acbf0b9f083
…t_id changes Register account_id in connection_host_fields so the update serializer forces re-entering the API key when the target Fireworks account changes, matching the convention used by other account-scoped sources (e.g. callrail). Generated-By: PostHog Code Task-Id: b73336b1-98a5-4bac-9551-6acbf0b9f083
Remove unreleasedSource=True so the finished source is visible in the connector catalog, and drop any test asserting the hidden state.
110da41 to
ba3fa5c
Compare
2 participants
Problem
Fireworks AI is a widely used inference and fine-tuning platform for open models, but its
fireworks_aiwarehouse source was only a scaffolded stub (unreleasedSource=True, no fields, no sync logic). Teams running training and inference on Fireworks can't join that platform data (training runs, dataset inventory, deployment fleet state) with their product data in PostHog.Why: fill in the scaffolded Fireworks AI connector end-to-end so the source can sync real data, following the warehouse-source architecture contract.
Changes
Implements the source per the
implementing-warehouse-sourcesskill, split into the standard three files:settings.py– declarative endpoint catalog for 10 tables:models,datasets,deployments,deployed_models,supervised_fine_tuning_jobs,reinforcement_fine_tuning_jobs,batch_inference_jobs,evaluation_jobs,evaluators,users. Primary key is the AIP resourcename(globally unique full path), partition key is the stablecreateTime(neverupdateTime).fireworks_ai.py– transport over the control-plane REST API (https://api.fireworks.ai/v1/accounts/{account_id}/...), Bearer auth viamake_tracked_session(),pageToken/nextPageTokenpagination (pageSize 200), tenacity retries for 429/5xx, and sanitized HTTP errors (no query strings or response bodies in stored error state). Account ID input is normalized so a pastedaccounts/my-accountprefix doesn't break the path.source.py–ResumableSourcewiring: page-token resume state saved after each yielded page,validate_credentials(invalid-account-id short-circuit, 200/401/403/404 mapping, 403 accepted at source-create per skill guidance),get_non_retryable_errorsfor 401/403/404, staticget_schemaswithlists_tables_without_credentials = True.Supporting pieces:
canonical_descriptions.pyfor all 10 tables, sourced from the official API reference.frontend/public/services/fireworks_ai.svg(official brand SVG from fireworks.ai).FireworksAISourceConfigregenerated (api_key,account_id).SOURCES.md: movedfireworks_aifrom Scaffolded to Implemented (HTTP / requests / tracked ✅).documenting-warehouse-sourcestemplate, kept atproducts/.../fireworks_ai/docs/fireworks-ai.mdxbecause no posthog.com checkout was available in this environment – it needs to move toposthog.com/contents/docs/cdp/sources/fireworks-ai.mdx(thedocsUrlalready points there).Deliberate scope decisions:
Note
filterparam but doesn't enumerate filterable fields, and server-side timestamp filtering couldn't be verified without live credentials, so no table advertises incremental sync (noted in code comments).apiKeysexcluded. The collection is nested per user and its schema carries key material – it must not land in a warehouse table.unreleasedSource=TruewithreleaseStatus=ALPHAuntil it's validated against a live account.No Django migration needed – the
FireworksAIenum value and its migration already landed with the scaffold;makemigrations --checkis clean.How did you test this code?
pytest products/warehouse_sources/.../fireworks_ai/tests/– 57 tests pass.test_fireworks_ai.pycovers the regressions no existing test catches: paginator dropping or looping the page token, resume state saved before yield (data loss on crash), pastedaccounts/prefix doubling the URL path, sanitized errors leaking pagination tokens, and unstable partition keys.test_fireworks_ai_source.pycovers credential status-code mapping (incl. account-not-found), non-retryable error matching, and pipeline plumbing.pytest .../sources/tests/(1551 cross-source guard tests incl. source categories) – pass.hogli ci:preflight --fix– 0 failures.nextPageToken, camelCase collection keys), and auth behavior verified against the Fireworks API reference plus unauthenticated probes of the live API (all list endpoints return 401 without a valid key). I could not run a live sync – no Fireworks credentials were available in this environment.Automatic notifications
Docs update
Doc added in-repo at
products/warehouse_sources/backend/temporal/data_imports/sources/fireworks_ai/docs/fireworks-ai.mdx; needs to land in the posthog.com repo (see Changes).🤖 Agent context
Autonomy: Fully autonomous
Authored with Claude Code (PostHog Code cloud task). Skills invoked:
implementing-warehouse-sources,documenting-warehouse-sources,writing-tests. Modeled on thetogether_aiandanthropicsources (closest analogs: Bearer-auth AI platform APIs; anthropic for the resumable page-token pattern). Chose a hand-rolled tracked-session transport overrest_source.RESTClientbecause the AIP page-token flow is a dozen lines and keeps resume checkpointing explicit. Rangenerate_source_configsand kept only the FireworksAI hunk – the generator also emitted unrelated env-dependent drift (Stripe config, formatting) that was reverted to keep the diff minimal.Created with PostHog Code