feat(data-warehouse): implement firecrawl import source - #70034
Conversation
Fill in the scaffolded Firecrawl source with working sync logic for its account-level operational data: team activity, credit/token usage history, active crawls, monitors, and monitor checks. All endpoints are full refresh (Firecrawl exposes no server-side timestamp filter). ResumableSource keyed on the pagination cursor/offset so Temporal can resume after heartbeat timeouts. Outbound HTTP routes through make_tracked_session(). Generated-By: PostHog Code Task-Id: 0aea4ecb-e898-45af-a21a-f1bd872fa2f4
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
![hex-security-app[bot]](https://avatars.githubusercontent.com/in/2665233?s=60&v=4){kind=small}
|
Reviews (1): Last reviewed commit: "feat(data-warehouse): implement firecraw..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
🤖 CI report✅ Bundle size — no changeUncompressed size of every built Total: 70.02 MiB · no change No file changed by more than 1000 B. Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report ✅ Eager graph — within budgetHow much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy
🟢 Largest files eagerly shipped from
|
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 668 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 277.3 KiB | ../node_modules/.pnpm/posthog-js@1.399.1/node_modules/posthog-js/dist/rrweb.js |
| 266.9 KiB | ../node_modules/.pnpm/@posthog+icons@0.37.4_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 224.6 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 221.5 KiB | ../node_modules/.pnpm/posthog-js@1.399.1/node_modules/posthog-js/dist/module.js |
| 164.0 KiB | src/queries/validators.js |
| 154.2 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.9 KiB | src/lib/api.ts |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 90.6 KiB | ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
⚠️ Dist folder size — 🔺 +1.5 KiB (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1438.40 MiB · 🔺 +1.5 KiB (+0.0%)
⚠️ Playwright — 1 flaky
🎭 Playwright report · View test results →
- Retention calculations, period, breakdown, and chart (chromium)
These issues are not necessarily caused by your changes.
Annoyed by this section? Help fix flakies and failures and it will go green!
- Redact the Firecrawl bearer token in tracked sessions (validate + get_rows) so a reflected credential can't land in HTTP logs/samples. - Fail fast on schema drift: index the row selector and monitor id instead of silently returning empty pages, which on full refresh would replace warehouse data with zero rows. - Fix mypy errors in the new tests (HTTPError response arg, union field narrowing, cursor page dict typing). - Run oxfmt on the source doc. Generated-By: PostHog Code Task-Id: f9e02e48-48ba-4432-95cb-aee420042e6b
Generated-By: PostHog Code Task-Id: f9e02e48-48ba-4432-95cb-aee420042e6b
estefaniarabadan
left a comment
estefaniarabadan
left a comment
There was a problem hiding this comment.
Isolated new data-warehouse source scaffold — self-contained under its own source directory with only the expected shared-registry additions (SOURCES.md, generated_configs.py, icon), tests included, CI green. Reviewed as part of a batch. LGTM 👍
2 participants
Problem
Firecrawl was a scaffolded (stubbed) data warehouse source. Firecrawl is a web scraping and crawling API, and beyond its scrape/crawl action endpoints it exposes account-level operational data worth syncing: job activity, credit and token spend over time, in-flight crawls, and change-detection monitors. This lets users track their Firecrawl usage and spend in PostHog alongside the rest of their data.
Changes
Filled in the scaffolded source end to end (the enum, schema, and generated-config wiring already landed on master):
team_activity,credit_usage_historical,token_usage_historical,active_crawls,monitors, andmonitor_checks(a fan-out over monitors, off by default).ResumableSourcekeyed on the pagination cursor (activity) or offset (monitors/checks) so Temporal can resume after a heartbeat timeout without restarting a paginated pull from scratch.settings.py(declarative endpoint catalog + primary/partition keys),firecrawl.py(auth, tracked-transport client, cursor/offset paginators, fan-out,SourceResponse),source.py(registration, schema list, credential validation, resumable wiring). Pluscanonical_descriptions.pysourced from the public API docs.make_tracked_session(); retries viatenacityon 429/5xx and transient network errors.frontend/public/services/firecrawl.svg.unreleasedSource=TruewithreleaseStatus="alpha", and moved the row into the Implemented table inSOURCES.md.Note
Full refresh only, by design. None of Firecrawl's account endpoints accept a server-side "updated since" filter, so marking any table incremental would be a false promise (every sync would still page the whole endpoint).
team_activityis a rolling log Firecrawl retains for only the last 24 hours, so it can't be backfilled - the doc tells users to sync frequently to accumulate history.Warning
I could not curl-verify behavior against the live API (no Firecrawl credentials in this environment). Endpoint shapes, pagination, and the absence of timestamp filters were taken from the current public v2 API reference. Two conservative choices flagged in code comments: offset pagination terminates on a short page (the list endpoints return no
has_more/total), and the credit/token-usage primary key isstartDatebecause we always request the team-level total (nobyApiKey).The user-facing posthog.com doc is included in this PR at
products/warehouse_sources/backend/temporal/data_imports/sources/firecrawl/docs/firecrawl.mdsince no posthog.com checkout was available here. It needs to land in the posthog.com repo atcontents/docs/cdp/sources/firecrawl.md(its slug matchesdocsUrl);audit_source_docsshould be run there.How did you test this code?
Automated tests I (Claude) actually ran, all passing:
tests/test_firecrawl.py(transport): retry classification (429/5xx retry, 4xx immediate, transient network reraise after cap), credential-probe status mapping, cursor pagination + resume, offset pagination termination, unpaginated fetch, monitor-checks fan-out + resume, and per-endpointSourceResponseprimary key / partitioning.tests/test_firecrawl_source.py(source class): schema list is full-refresh-only,monitor_checksoff by default, secretapi_keyfield, credential validation mapping, non-retryable error matching, and resumable-manager/source_for_pipelinewiring.test_source_categories.py(Firecrawl has a category) and the source-config generator.ruff checkandruff formatclean on all changed Python.I did not run a live end-to-end sync (no credentials).
Automatic notifications
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Authored by Claude (Claude Code). Invoked the
/implementing-warehouse-sources,/documenting-warehouse-sources, and/writing-testsskills.Design decisions: chose
ResumableSource(recommended for cursor/offset APIs) but shipped every table as full refresh once the API research confirmed no server-side timestamp filter exists - so no table advertises incremental/append. Kept endpoint metadata declarative insettings.pywith a generic cursor/offset paginator infirecrawl.pyrather than per-endpoint branches.monitor_checksfans out one request per monitor and is disabled by default to avoid surprising API cost. The generatedFirecrawlSourceConfigwas regenerated with the config generator; only the Firecrawl class change was kept (the generator surfaced unrelated pre-existing drift in other sources, which I reverted to keep the PR focused).Created with PostHog Code