(fix/monitoring) Email Confirmation - #3645
Merged
Merged
Conversation
Member
Author
|
@cubic review |
Contributor
@nickscamara I have started the AI code review. It will take a few minutes to complete. |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
2 issues found across 11 files
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
Contributor
There was a problem hiding this comment.
1 issue found across 5 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Fix all with cubic | Re-trigger cubic
TheophilusChinomona
added a commit
to TheophilusChinomona/firecrawl
that referenced
this pull request
Jun 23, 2026
…11.20) (#34) * test(js-sdk): add unit tests for string API key constructor Generated with AI Co-Authored-By: AI <ai@example.com> * fix(sdk): add deprecated V1 method aliases to V2 Python and JS clients Agents trained on V1 documentation call scrape_url() / scrapeUrl() and hit AttributeError/TypeError, often falling back to generic scraping. This adds 10 deprecated aliases in both Python (sync + async) and JS V2 clients, following the existing pattern used by stop_interactive_browser. Top-level Firecrawl and AsyncFirecrawl classes are also wired. Excluded: crawlUrlAndWatch/batchScrapeUrlsAndWatch (return type changed), extract (argument shape changed). Generated with AI Co-Authored-By: AI <ai@example.com> * fix(sdk): accept both camelCase and snake_case for ChangeTrackingFormat type FormatString accepts both "changeTracking" and "change_tracking". The Literal default should preserve this backward compatibility so callers who already pass type="changeTracking" explicitly do not get a ValidationError. Generated with AI Co-Authored-By: AI <ai@example.com> * fix(sdk): trim whitespace from API key to reject whitespace-only strings Whitespace-only API keys bypass the falsy check and cause confusing downstream 401 errors. Adding .trim() ensures " " is caught at construction time with a clear error message. Generated with AI Co-Authored-By: AI <ai@example.com> * fix(sdk): match V1 parameter name `id` in status/error alias signatures V1 used `id` as the parameter name for check_crawl_status, check_crawl_errors, check_batch_scrape_status, and check_batch_scrape_errors. Using the V1 name ensures keyword calls like check_crawl_status(id="abc") also work correctly. Generated with AI Co-Authored-By: AI <ai@example.com> * fix(python-sdk): accept direct scrape kwargs in crawl() and start_crawl() scrape() and batch_scrape() accept formats=["markdown"] as direct kwargs, but crawl() and start_crawl() require wrapping in ScrapeOptions(...). This inconsistency causes agents to hit TypeError after scrape(formats=[...]) works. Adds the same convenience kwargs to crawl() and start_crawl() in both sync and async clients. When scrape_options is explicitly provided, direct kwargs are silently ignored. Generated with AI Co-Authored-By: AI <ai@example.com> * fix(js-sdk): expose V2 methods on top-level Firecrawl prototype Agents that introspect Object.getPrototypeOf(app) only see ["constructor", "v1"]. V2 methods like scrape, search, crawl live on the parent prototype and are invisible to runtime discovery. Copies V2 method descriptors onto Firecrawl.prototype at module load. Skips constructor and existing own properties (preserves v1 getter). Auto-exposes future V2 methods without maintenance. Generated with AI Co-Authored-By: AI <ai@example.com> * fix(python-sdk): preserve ChangeTrackingFormat options through serialization ChangeTrackingFormat(modes=["git-diff"]) was being serialized to just the string "changeTracking", dropping modes, schema, prompt, and tag. Added explicit handling in both serialization branches to model_dump the full object. Also adds regression tests for JsonFormat and ChangeTrackingFormat construction and serialization. Generated with AI Co-Authored-By: AI <ai@example.com> * fix(sdk): update alias doc comments to indicate agent compatibility purpose Changes wording from "Deprecated v1 alias" to "V1 compatibility alias for agent recovery. Prefer <canonical>." across all 30 aliases in Python sync, Python async, and JS clients. Generated with AI Co-Authored-By: AI <ai@example.com> * fix(python-sdk): clean up _SCRAPE_OPTION_KEYS and add timeout to start_crawl - Remove integration from _SCRAPE_OPTION_KEYS (it is a crawl-level param, not a scrape option). Add clarifying comment. - Add timeout as direct scrape kwarg to sync start_crawl() only. crawl() keeps timeout as poll timeout to avoid collision. - Simplify async extraction now that integration is excluded. - Add tests: start_crawl(timeout=5000) forwards to ScrapeOptions, crawl(timeout=60) stays as poll timeout. Generated with AI Co-Authored-By: AI <ai@example.com> * test(js-sdk): strengthen prototype discoverability binding tests Replace mock-based test with descriptor identity check (copied function === V2 original) and a direct call test that proves this resolves to the Firecrawl instance through the copied descriptor. Generated with AI Co-Authored-By: AI <ai@example.com> * fix(api): wrap monitor webhook data in array Rust webhook-dispatcher declares `data: Vec<serde_json::Value>`, so object payloads from monitor.page and monitor.check.completed were dead-lettered as malformed_json. Wrap the single payload in an array to match the existing crawl/batch shape and the dispatcher's contract. * fix(api): update monitor webhook types to array data shape * fix(sdk): add helpful error when agents access SearchData.data LLM agents consistently hallucinate `.data` on search results (tested in 4 scenarios with Claude Code). The V2 SDK groups results by source type (.web, .news, .images) but agents trained on older API patterns expect a flat `.data` list. Instead of a bare AttributeError, both Python and JS SDKs now raise a message listing available sources and their counts, enabling one-shot self-correction. Generated with AI Co-Authored-By: AI <ai@example.com> * Bump SDK patch versions Co-authored-by: Cursor <cursoragent@cursor.com> * fix(audit): patch uuid and js-cookie vulnerabilities via pnpm overrides (#3589) * fix(audit): patch qs GHSA-Q8MJ-M7CP-5Q26 via pnpm overrides (#3596) * feat(monitoring): add user defined goal with llm validation (#3567) * feat(changeTracking): add optional AI judge ("goal") to classify diffs as meaningful or noise Adds a single new optional field on the changeTracking format — `goal` — that opts a caller into an AI judge. When set AND changeStatus === "changed", the judge classifies the diff as meaningful (matches the goal) or noise (page churn) and attaches the result to `document.changeTracking.judgment`. Backwards compatible: callers that don't set `goal` see no change in response shape or behavior. `changeStatus` is never overridden. - types.ts: new `goal: string (≤2000)` on changeTrackingFormatWithOptions, new optional `judgment` on Document.changeTracking (v1 + v2) - transformers/judgeChange.ts: gemini-2.5-flash-lite via @ai-sdk/google; goal is primary signal, markdown diff is source of truth, json diff augments when present; security guard against page-content injection - transformers/diff.ts: fires judge at end of "changed" branch when goal is set; fails open if judge errors - tests: unit tests (live Gemini, gated on GOOGLE_GENERATIVE_AI_API_KEY) for whitespace/timestamp noise, real price/MacBook signal, markdown-mode diffs; snips covering API contract + first-run no-judgment behavior * feat(monitoring): wire goal + judgment through monitor backend Threads the new changeTracking.goal field from the monitor record into each scrape call, extracts the resulting judgment from the response, and persists it to monitor_check_pages.judgment. Adds a derived monitor.page.meaningful webhook event so subscribers can opt into filtered notifications. Existing monitor.page webhook continues to fire on every page transition regardless of judgment — backwards compatible. - types.ts: add `goal` to createMonitorSchema + MonitorRow, add `judgment` to MonitorCheckPageInsert, register `monitor.page.meaningful` event - runner.ts: inject monitor.goal into the changeTracking format options inside withMonitorScrapeDefaults - results.ts: extractJudgmentFromDoc + derivePageWebhookEvents helpers, persist judgment on insertMonitorCheckPages, fire monitor.page.meaningful when judge classifies the change as meaningful Depends on (separate PRs): - firecrawl-db migration adding monitors.goal + monitor_check_pages.judgment - monitor CREATE/UPDATE endpoints accepting + persisting goal - store.ts SELECT statements adding `goal` to the monitor row column list * refactor(monitoring): move AI judge from scrape pipeline into monitor service The judge is fundamentally a monitoring concern — it filters alerts based on a per-monitor goal. Putting it on the public scrape API surface meant every SDK and consumer of /v2/scrape had to know about goal/judgment for a feature only the monitor service uses. Moving it keeps the scrape API unchanged and lets the judge evolve internally. Reverted (now back to upstream behavior): - controllers/{v1,v2}/types.ts: drop `goal` from changeTracking format options, drop `judgment` from Document.changeTracking - scraper/scrapeURL/transformers/diff.ts: drop the judge call - __tests__/snips/v2/change-tracking-judge.test.ts: deleted Moved + adapted: - services/monitoring/judgeChange.ts (was transformers/judgeChange.ts): takes a plain Logger instead of Meta — no longer coupled to scrapeURL - services/monitoring/judgeChange.test.ts: same 7 live-Gemini tests, imports updated Wired into monitor pipeline: - services/monitoring/diff-orchestrator.ts: accepts goal + extractionPrompt, calls judgeChange when goal is set + status==changed, returns judgment on the result - services/monitoring/runner.ts + results.ts: thread monitor.goal and the changeTracking format's prompt through to the orchestrator, persist resulting judgment to monitor_check_pages New tests: - services/monitoring/page-events.{ts,test.ts} (6 tests): pure-logic derivation of webhook events from status + judgment; backwards-compat guarantee that every event set always contains monitor.page - services/monitoring/diff-orchestrator.test.ts (5 tests): judge gating with mocked judge + GCS — skips when goal absent, skips on new/same, fires on changed, swallows judge errors All 41 tests across the monitoring service pass. * chore(diff): strip leftover blank line in transformers/diff.ts * fix(monitoring): persist goal on create + update (codex P1) Codex review caught: the create/update schemas accept `goal` but store.ts never wrote it. The runner reads monitor.goal to decide whether to call the judge, so without persistence the entire judge path was unreachable for monitors configured via the API. - createMonitor: include `goal: input.goal ?? null` in the INSERT - updateMonitor: patch `goal` when present in input (?? null so explicit clears work) * feat(monitoring): gate emails on judgment + surface meaningful/noise in body When a monitor has `goal` set and ALL changed pages were judged noise (and no new/removed/error activity), the summary email is suppressed. Otherwise the email fires with judgment info shown inline: - Summary line: "Changed: 12 (2 meaningful, 10 noise)" when judge ran - Per-row "[meaningful]" / "[noise]" badge next to status - Per-row reason as muted text below - Meaningful pages float to the top of the list Backwards compat: monitors without `goal` see no behavior change — no gating, no badges, identical email to today. - notification/monitoring_email.ts: gating + body enrichment - notification/monitoring_email.test.ts: 5 new tests covering gating paths - monitoring/runner.ts: thread judgment from PageResult into the email page payload * tighten judge prompt + harden webhook + judge call prompt: sync the v5 system prompt that came out of the desktop eval suite — five-rule structure with explicit hard-noise / goal-override / named-field / default-meaningful / default-noise tiers, diff context awareness, and a single-quoted reason citation format that doesn't trip JSON.parse. eval went from 60% to ~95% on a 184-fixture suite across news, research, finance, ecom, docs, jobs, social, sports, regulatory, weather verticals. parser: add a regex-fallback path when JSON.parse fails so a stray double quote in the reason no longer collapses to "default meaningful" via the outer catch. timeout: wrap the Gemini call in an AbortController with a 15s cap so a hung judge call can't stall the monitor pipeline for a URL. webhooks: add MONITOR_PAGE_MEANINGFUL to the WebhookEvent enum + data map, drop the as-any cast in results.ts, and fan out the two events with Promise.all instead of awaiting them serially. tests: two new live-gemini cases pin the named-field rule. * address codex + auggie review findings - email gating: pass the full filtered page list to sendMonitoringEmailSummary so the noise-vs-meaningful decision sees every changed page. previously the runner sliced to 25 before gating, which could suppress alerts when a meaningful page sat past position 25. the email renderer still caps for display. - goal normalization: trim incoming goals and treat empty/whitespace-only as null. users who leave the field blank no longer get quietly opted into AI judging. - migration safety: only include the goal column in the create-monitor insert when the caller provided one. keeps create working in environments where the firecrawl-db migration hasn't landed yet. - judge retries: wrap the gemini call in a 3-attempt loop with 8s per-attempt timeout and jittered backoff. only transient errors (429, 5xx, timeouts, network blips) retry. previously a single transient failure collapsed to "default meaningful". * split judge_enabled from goal text users want to write a goal as a persistent draft and toggle the AI judge on and off without losing that text. introduce a separate judge_enabled boolean on monitors; the judge only runs when judge_enabled is true and the goal is non-empty. - types: add judgeEnabled to create/update schemas, add judge_enabled to MonitorRow - store: persist judge_enabled on insert (only when caller provided one, matching the goal-column defense) and on patch - runner + results: gate the judge call on judge_enabled && goal so toggling off skips the LLM entirely - email: gate the noise-suppression on judge_enabled (with goal) so monitors with a saved-but-disabled goal still get email-on-any-change - test: extend the email gating fixture to set judge_enabled when goal is set * fix(monitoring): only suppress email when changed-page list is complete The caller of sendMonitoringEmailSummary passes a paginated subset (limit 100). On a check with 101+ changed pages where the first 100 were judged noise, the gate would suppress the email even if an unseen page was meaningful. Compare changedPages.length against check.changed_count and fail open when the list is truncated. Also drop unused export on BlockContext to unblock knip pre-commit. * fix(monitoring): drop fragile error/parse fallbacks in judge - isTransientJudgeError now keys off structured fields only (AbortError/TimeoutError name, Node syscall code whitelist, HTTP 408/425/429 + 5xx). Removes message-substring matching that over-fired on "network" and missed locale variants. - Drop the regex-based JSON parse fallback. If JSON.parse fails we now return the same fail-open default used elsewhere instead of fabricating a judgment from a malformed response. * chore(monitoring): consolidate tests + drop noisy comments - Merge judgeChange.test, diff-orchestrator.test, page-events.test into one meaningful-monitoring.test.ts. Cuts redundant cases (e.g. the derivePageWebhookEvents matrix collapses to one assertion). - Remove restate-the-what comments throughout. Keep only WHY comments (truncation safety, forward-compat insert) and trim those. * fix(monitoring): strict meaningful parse + split judge tests - Replace Boolean(parsed.meaningful) with strict ===true/===false check; default to true (fail open) when the field is missing or malformed. Avoids flipping a string 'false' or 0 into a spurious noise verdict. - Split judgeChange tests back into their own file. The previous consolidation broke the live-Gemini tests because the meaningful-monitoring file mocks ./judgeChange at module load, which would intercept the calls. * Remove transient judge error checks; add monitor fields Remove the isTransientJudgeError helper and associated special-case logic from judgeChange.ts: the judge call now simply retries up to JUDGE_MAX_ATTEMPTS and uses the same backoff/jitter; logging and the fallback reason message were simplified accordingly. Add new optional monitor fields to SDK types so monitors can carry a goal and a flag to enable/disable the judge: in the JS SDK add goal and judgeEnabled to CreateMonitorRequest and UpdateMonitorRequest; in the Python SDK add goal and judge_enabled (alias judgeEnabled) to MonitorCreateRequest and MonitorUpdateRequest. These fields are optional and intended to allow configuring monitor goals and toggling the judge behavior. * Include diff text/json in monitoring payloads Expose computed diffs through the monitoring pipeline: add diffText and diffJson to the MonitorPageDiffResult type, populate them in computeAndPersistPageDiff (when available), and thread them through to sendMonitorPageWebhook. The webhook payload now includes a diff object (text and/or json) so downstream consumers receive the actual change content alongside existing metadata. * feat(monitoring): bill +1 credit per judge invocation - billTeam fires after each successful judge call in recordMonitorScrapeSuccess. Only changed pages on goal+ judge_enabled monitors incur the extra credit; unchanged pages or monitors without a goal cost the same as before. - estimateMonitorCreditsPerRun accepts judgeEnabled; doubles the per-run estimate as the upper bound when on, so the monthly figure stored on monitor rows reflects the true ceiling cost. Wired through create + update paths. - Embed diff text/json into the monitor.page webhook payload so consumers don't need a follow-up API call for the diff. * fix(monitoring): merged credit re-estimate + valid billing metadata - updateMonitor now reads the existing row and merges patch inputs with current state before re-estimating credits. Three regressions this addresses: 1. Goal/judge-only updates were skipped entirely because the recalc was gated on `patch.targets` being present. 2. A targets-only update would drop the existing judge multiplier because judgeEnabled wasn't merged in. 3. Schedule-only updates didn't re-estimate either. Also handles intervalMs falling back to the merged cron. - billTeam now receives a proper BillingMetadata ({endpoint: "monitor", jobId: checkId}) instead of an ad-hoc object cast through `as any`, so endpoint attribution in Autumn stays correct. * fix(monitoring): plug API surface holes for goal + judgment - Zod accepts goal: null on create/update so clearing the goal via PATCH no longer 400s. - serializeMonitor returns goal + judgeEnabled so clients can read back what they wrote. - Check-page list response includes judgment so consumers fetching diff history get the same data as the webhook. - JS + Python SDK Monitor / MonitorCheckPage types gain goal, judgeEnabled, judgment. * feat(monitoring): render the unified diff inline in summary email - buildHtml emits a styled <pre> block per page when diffText is present: green-tinted + lines, red-tinted - lines, purple hunk headers, capped at 24 lines / 200 chars per line with a "… N more lines" footer. - runner now loads up to 5 meaningful-changed diff artifacts from GCS in parallel before calling sendMonitoringEmailSummary and threads the text through. Per-page errors swallowed so a single GCS blip doesn't drop the entire email. * fix(monitoring): include judge multiplier in per-check credit reservation createMonitorCheck was calling estimateMonitorCreditsPerRun without the judgeEnabled flag, so the per-run estimated_credits was the scrape-only cost. Final billing was correct (results.ts bills +1 per judge call) but the Autumn reservation lock under-reserved on judge-heavy monitors near their credit ceiling. Now we pass the same merged judge-on flag the monthly estimate uses on monitor row create/update, keeping the lock and the final bill in sync. * chore(webhook): declare judgment + diff on MonitorPageData The send site (results.ts::sendMonitorPageWebhook) has been including these fields in the payload for both monitor.page and monitor.page.meaningful events, but the public TS interface omitted them and the send call cast through any. Now the interface accurately describes the wire payload: judgment?: { meaningful, confidence, reason, fields } | null diff?: { text?, json? } | null Same shapes as the JS/Python SDKs' MonitorCheckPage types so a consumer typing their webhook handler from this interface gets the full shape. * chore(judge): make 'you see only the diff' framing explicit Adds a short framing paragraph at the top of the system prompt spelling out two things the judge had to infer before: - it's part of a long-running monitor comparing consecutive scrapes - the full page is not available, only the diff + surrounding context ~50 tokens. Doesn't fix any specific failing case in the eval — it anchors the model's mental model upfront so reasons no longer drift into 'the page now shows X' phrasing as if the model had access to the rendered page. * collapse monitor.page.meaningful into monitor.page with isMeaningful flag * auto-enable judge when goal is set * Nick: * NIck: --------- Co-authored-by: Nicolas <20311743+nickscamara@users.noreply.github.com> * fix(api/gcs-jobs): fix GCS storage filename pattern (#3605) * fix(api/gcs-jobs): consolidate GCS upload logic into a common function (#3606) * chore(api): remove autumn vs legacy credit divergence log (#3607) Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: micahstairs <micah@sideguide.dev> * fix(api/gcs-jobs): preserve raw errors from GCS by passing a noop retryableErrorFn * fix(api/zdrcleaner): clear dr_clean_by in batches of 100 * fix: new zdr cleanup method (#3611) * feat(api/zdrcleaner): heartbeat support * fix(api/zdrcleaner): no await on heartbeat * fix(api/zdrcleaner): silence logs from removeJobFromGCS * feat(monitoring): default timeout to 5000 to mitigate false positives on load (#3612) * Update runner.ts * Update runner.ts * fix(api/gcs-jobs): use less calls to GCS upon write * fix(api/monitoring): prevent double webhook send and backfill deliver… (#3618) * fix(api/monitoring): prevent double webhook send and backfill delivery status from webhook_logs * Update apps/api/src/services/monitoring/runner.ts Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> --------- Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> * jitter scheduler claim to avoid thundering herd (#3619) * fix(api/monitoring): jitter scheduler claim to avoid thundering herd * fix(api/monitoring): isolate jitter defer failures per monitor * fix(api/monitoring): fall through to enqueue when defer write fails * fix(api/monitor): accept origin field in create/update body Every other v2 endpoint declares `origin: z.string().optional().prefault("api")` on its request schema, but the monitor schema used `z.strictObject` without it. SDKs auto-inject `origin` into POST bodies (e.g. firecrawl-py 4.28's http_client.py:91), which caused `/v2/monitor` to reject with `Unrecognized key: "origin"`. Adding the field brings the monitor schema in line with the rest of v2 and unblocks both the Python and JS SDKs. * fix(python-sdk/monitor): expose goal and judge_enabled in create/update `MonitorCreateRequest` and `MonitorUpdateRequest` already carry `goal` and `judge_enabled`, but the client wrappers `create_monitor()` and `update_monitor()` didn't accept them as kwargs (sync + async), so callers had to drop to raw HTTP to set either field. Add the kwargs and forward them to the request models. * Add meaningfulChange to monitoring judgments Extend judge output and related types to include a meaningfulChange field containing the full verbatim meaningful diff text. Update the judge prompt/instructions to require meaningfulChange, change JUDGE_MODEL_NAME to "gemini-3-flash-preview", and ensure default/error responses populate meaningfulChange as an empty string. Add the new field to service/type interfaces, webhook types, results handling, and update the test FAKE_JUDGMENT accordingly. * Update judgeChange.ts * Update judgeChange.ts * Add meaningfulChange field and tighten judge text Add a new meaningfulChange string to the Judgment type to carry the full goal-relevant verbatim changed text. Update judgeChange prompt/instructions to (1) include the meaningfulChange output, (2) forbid mentioning system prompts, internal rules, policy names, or rule-number phrases in the reason, and (3) emphasize providing a user-facing rationale only while keeping the existing JSON/quoting guidance. * chore(api): remove AUTUMN_CHECK_ENABLED flag (#3608) Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: micahstairs <micah@sideguide.dev> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * chore(api): rip out AUTUMN_EXPERIMENT and AUTUMN_EXPERIMENT_PERCENT (#3626) Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: micahstairs <micah@sideguide.dev> * fix(api/scrapeURL/fire-engine): new file size exceeds limit error detection * Update judgeChange.ts * Replace meaningfulChange with meaningfulChanges Change the single-string meaningfulChange to a structured meaningfulChanges array of events across monitoring, webhook, notification, and JS SDK types. Add MeaningfulChangeEvent shape (type/before/after), update judgeChange to parse, sanitize and truncate up to 5 meaningful change items (with inference of type and caps), and adjust default/error handling to return an empty array. Update tests and result/notification plumbing to use the new field. * Update judgeChange.ts * Add per-change reason and remove 'moved' type Add a short user-facing "reason" field for each meaningfulChanges entry and remove the special "moved" change type (treat rank/order changes as "changed"). Update type definitions across API, webhook, notification, and JS SDK, and adapt judgeChange logic to coerce/truncate the new reason field (with a new cap). Also refine the rank-movement guidance in judgeChange and update tests to include the new reason property. * feat(api): * fix(api/scrapeURL/pdf): fix size cap enforcement on PDF scraping * fix(api/scrapeURL/pdf): raise PDF download limit to 50MB * fix(api): increase timeout * Update judgeChange.ts * Update judgeChange.ts * Update judgeChange.test.ts * Update judgeChange.ts * Update judgeChange.ts * fix(api/billing): stop falling back to ACUC when Autumn check fails (#3631) Autumn is the source of truth for credits and ACUC accounting has diverged for many teams (e.g. teams showing negative remaining_credits in ACUC while being current in Autumn). The previous code ran both checks in parallel and silently fell back to ACUC whenever Autumn returned null on error, producing spurious 402s. The middleware now consults Autumn only and fails open on transient Autumn failures, matching the behavior of browser.ts and scrape-browser.ts. Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> * Update judgeChange.ts * fix(api/billing): honor Autumn overage allowance for crawl (#3630) When Autumn permits a request via overage, it returns `{ allowed: true, remaining: 0 }`. The middleware previously copied `remaining` straight onto `req.account.remainingCredits`, which the crawl controllers then used to clamp `crawlerOptions.limit` via `Math.min(remainingCredits, limit)` — silently reducing the crawl limit to 0 even though Autumn had said the request was allowed. Treat an Autumn-allowed result as unconstrained from the middleware's perspective, matching the existing `USE_DB_AUTHENTICATION=false` behavior in the crawl controllers. Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: micahstairs <micah@sideguide.dev> * increase monitor judge timeout * stop truncating monitor judge input * remove monitor scrape waitfor * fix(api): ignore queue full errors in Sentry (#3635) * fix(api): ignore queue full errors in sentry * fix(api): parse sentry exception payload values * Update runner.ts * Add meaningful change support to monitors Introduce a MonitorMeaningfulChange type and surface goal-relevant changes on monitor judgments. Update MonitorPageJudgment in Go to include MeaningfulChanges and in Python to include meaningful_changes (with alias mapping and populate_by_name) so JSON payloads with "meaningfulChanges" parse correctly. Add unit tests in both Go and Python to verify parsing of meaningful change entries. * Update types.py * align monitor judge input with shown diff * bump sdk versions * bump python sdk version * bump python sdk to 4.28.2 * support am pm monitor schedules * (fix/monitoring) Email Confirmation (#3645) * Nick: * Nick: * Nick: * Nick: * Nick: * repair monitor crawl finalization * repair monitor crawl finalization * fix monitoring recipient knip exports * repair monitor crawl target runs * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * drop monitor store cleanup * drop monitor runner changes * fix(api): ignore unsupported actions sentry errors (#3652) * fix(api/wikipedia): emit og:image meta tag from Enterprise API article.image.content_url The Wikipedia engine synthesizes its own <head> but did not include og:image, so metadata.ogImage was missing whenever a Wikimedia URL was routed to this engine (~50% of the time per the engine A/B coin flip), even though the real wikipedia.org HTML includes it. The Wikimedia Enterprise API article response already exposes the article's main image at `image.content_url` per the data dictionary (https://enterprise.wikimedia.com/docs/data-dictionary/#image), so we just read it from the existing response and emit og:image — no extra API call needed. * fix(api): handle NuQ AMQP shutdown closes (#3654) * fix(api): sanitize search log queries for Postgres (#3655) * Refactor monitoring credit accounting and metadata Add detailed credit estimation and tracking across monitoring flows. Introduces new credit constants and functions (estimateActualCredits, calculateMonitorCheckActualCredits, calculateMonitorCheckActualCreditsFromPages) and refactors estimateMonitorCreditsPerRun to account for JSON vs base credits, format bonuses, lockdown/proxy options and PDF page credits. recordMonitorScrapeSuccess and diffAndPersistPage now persist contentType, numPages and creditsUsed metadata and propagate per-page credits from runScrape/runCrawl. Removes direct per-judge billing (billTeam import and billing calls). Adds unit tests for runner and store helpers and implements batched DB calculation for actual monitor check credits during reconciliation. * Update runner.ts * Await and use team flags in credit estimation Make estimateActualCredits asynchronous and ensure it resolves team feature flags before billing calculations. Tests updated to await/resolves the promise and to pass the new param shape (doc, options, internalOptions, teamId). estimateActualCredits now retrieves flags from internalOptions.teamFlags or falls back to getACUCTeam(teamId) and passes them into calculateCreditsToBeBilled. runCrawlTarget now includes teamFlags in its internalOptions and awaits estimateActualCredits when accumulating page credits. * Refactor scrape credit calc & billing flow Move scrape credit calculation into the worker and simplify billing: add calculateScrapeJobCredits helper (returns null for explicit monitor scrapes), call it early in billScrapeJob, and streamline autumnService tracking + billing queue enqueue logic. Simplify estimateActualCredits to a lightweight, synchronous fallback (use reported credits or 1) and update monitoring runner to use the new API (remove teamFlags lookup and async credit fetches). Revise per-page credit estimation logic to compute format/proxy/lockdown bonuses incrementally. Expand tests to cover many billing scenarios (formats, PDFs, proxies, ZDR, cost tracking, bypassed jobs, errors and unsupported features). * Update store.ts * Charge judge credits only for persisted judgments Add a helper (judgeCreditsForPage) to encapsulate judge-credit logic and replace the inline ternary. The helper charges JUDGE_CREDITS_PER_PAGE only when a judgment is persisted (non-null), regardless of the judgment's meaningfulness. Also add a unit test to verify credits are counted only for persisted judgments (covers undefined, null, meaningful=false, meaningful=true cases). * Update README.md * Record credits for failed monitor scrapes Pass the billed credits into recordMonitorScrapeFailure and persist them on error records (metadata.creditsUsed). Add a unit test to ensure credit calculations use recorded credits for error pages. This ensures monitoring credit accounting includes failed scrapes. * Record proxy/postprocessor metadata & billing refactor Persist proxyUsed and postprocessorsUsed in monitor scrape records and when diffing pages. Enhance monitor credit calculation to fall back to retained monitor metadata for PDFs, stealth proxy and X/Twitter postprocessor usage (adds X_TWITTER_POSTPROCESSOR_CREDIT_BONUS and MonitorCreditMetadata), and import shouldParsePDF for PDF detection. Refactor scrape-worker billing: remove the older calculateScrapeJobCredits export, use calculateCreditsToBeBilled in billScrapeJob, simplify billing flow and error/refund handling, and update callsite to recordMonitorScrapeFailure. Tests updated: add store tests for metadata-based fallback credits and remove now-obsolete calculateScrapeJobCredits tests from runner.test. * feat(api): add WWW-Authenticate header on auth 401 for agent discovery * refactor: update Firecrawl import paths and package dependencies (#3634) * refactor: update Firecrawl import paths and package dependencies Changed import statements in various files to use the new 'firecrawl' package instead of '@mendable/firecrawl-js'. Updated package.json files to reflect the new dependency. This ensures consistency across the codebase and aligns with the latest SDK version. * refactor: update Firecrawl import statements across multiple files Changed import statements to use named imports from the 'firecrawl' package instead of default imports. This update ensures consistency in the usage of the Firecrawl SDK across various JavaScript and TypeScript files, aligning with the latest SDK conventions. * refactor: standardize Firecrawl import statements in README.md Updated import statements in the README.md file to use named imports from the 'firecrawl' package instead of default imports from '@mendable/firecrawl-js'. This change aligns the documentation with the latest SDK conventions and enhances consistency across the codebase. * chore: remove deprecated '@mendable/firecrawl-js' dependency from pnpm-lock.yaml Eliminated references to the '@mendable/firecrawl-js' package in pnpm-lock.yaml, reflecting the transition to the new 'firecrawl' package. This update helps streamline the dependency management in the project. * re-throw add feature error (#3667) * Add consistentRead option for monitor DB reads Introduce a consistentRead flag across monitoring store functions and use the primary supabase_service for strong-consistency reads when requested. Update runner to request consistentRead: true for critical read paths (recovering runs, listing active pages, counting pages, calculating credits, etc.) so reconciliations and crawls avoid stale read-replica data. Also switch getMonitorPage to use the primary service and update the scheduler test monitor fixture to include schedule_cron and schedule_timezone. * Treat 'enhanced' proxy as premium in billing Count metadata.proxyUsed == 'enhanced' as a premium proxy when calculating monitor check credits. Introduces a usedPremiumProxy flag and updates the condition so enhanced proxies trigger the SCRAPE_OPTION_CREDIT_BONUS (in addition to stealth and the previous fallback). Adds a unit test to verify enhanced proxy metadata results in premium billing while preserving behavior when metadata reports basic. * fix: CVE-2026-41676 security vulnerability (#3664) Automated dependency upgrade by OrbisAI Security * fix(api): allow search for teams with searchZDR forced Previously, search endpoints (v1/v2 /search and /x402/search) rejected any team whose flags resolved searchZDR to "forced", returning a 400. Scrape already supports forced ZDR and cleans data up after the job, so search should behave the same way. Now we propagate the team's forced-ZDR state through the controller into executeSearch and the underlying scrape jobs, so scraped pages are cleaned up just like the scrape endpoint. The job-log entries also reflect the resolved ZDR state instead of being hardcoded to false. * feat: stable sort * feat(api): split searchZDR forced into forced-zdr and forced-anon The search /enterprise array has two distinct modes ("zdr" and "anon") with different pricing and upstream behavior, but the previous searchZDR: "forced" flag conflated them. Split into: - searchZDR: "forced-zdr" -> always-on ZDR enterprise mode - searchZDR: "forced-anon" -> always-on anonymous routing - searchZDR: "forced" -> deprecated alias for forced-zdr getSearchForcedKind(flags) returns "zdr" | "anon" | null so the controller can inject the correct value into req.body.enterprise and let the existing pipeline handle billing and routing. Also fix the issue identified by cubic: zeroDataRetention in the log context and applyZdrScope is now derived from the resolved request state (team-forced OR per-request enterprise:zdr|anon), so "allowed" teams opting in per-request are no longer misclassified as non-ZDR in logs or error capture. * feat(api): Implement fire-privacy (#3525) * feat(api): wire redactPII into v2 scrape via fire-privacy Adds a `pii` format and `redactPII: boolean` option to v2 scrape. When both are set, calls fire-privacy (`/redact`) with `mode: model`, `operator: replace`, `language: en`, and attaches a `pii` block to the document with `status`, `redactedMarkdown`, `spans`, and `truncatedAt`. Original `markdown` is left untouched. Fail-soft: 503 -> service_at_capacity, 5xx/413 -> error, >5s -> timeout. The scrape itself always succeeds; the status surfaces what happened. Boolean-only on purpose so we can grow to `bool | object` later without breaking the surface. Also adds `src/controllers/v0/admin/rotate-api-key.ts` to the knip ignore list — pre-existing orphan from #3520, blocking the pre-commit hook. * feat(api): redactPII options — mode / entities / replaceStyle Boolean form stays the common case. Object form lets callers tune the three things fire-privacy exposes: - mode: "accurate" (default, OPF model only — F1 0.73, precision 0.87, cleanest output) / "aggressive" (model + Presidio, higher recall at precision cost) / "fast" (Presidio-only, no GPU, ~2x throughput). Maps to fire-privacy's internal model / both / heuristics modes inside the client. - entities: optional allowlist over { PERSON, EMAIL, PHONE, LOCATION, FINANCIAL, SECRET }. Spans are filtered server-side by mapping each kind into a unified bucket; unmapped kinds drop under any allowlist. When filtering prunes spans, the redactedMarkdown is re-rendered from the original + remaining spans (otherwise we trust fire-privacy's redacted_text since it used the operator we asked for). - replaceStyle: "tag" (default, <KIND> placeholder) / "mask" (* x span length) / "remove" (drop the chars). Maps to fire-privacy's operator field. The Zod schema is boolean | object with a transform that normalizes true to a defaults object and false/unset to undefined — downstream code keeps the same truthy check. strictObject so typos in the options form get rejected. Tests cover each mode, each replaceStyle, the entities filter (including re-render with mask + remove), unmapped-kind drop, unknown mode rejection, and the strict-object rejection. * feat(api): bill redactPII at +4 (lockdown tier), +4 per extra PDF page redactPII is a peer premium feature alongside lockdown / audio / video / stealth, so it picks up the same flat +4 bonus when set. PDF parses all pass through fire-privacy too, so each additional page picks up another +4 on top of the existing +1 page parse cost. Examples: - HTML page with redactPII: 1 + 4 = 5 credits - 10-page PDF with redactPII: 1 + 9 + 4 + 36 = 50 credits - 10-page PDF without redactPII: 1 + 9 = 10 credits Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(api): chunked redactPII, 250KB ceiling, all-or-nothing failure The fire-privacy /redact endpoint has a 100KB request cap and only sees the first 32K chars of any single call. A 100-page PDF parsed to markdown easily exceeds both, so the client now chunks long inputs against fire-privacy at bounded concurrency. - New fire-privacy-chunker.ts: paragraph -> line -> sentence -> whitespace splitter, 28K chars / 95KB per chunk, leaves headroom below the upstream caps. Validated against the scaling eval harness in fire-privacy (eval/scaling/) — zero span divergence vs single-shot at sizes where both fit. - redactText() now: returns "skipped_too_large" above 250KB (~80 PDF pages, ~10 chunks at the upstream caps), chunks everything else, fans out at concurrency=3 so a single big call stays under 50% of the 6-pod fleet, all-or-nothing on chunk failure (partial redaction is worse than none). - Spans get lifted into source coordinates during merge so callers see offsets into the original markdown, not the chunk. - Added PIIStatus = "skipped_too_large" to the public union. Tests: 25 client cases + 9 chunker cases, all passing. Covers chunk merging, offset correctness, ceiling enforcement, fanout parallelism, and chunk-error propagation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(api): tighten redactPII response shape - Collapse PIIStatus to ok|skipped|failed with a `reason` discriminator (replaces flat 6-value enum, including the multi-word skipped_too_large). - Add `entity` to each span — the unified public bucket — alongside the granular `kind`. Unmapped kinds (ORGANIZATION, DATE_TIME, etc.) keep `kind` but omit `entity`, and drop under any entities allowlist. - Strongly type span `source` as model|heuristics|unknown, classified from fire-privacy's recognizer name. - Make span `score` optional — no more `score: 0` colliding with valid zero-confidence values. - Add per-entity `counts` so callers don't have to reduce over `spans`. - Drop `truncatedAt` — vestigial post-chunking (was already documented as "should be null in practice"). - Mirror types into js-sdk and python-sdk, including the redactPII / `pii` format on ScrapeOptions and the snake_case→camelCase mapping. * fix(api): address cubic review on redactPII - chunkMarkdown: validate maxChars/maxBytes are > 0 (otherwise the tentativeEnd never advances past the cursor and the chunk loop spins forever). Throws RangeError instead. - transformers: only require markdown derivation for the `pii` format when `redactPII` is actually enabled — otherwise performRedactPII bails immediately and the derived markdown is wasted work. * fix(api): drop FIRE_PRIVACY_URL default Make FIRE_PRIVACY_URL optional with no default — deployments without fire-privacy should not silently target an in-cluster hostname that doesn't exist. When the URL is unset and a caller asks for redactPII, short-circuit to a failed/error block with a clear log rather than firing a request to `undefined/redact`. * feat(api): redactPII swaps document.markdown and implies onlyMainContent - After redaction, replace document.markdown with pii.redactedMarkdown. Leaving the raw (PII-laden) version next to the redacted one is a footgun — callers log/forward it without thinking. Fail closed when redactedMarkdown is null (failed / skipped:too_large): no markdown at all rather than silently falling back to raw. - redactPII silently forces onlyMainContent: true (same precedence pattern as lockdown). Boilerplate (nav, footer, "Privacy Policy" links) rarely contains real PII and produces noisy redactions that look like false positives. Smaller surface is also cheaper/faster. * fix(api): address redactPII review feedback --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(security): resolve pnpm audit failures from 2026-06-01 audit (#3689) * fix(security): resolve pnpm audit failures Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * chore(js-sdk): bump firecrawl sdk version Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * Simplify redactPII API (#3691) * feat(sdk): add redactPII option (#3695) * Add redactPII to SDK options * Remove unreleased PII wording from SDK PR * Fix Elixir batch redactPII option * Use test site for redactPII scrape snippet * fix: remove use of roastmywebsite.ai * feat(api): move from Supabase SDK to Drizzle (#3698) * fix(api/log_job): remove log verbosity * fix(api/scrape-worker): wait for logRequest before error insert branch * fix(api/scrapeURL): better contentType tracking * feat(api): content_type in scrapes table * fix(api/log_job/logScrape): do not attempt to insert change tracking records for preview team scrapes * fix(api/db/rpc): coerce api_key_id from ACUC to Number * fix(api/gcs): treat 503 as slowdown for better success rate * fix(api/crawl): disallow broken delay parameters * fix(api/nuq): bad mapping of property names when using rabbitmq * fix(api/v2/crawl-cancel): base it off NuQ not redis * fix(api/db): max connection pool size: 10->20 * fix(api): invalid delay parameter in test * fix(llm): bound tiktoken trimming to prevent event-loop freeze (#3707) * feat: deterministic json format (#3706) * wip * wip * improvements and fixes * feat(billing): adjust credit calculation for deterministicJson format * fix cubic issues * prompt for concurreny askLlm * refactor: simplify error handling in extractDeterministicJson function * fix: make typescript a dep (#3710) * fix: clarify usage of askLlm for structured data (#3711) * feat: enhance extractor generation with previous code context (#3712) * chore: improve extractor system prompt (#3715) * fix: backlog timeout calculation (#3720) * feat: extract post content as markdown (#3722) * feat: extract post content as markdown * fix: escape markdown in post text * fix(credits): surface negative remaining credits for overage (#3724) * fix(credits): surface negative remaining credits for overage Autumn caps balance.remaining at 0 when overage_allowed is false, so the /v1/team/credit-usage and /v2/team/credit-usage endpoints returned 0 for teams that had exceeded their quota — even when the true balance was negative. Derive remaining from granted - usage so overage shows as a signed negative balance. Co-Authored-By: micahstairs <micah@sideguide.dev> * chore: simplify comment Co-Authored-By: micahstairs <micah@sideguide.dev> * chore: simplify test comment Co-Authored-By: micahstairs <micah@sideguide.dev> --------- Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: micahstairs <micah@sideguide.dev> * fix(billing): label unresolvable apiKeyIds as Unknown in historical usage (#3727) Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: micahstairs <micah@sideguide.dev> * Add gstack to the integration enum (#3732) gstack (Garry Tan's Claude Code skill suite) now uses Firecrawl as its native web search + fetch. Registering 'gstack' lets those calls set integration:'gstack' for usage attribution. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(api): bill search balance/usage against SEARCH_CREDITS (#3735) Search credit checks and usage tracking now target a dedicated SEARCH_CREDITS Autumn feature instead of the shared CREDITS feature. - Thread an optional featureId through the Autumn check/lock/track/refund methods, defaulting to CREDITS so scrape/crawl/extract are unchanged. - Derive the feature ID from the billing endpoint (search -> SEARCH_CREDITS) in billTeam and the batch billing tracker/refund. - Pass SEARCH_CREDITS to the search-route credit-check middleware (v1 + v2) and to the search-feedback refund. - Scrapes performed inside a search bill under their own endpoint, so they correctly stay on CREDITS. Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: micahstairs <micah@sideguide.dev> * feat(api): bill search+scrape scrape credits against SEARCH_CREDITS (#3738) When a search request passes scrapeOptions, the scrape sub-jobs already carry billing.endpoint="search", so the batched billing path routes their credits to SEARCH_CREDITS (via featureIdForBillingEndpoint, #3735). But the scrape worker's request-scoped Autumn track (and its failure refund) in billScrapeJob hardcoded CREDITS, so with request-tracking enabled those scrape credits leaked back to CREDITS. Derive the feature id from billing.endpoint and pass it to the request-scoped trackCredits/refundCredits so both billing arms agree: search-initiated scrapes meter against SEARCH_CREDITS, standalone scrapes stay on CREDITS. Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: micahstairs <micah@sideguide.dev> * fix(api/test): update stale v1 delay fixture to respect 60s cap (#3740) The delay parameter was capped at 60s in d5976f582, but the v1 types-validation fixture still used delay: 1000, causing the crawlRequestSchema test to fail. v2 fixtures were already updated. Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: micahstairs <micah@sideguide.dev> * fix(api/search-feedback): consider zdr * fix(api): free monitor usage (#3742) * fix(api): * feat(js-sdk): * fix(js-sdk): sync pnpm-lock overrides with package.json (#3748) The js-sdk and js-sdk/firecrawl package.json files declare pnpm.overrides (security bumps for axios, brace-expansion, rollup, minimatch, etc.) but the committed lockfiles were missing the matching overrides block. This makes `pnpm install --frozen-lockfile` fail with ERR_PNPM_LOCKFILE_CONFIG_MISMATCH in any clean checkout (CI, Docker builds). Regenerate both lockfiles so the overrides block matches package.json. Co-authored-by: micahstairs <micah@sideguide.dev> * feat(workflows): dispatch js sdk publish manually * feat(api/research-proxy): forward team id in Firecrawl-Team-Id header Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(audit): patch shell-quote critical GHSA-w7jw-789q-3m8p via pnpm override (#3747) * feat(api): emit api_surface_first_used PostHog milestone Fire a one-time PostHog event the first time a team makes a request from each surface (playground / sdk / mcp / cli / api / monitor), derived from the existing free-form `origin` field on every request. - New dependency-free PostHog capture (POST to /capture/, env-gated on POSTHOG_API_KEY, fire-and-forget, never throws into the request path). - originToSurface() normalizes the free-form origin into a coarse surface. - Dedup via Redis SET NX keyed on (team_id, surface) — no DB change. Volume is bounded by #teams x #surfaces, independent of request volume. - Hooked into logRequest (fire-and-forget). Caveats documented inline: team-level (no email) so use group analysis or resolve api_key owner for person attribution; eviction can re-fire the milestone — back with a Postgres table if exactly-once is needed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(api): key api_surface_first_used to the api-key owner's person Resolve the API key owner's email (api_keys.owner_id -> users.email) and use it as the PostHog distinct_id so the milestone attributes to the same person the dashboard identifies (enables use as an experiment metric). Falls back to team_id when the email can't be resolved. The lookup runs only on the gated-first event (read replica), so it's cheap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(api): address CodeRabbit review on first-surface tracking - Skip tracking for zero-data-retention requests (don't send their metadata to PostHog). - Bail before the Redis SETNX when POSTHOG_API_KEY is unset, so the dedup marker isn't burned without an event being emitted. - Scope the api-key owner lookup to the team so a mismatched apiKeyId can't attribute the milestone to another team's owner email. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(security): resolve pnpm audit failures (#3754) Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(api): remove pii format surface (#3741) * fix(api): reuse redis client in health checks (#3671) Co-authored-by: Gergő Móricz <mo.geryy@gmail.com> * feat(api): add prometheus integration value (#3760) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(security): patch @grpc/grpc-js CVEs and allowlist joi advisory in test-suite (#3762) * fix(api): shrink index DB pool to stay under pgbouncer max_client_conn (#3765) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(api): stabilize self-hosted CI — v2 cancel owner check + query test gating (#3767) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * feat(api): add generic video discovery results (#3764) * feat(api): add generic video discovery results * fix(api): preserve legacy video fallback * fix(api): keep local engines off video * fix(api): skip generic video discovery for youtube * feat(v2/crawl): expose createdAt, completedAt, and duration on crawl status Customers had no way to determine actual crawl wall-clock time from the API. GET /v2/crawl/{id} now returns: - createdAt (ISO timestamp from StoredCrawl.createdAt) - completedAt (ISO timestamp of the last finished child scrape, only present when the crawl is in a terminal state) - duration (seconds; createdAt to completedAt for terminal crawls, or createdAt to now for in-progress crawls) Adds getLastDoneJobTimestamp helper in crawl-redis.ts that reads the highest score from the existing crawl:{id}:jobs_donez_ordered ZSET. Applies to both /v2/crawl/{id} and /v2/batch/scrape/{id} since both routes share crawlStatusController. OpenAPI schema and a snip test that asserts the new fields are present and consistent are included. Co-Authored-By: micahstairs <micah@sideguide.dev> * fix(v2/crawl): make kickoff-failure duration stable across polls Previously when a crawl failed at kickoff, GET /v2/crawl/{id} returned duration = (now - createdAt), which grew on every subsequent poll even though the crawl was already in a terminal state. A kickoff failure finishes instantly, so return a stable duration of 0 and set completedAt to createdAt. Identified by cubic. Co-Authored-By: micahstairs <micah@sideguide.dev> * feat(api): Dragonfly cache for index URL->id lookups + canonical lookup log (#3770) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * feat(api): negative caching for index URL->id lookups (v2) (#3775) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * feat(api/billing): bill deterministic json at 10 credits on codegen, 3 on cached script (#3750) * chore(api): experimental (WIP) (#3776) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(api): repair build from #3776 (complete flag-gated WIP) #3776 squash-merged before the config/types/wiring changes landed, leaving files referencing config + flags that didn't exist. Adds the missing pieces so the tree builds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(security): upgrade esbuild to 0.28.1 (GHSA-gv7w-rqvm-qjhr, GHSA-g7r4-m6w7-qqqr) (#3777) * feat(nuq): rebuild queue + concurrency limiting on FoundationDB (core engine) (#3758) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * fix(api): prevent optional fdb probes from hanging * fix(api): relax optional fdb worker health check * ci(api): publish semver image tags (#3781) * feat(api): separate NuQ FDB worker (#3779) * chore(api): fix knip complaints (#3778) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(security): resolve pnpm audit failures from 2026-06-13 audit (#3780) Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(api): recover crawl-finish context when FDB sheds member data (#3783) Co-authored-by: Cursor <cursoragent@cursor.com> * chore: updates (#3787) * fix(security): resolve pnpm audit failures from 2026-06-15 audit (#3785) * fix(security): resolve pnpm audit failures Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(security): resolve new pnpm audit failures Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * chore(js-sdk): bump package version Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * feat(api): add generic endpoint feedback (#3693) * feat(api): add generic endpoint feedback * refactor(api): split endpoint feedback controller * refactor(api): simplify feedback recording flow * refactor(api): store endpoint feedback in search feedback * Remove raw feedback payload fields * Cap feedback metadata payload * fix(api): scope feedback ZDR persistence skip * fix(api): apply search feedback policy to generic endpoint * fix(api): lower generic feedback refund cap default * fix(api): share search feedback validation * test(api): migrate test suite from Jest to Vitest (#3789) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(rust-sdk): update client tests for keyless construction (#3788) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(api/security): upgrade AI SDK family to non-vulnerable provider-utils (GHSA-866g-f22w-33x8) (#3786) Resolves GHSA-866g-f22w-33x8 (uncontrolled resource consumption in @ai-sdk/provider-utils, affected range <= 3.0.97) by upgrading the API's direct AI SDK provider packages to their current AI-SDK-v6-compatible majors, which depend on provider-utils 4.x. No cross-major pnpm override is used. - @ai-sdk/anthropic ^2.0.41 -> ^3.0.84 - @ai-sdk/openai 2.0.64 -> 3.0.71 - @ai-sdk/groq ^2.0.28 -> ^3.0.41 - @ai-sdk/google-vertex ^3.0.86 -> ^4.0.145 - @ai-sdk/deepinfra ^1.0.27 -> ^2.0.54 - @ai-sdk/fireworks ^1.0.27 -> ^2.0.56 - @openrouter/ai-sdk-provider ^0.4.5 -> ^2.9.1 - ollama-ai-provider ^1.2.0 -> ollama-ai-provider-v2 ^3.6.0 ollama-ai-provider is stale on provider-utils 2.x; the maintained ollama-ai-provider-v2 fork exposes the same createOllama/baseURL API on provider-utils 4.x, so generic-ai.ts only needed the import swap. The lockfile now resolves @ai-sdk/provider-utils 4.x exclusively, and all providers return specificationVersion v3 models matching the existing ai@6 core. Typecheck and a provider-wiring smoke test pass; the resolved advisory is removed from audit-ci.jsonc and the audit passes. Co-authored-by: mogery <mogery@sideguide.dev> * feat(api/search): reimplement highlights beta on query-highlights modal service Replace the per-sentence /v1/highlight semantic model with the query-finetuned Query Highlights modal service. Pages are sent through a single /batch_highlight call (resident-GPU batch), markdown is passed as-is so the service does the training-matched line-span splitting, and each result's pruned_markdown is mapped back to the web (description) / news (snippet) field. - config: add HIGHLIGHT_MODEL_TOKEN (bearer) alongside HIGHLIGHT_MODEL_URL - highlights: parallel index lookups -> single batch score call -> apply; envReady now also requires the token - add unit tests for the batch client (endpoint, auth, mapping, fallbacks) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(api): fix knip unused-export failures; forbid bypassing knip Un-export internally-only symbols flagged by knip (keyless.ts limits/prefix/ helper/type, posthog.ts originToSurface) so the pre-commit knip check passes cleanly. Add a CLAUDE.md rule to never bypass knip failures with --no-verify. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(api/keyless): optional Spur Context IP reputation check (#3792) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(api/search): use trained span pipeline for highlights beta (sentence parse + assembleAnswer) The first pass sent raw markdown and used the modal service's generic newline fallback (pruned_markdown), which skips the candidate-span generation the model was trained on and drops structure-aware reassembly. Align search highlights with the scrape `highlights` format: parse markdown into the same spans (parseMarkdownToSentences), send them as explicit `lines`, take the returned highlights[].index, and rebuild with assembleAnswer (re-includes table headers, rebuilds fenced code). - extract parseMarkdownToSentences + assembleAnswer into shared lib/highlight-spans.ts; query.ts now imports them (single source of truth) - highlight-model.ts: send `lines`, return selected indices per page - highlights.ts: parse spans -> batch score -> assembleAnswer per hit - tests for the shared span parse/reassembly (tables, code) + the batch client Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(api/search): cap highlights at top_k=12 per page Send top_k=12 on the query-highlights batch request so each page keeps at most the 12 highest-scoring spans after thresholding. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(api/search): neighbor + group-aware budgeting for highlights Replace the service-side raw-line char budget with client-side budgeting over the model's scored spans (highlight-budget.ts): - neighbor policy: each selected line may pull in its ±1 adjacent lines for context, capped at 35% of the budget so context can't crowd out answer lines. - group budget: merge adjacent selected lines into blocks, rank blocks by best span score, emit best blocks in page order until the char budget is reached — so many tiny unrelated lines don't each eat the budget. generateHighlightsBatch now returns scored spans (index+score) and no longer sends max_highlight_chars (budgeting moved client-side). highlights.ts runs selectHighlightIndices before assembleAnswer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(security): pin @opentelemetry/core >=2.8.0 in test-suite (GHSA-8988-4f7v-96qf) (#3791) * fix(security): pin @opentelemetry/core to 2.8.0 to fix GHSA-8988-4f7v-96qf (#3790) * [codex] make parse keyless (#3793) * [codex] Reserve keyless credits atomically (#3796) * fix(api/keyless): restore keyless_credit_usage audit logging after reserve refactor #3796 moved keyless credit accounting from chargeKeylessCredits (Redis counter + keyless_credit_usage insert) to reserveKeylessCredits/adjustKeylessCredits, which only touch Redis. The worker's chargeKeylessCredits call is gated on !keylessReserved, so for the now-default reserved path the audit insert never ran and keyless_credit_usage stopped getting rows. Extract the audit insert into logKeylessCreditUsage() and call it from each controller's reconciliation point with the actual billed credits. The worker fallback still logs via chargeKeylessCredits. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [codex] Add v2 research proxy and SDKs (#3794) * chore(sdks): bump versions to release v2 research support Follow-up to #3794 which added the research methods across all SDKs but did not bump versions, so the publish workflows no-op'd at the version gate. - js: 4.26.0 -> 4.27.0 - python: 4.28.3 -> 4.29.0 - rust: 2.8.1 -> 2.9.0 - go: 1.6.1 -> 1.7.0 - ruby: 1.8.1 -> 1.9.0 - php: 1.6.1 -> 1.7.0 - java: 1.9.1 -> 1.10.0 - dotnet: 1.7.1 -> 1.8.0 - elixir: 1.6.1 -> 1.7.0 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(test-site): upgrade astro to 6.4.7 to fix GHSA-JRPJ-WCV7-9FH9 and GHSA-2PVR-WF23-7PC7 (#3801) * feat(interact): return cdpUrl in the scrape …
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by cubic
Adds opt-in and unsubscribe for monitor email notifications. External recipients must confirm before they get alerts; team members are auto-confirmed.
New Features
/v2/monitor/email/confirmand/v2/monitor/email/unsubscribe(token in body; unauthenticated). Query-string tokens are rejected; JSON errors useinvalid_token(400) andnot_found(404).pending, send confirmation emails for new externals, and returnemailRecipientSubscriptions(includesstatus,source, andconfirmationEmailSent).GET /v2/monitor/:idalso returns current subscriptions.last_notified_atfor delivered recipients.confirmed; partial/missing rows are treated aspending.Migration
emailRecipientSubscriptionstoMonitorinapps/js-sdk/firecrawlandapps/python-sdk/firecrawl.pendinguntil they confirm; team-default delivery (no recipients set) is unchanged.Written for commit 91d9231. Summary will update on new commits.
Review in cubic