Repository navigation
Error Codes scraping
Part of Error codes: every code @ultimat3/scraping registers. x errors explain <CODE> --json prints the same row.
| Code | Means | Typical cause | Fix |
|---|---|---|---|
X_CDP_BROWSER_MISSING |
no Chrome or Chromium is installed for the e2e driver to launch | the driver launches a real browser over raw CDP and carries no dependency to do it, so what it needs is a binary on disk — CHROME_PATH first, then the four /usr/bin names GitHub-hosted ubuntu-latest ships. Reached only by a caller that has already decided a browser is required: openE2eBrowserIfAvailable() answers undefined instead, so a machine with no browser SKIPS rather than failing |
export CHROME_PATH=/usr/bin/google-chrome — or leave it unset and let the browser-backed suite skip |
X_CDP_CALL_FAILED |
the browser refused a DevTools call | the call reached the browser and came back an error frame, or the connection died under it — every in-flight call is settled on close, so a dead browser reports this rather than one deadline per call | print what the page had: await page.evaluate(() => document.body.innerHTML) — a call that refuses the same way means the browser itself is gone, so read the launch above it |
X_CDP_LAUNCH_FAILED |
the browser started and never announced a DevTools endpoint | Chrome is driven over --remote-debugging-pipe, so its first answer IS the announcement. It gets the launch deadline (60 s, or timeoutMs when that is larger) and, unanswered, is reaped and started ONCE more on a fresh profile. The cause carries each launch: whether it exited (with its code) or was still silent at the deadline, and its last stderr lines — D-Bus lines alone are start-up noise, so a deadline beside them is a machine too slow, an exit code a crash, a rejected flag or a sandbox refusal |
/usr/bin/google-chrome --headless=new --no-sandbox --disable-dev-shm-usage --disable-gpu --remote-debugging-port=0 about:blank — the binary the cause names; a healthy one prints DevTools listening on ws://…
|
X_CDP_TIMEOUT |
a DevTools call did not answer inside its deadline | its own code rather than X_TIMEOUT, because the actionable half is WHICH call: a hung Page.navigate is an app that never finishes responding, a hung Runtime.evaluate is an expression that never settles. A call the connection timed carries what the wire did meanwhile — meta.reading is target-gone (its session detached, or its target was destroyed or crashed), lost-in-transport (meta.framesDropped frames arrived unparseable after the call) or no-answer (the target was there and silent; meta.framesArrived, meta.lastFrames, meta.navigations, meta.transportOpen say what else happened) |
no-answer: raise timeoutMs on installE2eDriver({ timeoutMs }), or find the request the page is still waiting on. The other two: x test e2e --json again — the cause names what went |
X_SCRAPE_AUTH_FAILED |
the credentials were rejected |
TERMINAL, always, and the retry table cannot be talked out of it. It is in NEVER_RETRIED, so no retry policy, no backoff and no recovery hook can reach it. A site that locks an account after three wrong attempts turns a retrying framework into the thing that destroys the user's account |
correct the credential in .env.local for the name listed in secrets: on the scrape() definition — this run will not be retried, deliberately: a second wrong attempt is how an account gets locked |
X_SCRAPE_BLOCKED |
the site refused this client — the identity is spent | retryable, and it burns the session first. Retrying a block on the same persisted identity re-trips it every time: the flagged cookies are the thing being refused, so the retry has to arrive as somebody else or it is arithmetic, not a retry | the session is burned and the next attempt starts a new identity — lower rate: on the scrape() definition if this repeats |
X_SCRAPE_BODY_TOO_LARGE |
the HTTP leg's response body ran past maxBytes
|
terminal. A response's size is a property of the endpoint, so attempt 2 reads the same bytes and dies identically. The default is 32 MiB (DEFAULT_HTTP_MAX_BYTES) — generous for the JSON endpoint behind a paginated page, far under what OOM-kills a worker. Before this the leg did await response.text(), which materialises the whole body: AbortSignal.timeout bounds TIME, not bytes, so a 30s stream at 50 MB/s was a 1.5 GB allocation and the run persisted nothing |
raise it for the one call — http.request(url, { maxBytes: 64 * 1024 * 1024 }) — or narrow the endpoint so it answers a page rather than a corpus. Never raise it globally: the cap is what stops one hostile origin taking the worker down |
X_SCRAPE_BROWSER_MISSING |
there is no browser on this machine for the driver to launch |
Terminal, where its sibling X_SCRAPE_BROWSER_UNREACHABLE is retryable, and the split is the whole point. A launch that fails because there is no browser binary — or because no executablePath was configured, which is every puppeteer-core with nothing passed — is a misconfiguration: attempt 2 spawns the same absent path, five times on a five-attempt policy, and on an authenticated target that is five arrivals at a login for no chance of a different answer. Raised by browserUnreachable() when the launcher's own words say the browser is absent (must be specified, ENOENT, EACCES, Could not find Chrome); everything else stays the retryable transport code |
install a Chrome or Chromium and name it — localBrowser({ launcher, executablePath: env.CHROME_PATH }) — or attach to one somebody else is running with remoteBrowser({ cdpUrl: env.SCRAPE_CDP_URL }) |
X_SCRAPE_BROWSER_UNREACHABLE |
the browser went away mid-run | retryable. The browser process or host died, or stopped answering, after the attach succeeded | raise watchdog: { idleMs } on the scrape() definition if the site is genuinely slow, otherwise re-run once the browser host is back |
X_SCRAPE_CAPTURE_INVALID |
the capture names a framing no picture can be taken with |
terminal — the rectangle is the caller's own literal, so attempt 2 passes the identical one, and it is refused before any driver runs, so there is no half-done capture to resume. Three shapes: fullPage beside a clip (CDP silently ignores one, so the caller gets a picture they did not ask for), a rectangle with no area or a non-finite number, and a clip on page.pdf(), which has no crop and would answer a whole-page PDF to a caller who asked for one component. A rectangle merely BELOW THE FOLD is accepted — this package sets and reads no viewport, and refusing it would reject the component-screenshot case it exists for |
drop fullPage or drop clip; measure the element with page.query('<selector>') and pass a rectangle with a positive width and height; a PDF has no crop — use page.screenshot({ clip })
|
X_SCRAPE_CDP_ATTACH_FAILED |
the CDP endpoint refused the attach |
retryable. A browser host still starting, a wrong cdpUrl, or an endpoint that is up and not speaking CDP. Also raised when a CdpResolver on remoteBrowser({ cdpUrl }) could not rent a browser. The endpoint is named by scheme and host only — meta.cdpUrl is wss://host:port — because a connect URL carries a token; the underlying throw is rendered through core's renderThrowable with URL parts stripped |
curl "$CDP_URL/json/version" to confirm the endpoint answers, then pass that webSocketDebuggerUrl to remoteBrowser({ cdpUrl })
|
X_SCRAPE_DOWNLOAD_TIMEOUT |
the download never landed | retryable. A slow file, or a trigger that is not the element that starts the download | raise the timeout on page.download({ timeout }), or confirm the trigger is the element that starts the download |
X_SCRAPE_DRIVER_UNKNOWN |
no browser driver is installed for this run |
terminal. No scrapeDriver in apps/<app>/runtime.ts and the definition passed no driver:; cause lists what is installed, or none
|
apps/<app>/runtime.ts: export const runtime = { scrapeDriver: localBrowser() } — or pass driver: fakeBrowser() on the scrape() definition |
X_SCRAPE_EGRESS_IN_PAYLOAD |
the run exit carries a credential that is also in the job payload | scrape({ egress }) answered a proxy URL whose password is readable out of the run's own input, so it rode the queue payload and is stored in x_jobs in the clear. Refused before the browser opens; terminal, because every remaining attempt reads the same row | look the exit up by id inside the worker — egress: async ({ connectionId }) => (await repo.connectionById(connectionId))?.exit — and drop the exit field from the scrape input |
X_SCRAPE_EGRESS_UNSUPPORTED |
this driver cannot give the session the exit the run asked for |
terminal. egress: named an exit for this run and the driver cannot dial it: remoteBrowser({ cdpUrl }) with a fixed URL attaches to a browser already bound to another exit, or the launcher has no page.authenticate() for an exit that carries credentials. The cause names scheme and host only, never the credentials |
pass remoteBrowser({ cdpUrl: resolver }) so each run rents a browser on its own exit, or drop egress: from the scrape() definition and set the exit once on the driver — remoteBrowser({ proxy }) |
X_SCRAPE_FIXTURE_MISSING |
the fixture driver has no recording for this request | terminal. An offline driver never reaches the network, so an unrecorded URL has no answer to give | add <fixtureDir>/<the recording file> holding { "url": "…", "html": "…" }, or point fixtureBrowser() at the directory that already holds it |
X_SCRAPE_FIXTURE_STALE |
the recording is older than the fixture max age |
terminal. A test passing against a year-old copy of a page proves the test still parses the copy, not the site. A recordedAt that is not a date is stale too: its age cannot be shown to be inside maxAge
|
re-record the fixture directory, or raise maxAge on fixtureBrowser({ maxAge }) with the reason an old recording still proves something |
X_SCRAPE_FRAME_OUT_OF_PROCESS |
the frame runs in another renderer process the page session cannot read | a cross-site iframe under site isolation renders in its own process, so the page session cannot read its accessibility tree |
localBrowser({ launcher, options: { args: ['--disable-site-isolation-trials'] } }) — keeps a cross-site frame in the page's process; or read it with page.query() |
X_SCRAPE_HOST_BLOCKED |
the page asked for a host allowHosts does not list |
terminal. A third-party asset, an analytics beacon or a redirect off the allow-list. The allow-list is default deny, so silence blocks. Also raised on the HTTP leg when a host only a wildcard ('*', '*.example.com') admitted resolves to a loopback, private or link-local address — the cause names the address class, never the address |
add the host to allowHosts on the scrape() definition — cause names both the requested URL and the current list; for an inward host, name it exactly |
X_SCRAPE_HTTP_FAILED |
the site answered the HTTP leg with a non-2xx status |
retryable by default — 429, 5xx and a deploy window are transient far more often than not. A 4xx that isRetryableStatus (@ultimat3/core) calls permanent is thrown with a per-instance terminal override, because the same code is both and only the throw site knows which it saw — 408, 409, 425 and 429 keep their attempts As of 2026-08-23. This row said "below 500 other than 429" until then, and a private copy of the table here called 408 and 425 terminal while the rest of the framework called them retryable; a terminal classification dead-letters the run on the attempt that failed rather than spending its declared attempts
|
confirm the endpoint the browser leg calls is the one this request names — page.network() lists every URL the page actually fetched |
X_SCRAPE_KEY_INVALID |
the key chord names no key a browser can press |
terminal — the chord is the caller's own literal, so attempt 2 passes the identical string. Refused by parseKeyChord BEFORE any key goes down, on every driver: press() holds each modifier, presses the key and releases in reverse, and a chord refused halfway through would leave Control held on the page for every later verb. Four shapes: an unknown modifier (Ctrl, Cmd, meta — the browser holds only Meta, Control, Alt, Shift), an empty key or a trailing +, an empty chord, and a modifier written twice |
page.press('Meta+K') — spell the chord as zero or more of Meta, Control, Alt, Shift joined by + and then the key — 'Meta+K', 'Escape', 'Shift+Tab' — using the browser's own key names; type a literal + with page.type()
|
X_SCRAPE_LAUNCHER_INCOMPLETE |
the launcher's page or browser lacks a method the driver forwards to | the launcher hands back a page or browser missing a method every page verb forwards to, usually an old puppeteer-core |
bun add puppeteer-core@latest — then localBrowser({ launcher: puppeteer }) — a current puppeteer-core carries every one |
X_SCRAPE_LAUNCH_ARGS_INVALID |
the launch options carry arguments this driver cannot hand the browser |
terminal. Refused before anything is launched: localBrowser({ options: { args } }) is not an array of strings, or it sets a proxy switch (--proxy-server, --proxy-bypass-list, --proxy-pac-url, --proxy-auto-detect, --no-proxy-server, either prefix, any case). The exit is the driver's proxy or the run's egress, appended after the caller's args; a second one in the args would let the browser leave by a route the session does not report. The fix lists every localBrowser( call and proxy switch; the cause names the edit — delete the switch and pass the exit as localBrowser({ proxy }) or scrape({ egress }), or pass args as a list of strings |
grep -rnE --include='*.ts' -e 'localBrowser\(' -e '--(no-)?proxy-' . |
X_SCRAPE_NOT_ACTIONABLE |
the element is present and cannot be acted on |
terminal — something is in the way. Disabled, hidden, zero-sized, or covered by an overlay; cause names which. The separation from X_SCRAPE_SELECTOR_MISSING is the point — one is a rewrite, the other is a wait or a dismiss |
wait for the state that unblocks it — page.waitFor(selector, { state: 'enabled' }) — or dismiss whatever overlays it before the click |
X_SCRAPE_OUTPUT_INVALID |
the extracted rows do not match the extract schema | terminal. The selectors still match and what they yield no longer parses — a column that became empty, a number that became a range, a date that changed format | align the extract schema with the page — the run's page.html artifact holds the markup the rows came from |
X_SCRAPE_PAGE_CRASHED |
the renderer process died | terminal, and it is the one liveness fault that is. A dead renderer takes its tab's state with it, so a re-run of a half-submitted form is the incident rather than the recovery. Out of memory far more often than a bug | lower concurrency: on the scrape() definition, or raise the container memory limit in docker-compose.prod.yml
|
X_SCRAPE_PDF_BUSY |
every pdf() render slot stayed busy past the queue deadline | more renders arrived at once than the renderer launches browsers for (maxConcurrent, default 2), and this one waited out queueTimeoutMs — each Chrome holds 150-300 MB, so the bound keeps a burst from exhausting the web role memory |
pdf({ renderer: chromePdfRenderer({ maxConcurrent: 4 }) }) — only on a host with the memory for it; a caller retries after the Retry-After it was answered |
X_SCRAPE_PDF_OPTIONS_INVALID |
the PDF request names no single page size a print engine can lay out | page.pdf() was given format beside widthMm/heightMm, one of widthMm/heightMm alone, a size beside preferCssPageSize, or a measure that is not a finite number | page.pdf({ format: 'a4' }) — exactly one size: a format, or widthMm with heightMm, or preferCssPageSize alone |
X_SCRAPE_PDF_ORIGIN_MISSING |
pdf() has no origin to render the route on | APP_URL is unset in this process and the pdf() declaration names no origin, so there is no server for the browser to load the route from | set APP_URL=https://your.app in the environment, or pdf({ origin: () => 'http://127.0.0.1:3000' }) in the declaring file |
X_SCRAPE_PDF_ROUTE_FAILED |
the route pdf() printed answered an error status | the page the browser loaded answered 4xx or 5xx — a missing record, a route the printing browser is not authorised for (no headers: given), or a server error — and printing it would store an error page as the document | curl -i "$APP_URL<the path the cause names>" — then pass pdf({ headers }) for a private route, or fix the page |
X_SCRAPE_PDF_ROUTE_INVALID |
a pdf() route is not a path on this app | route() answered something other than a path beginning with one slash — a full URL, a protocol-relative //host path or a backslash — and the printing browser only ever loads the app own origin | return a path from route(): pdf({ route: ({ input }) => '/invoices/' + input.id + '/print' }) |
X_SCRAPE_PROFILE_LOCKED |
another process holds this browser profile |
terminal. Two runs pointed at one profileDir, or a previous browser died without clearing its lock |
rm -f <profileDir>/SingletonLock once no browser is using it, or give this run its own localBrowser({ profileDir })
|
X_SCRAPE_PROMPT_UNANSWERED |
a login step asked for a code and nothing answered |
terminal, and in NEVER_RETRIED — an out-of-band code is not something a second attempt invents. Three raises: the scrape() declares no prompt; eventPrompt({ timeout }) saw no answer published within the timeout; or the event published for the prompt carries no sealed answer — a hand-written publishEvent()
|
no prompt: add prompt: async (request) => … to the scrape() definition. Timed out or unsealed: answerPrompt({ runId, index, answer }) from a process on the same stored event bus, with the run id and index the cause's event name ends with |
X_SCRAPE_RECOVER_REFUSED |
the recovery hook declined to recover this failure |
terminal. recover() was declared and returned no strategy for this failure, so the original failure stands and the run ends where the hook left it |
widen recover() to handle this failure, or drop recover: and let the job's retry policy own it |
X_SCRAPE_REDIRECT_LOOP |
the HTTP leg followed its hop limit of redirects without an answer |
terminal. A chain that does not settle is a property of the endpoint and of the session reaching it — attempt 2 restores the same cookies and walks the same MAX_REDIRECT_HOPS (10) hops. The reachable case is a login redirecting to a page that redirects back to the login. Every hop is screened against allowHosts and robots and recorded in page.network(), so the chain is readable |
request the URL the chain settles on directly — page.network() lists every hop — or repair the credential in secrets:
|
X_SCRAPE_REMOTE_REQUIRED |
this driver needs a cdpUrl and was given none |
terminal. remoteBrowser() attaches to a browser somebody else started; there is nothing to attach to |
pass remoteBrowser({ cdpUrl: env.SCRAPE_CDP_URL }), or use localBrowser({ launcher }) to start one in this container |
X_SCRAPE_ROBOTS_DISALLOWED |
robots.txt disallows this path |
terminal. The site's own file refuses this path for the declared user agent | declare robots: { ignore: '<the written reason this run is permitted>' } on the scrape() definition, or scrape a path robots.txt allows |
X_SCRAPE_SECRET_EXPOSED |
an artifact would have carried a secret this run typed | terminal. A screenshot or a video was requested after a secret was typed into the page, and pixels cannot be redacted afterwards. Refused rather than captured-and-scrubbed | call artifact.html() instead — page HTML is redacted by value and password fields are blanked — or take the capture before the secret is typed |
X_SCRAPE_SELECTOR_MISSING |
the selector never appeared inside its window |
terminal — the page changed. The markup this scraper was written against is not what the site served. Deliberately distinct from X_SCRAPE_NOT_ACTIONABLE: nothing matched at all, so no wait and no retry can produce one |
open the run's page.html artifact and re-derive the selector, or raise the timeout on this waitFor({ timeout })
|
X_SCRAPE_SESSION_EXPIRED |
the restored session is no longer valid and nothing can renew it |
terminal, and it burns the session: the stored state failed its auth.validate() probe and the definition declares no auth.login, so there is nothing to replace it with |
add auth: { login } to the scrape() definition so an expired session can be replaced, or drop auth.validate and let the body handle the logged-out page |
X_SCRAPE_TIMEOUT |
the step exceeded its wall-clock budget | retryable. A slow site, a slow network, or a pass doing too much inside one budget | raise timeout: on the scrape() definition, or split the pass so each step.run stays inside one budget |
X_SCRAPE_WATCHDOG_STOPPED |
the wedge watchdog stopped measuring and the run was ended | the clock the scrape definition (or apps/<app>/runtime.ts) supplied has a sleep() that rejects, so the guard's poll loop died and nothing was left watching for a wedged browser. Ended rather than run unwatched: a guard that stopped measuring is the first incident with nothing armed. A separate code from X_SCRAPE_WEDGED on purpose — that one means the page stopped answering, and sending a reader there to investigate a page that is fine is the whole cost of reusing it. Terminal, where its sibling is retryable: attempt 2 reaches the same clock identically, so retrying buys five browser launches that die at the first poll — and on an authenticated target, five arrivals at a login |
drop the custom clock: from the scrape() definition and any clock: line in apps/<app>/runtime.ts — @ultimat3/core's systemClock is the only SleepClock whose sleep() cannot reject — then re-run |
X_SCRAPE_WEDGED |
the browser stopped answering and was killed |
retryable. No browser activity at all for watchdog.idleMs, so the watchdog killed the process and closed the browser — an attached one too. The session went with it |
raise watchdog: { idleMs } on the scrape() definition if the site is genuinely this slow, or call page.activity() from the channel the framework cannot see (your own CDP session, a warm-up), otherwise re-run — the browser was closed and its session is gone |
X_SCRAPE_YIELD_COLLAPSED |
the run succeeded and returned far too little |
terminal, and the silent-green alarm. Two triggers: fewer rows than expect.minRows, or a fall past expect.maxDrop against the trailing median of the last expect.window runs (default 7, and no baseline below 3 samples). A run that succeeds and returns nothing is the worst failure a scraper has, because it stays green. A collapsed run is deliberately not recorded — check, then record, and record only a run that passed, or three broken runs at 2 rows make the median 2 and the alarm silences itself exactly when it was working |
open the run's page.html artifact and compare it with the extract selectors — a collapse is the page changing far more often than the data changing |
X_SCRAPE_YIELD_HISTORY_MISSING |
a maxDrop is declared and no history store can supply its baseline |
terminal, raised at declaration time. expect.maxDrop measures a fall against a trailing baseline, and with no history: store expect.ts reads [] forever — so X_SCRAPE_YIELD_COLLAPSED can never fire and the alarm is silently inert. expect.minRows is an absolute floor checked before the history gate and stays legal on its own |
add history: memoryYieldHistory() for a test, or your own YieldHistory in production — or drop expect.maxDrop and keep expect.minRows
|
@ultimat3/scraping owns its codes in SCRAPE_OWNED_ERROR_CODES and borrows two (X_NOT_IMPLEMENTED, X_ENV_MISSING) — the count is not written here because nothing derives it, and a number in prose beside a list that grows is a number that goes stale; bun run manifest re-derives the owner of every code. Every one is classified retryable or terminal once, in SCRAPE_ERROR_RETRY (packages/scraping/src/errors.ts), rather than re-decided by whichever catch saw it — a scraper's whole operational question is "was that the site being slow, or the site being different?"
The rule, stated once: retryable means the same code, run again, has a real chance of a different answer. Transport, budget and liveness faults qualify. Anything that says "the page is not the page this scraper was written against" does not — attempt 5 hammers a site with a request that cannot succeed, and on an authenticated target that is how three wrong attempts lock an account.
Two sets override the table, in failures.ts:
| Set | Codes | Effect |
|---|---|---|
NEVER_RETRIED |
X_SCRAPE_AUTH_FAILED, X_SCRAPE_PROMPT_UNANSWERED
|
no retry policy, backoff or recovery hook can reach them |
BURNS_SESSION |
X_SCRAPE_BLOCKED, X_SCRAPE_SESSION_EXPIRED
|
the persisted identity is discarded before the retry — a flagged profile stays flagged, so reloading it re-trips the same block every time |
X_NOT_IMPLEMENTED and X_ENV_MISSING are @ultimat3/core's and are documented on its page: the first is the honest stub the agent recovery seam throws, the second is what a declared secret with no value in the environment is — a missing environment variable, not a scraping concept needing its own code.
Ultimate — v31.44.0 As of 2026-10. The full-stack framework where the primary developer is an AI agent: bunx create-ultimate myapp → Getting started · live demo. MIT licensed. What npm serves is npm view @ultimat3/core version, never this line, and this footer is the only page that stamps a version.
Repository · Issues · Changelog · llms.txt
Edits to these pages are synced from wiki/ in the repository — change the file there, not the wiki, or the next sync overwrites it.
Start
- Getting started
- Installation
- Shapes
- Bare VM
- Project layout
- Migrating an existing app
- Compared to other stacks
Tutorials
- 1 · First app
- 2 · First feature
- 3 · Auth and admin
- 4 · Jobs and realtime
- 5 · Deploy free
- 6 · Growing up
Primitives
- The eight primitives
- Building your own base
- Actions
- Entities and migrations
- Policies and authz
- Queries and live queries
- Client data
- Jobs and workflows
- Scheduled tasks
- Routes and render modes
Capabilities
- Realtime
- Caching and invalidation
- Batching and preloading
- N+1 detection
- PWA and offline
- Client navigation
- MCP and AI
- Agents
- Admin dashboard
- Scraping
- Auth
- Notify
- Storage and uploads
- Feature flags
- SEO
- Static assets
Cross-cutting
- I18n
- Theming
- UI components
- Interface rules
- Timezones and dates
- Money
- Outbound HTTP
- Resource management
- Migrations and backfills
- Testing
- CI: the gate across parallel jobs
Reference