Skip to content

Error Codes scraping

github-actions[bot] edited this page Nov 16, 2026 · 2 revisions

Error codes: @ultimat3/scraping

Part of Error codes: every code @ultimat3/scraping registers. x errors explain <CODE> --json prints the same row.

Code Means Typical cause Fix
X_CDP_BROWSER_MISSING no Chrome or Chromium is installed for the e2e driver to launch the driver launches a real browser over raw CDP and carries no dependency to do it, so what it needs is a binary on disk — CHROME_PATH first, then the four /usr/bin names GitHub-hosted ubuntu-latest ships. Reached only by a caller that has already decided a browser is required: openE2eBrowserIfAvailable() answers undefined instead, so a machine with no browser SKIPS rather than failing export CHROME_PATH=/usr/bin/google-chrome — or leave it unset and let the browser-backed suite skip
X_CDP_CALL_FAILED the browser refused a DevTools call the call reached the browser and came back an error frame, or the connection died under it — every in-flight call is settled on close, so a dead browser reports this rather than one deadline per call print what the page had: await page.evaluate(() => document.body.innerHTML) — a call that refuses the same way means the browser itself is gone, so read the launch above it
X_CDP_LAUNCH_FAILED the browser started and never announced a DevTools endpoint Chrome is driven over --remote-debugging-pipe, so its first answer IS the announcement. It gets the launch deadline (60 s, or timeoutMs when that is larger) and, unanswered, is reaped and started ONCE more on a fresh profile. The cause carries each launch: whether it exited (with its code) or was still silent at the deadline, and its last stderr lines — D-Bus lines alone are start-up noise, so a deadline beside them is a machine too slow, an exit code a crash, a rejected flag or a sandbox refusal /usr/bin/google-chrome --headless=new --no-sandbox --disable-dev-shm-usage --disable-gpu --remote-debugging-port=0 about:blank — the binary the cause names; a healthy one prints DevTools listening on ws://…
X_CDP_TIMEOUT a DevTools call did not answer inside its deadline its own code rather than X_TIMEOUT, because the actionable half is WHICH call: a hung Page.navigate is an app that never finishes responding, a hung Runtime.evaluate is an expression that never settles. A call the connection timed carries what the wire did meanwhile — meta.reading is target-gone (its session detached, or its target was destroyed or crashed), lost-in-transport (meta.framesDropped frames arrived unparseable after the call) or no-answer (the target was there and silent; meta.framesArrived, meta.lastFrames, meta.navigations, meta.transportOpen say what else happened) no-answer: raise timeoutMs on installE2eDriver({ timeoutMs }), or find the request the page is still waiting on. The other two: x test e2e --json again — the cause names what went
X_SCRAPE_AUTH_FAILED the credentials were rejected TERMINAL, always, and the retry table cannot be talked out of it. It is in NEVER_RETRIED, so no retry policy, no backoff and no recovery hook can reach it. A site that locks an account after three wrong attempts turns a retrying framework into the thing that destroys the user's account correct the credential in .env.local for the name listed in secrets: on the scrape() definition — this run will not be retried, deliberately: a second wrong attempt is how an account gets locked
X_SCRAPE_BLOCKED the site refused this client — the identity is spent retryable, and it burns the session first. Retrying a block on the same persisted identity re-trips it every time: the flagged cookies are the thing being refused, so the retry has to arrive as somebody else or it is arithmetic, not a retry the session is burned and the next attempt starts a new identity — lower rate: on the scrape() definition if this repeats
X_SCRAPE_BODY_TOO_LARGE the HTTP leg's response body ran past maxBytes terminal. A response's size is a property of the endpoint, so attempt 2 reads the same bytes and dies identically. The default is 32 MiB (DEFAULT_HTTP_MAX_BYTES) — generous for the JSON endpoint behind a paginated page, far under what OOM-kills a worker. Before this the leg did await response.text(), which materialises the whole body: AbortSignal.timeout bounds TIME, not bytes, so a 30s stream at 50 MB/s was a 1.5 GB allocation and the run persisted nothing raise it for the one call — http.request(url, { maxBytes: 64 * 1024 * 1024 }) — or narrow the endpoint so it answers a page rather than a corpus. Never raise it globally: the cap is what stops one hostile origin taking the worker down
X_SCRAPE_BROWSER_MISSING there is no browser on this machine for the driver to launch Terminal, where its sibling X_SCRAPE_BROWSER_UNREACHABLE is retryable, and the split is the whole point. A launch that fails because there is no browser binary — or because no executablePath was configured, which is every puppeteer-core with nothing passed — is a misconfiguration: attempt 2 spawns the same absent path, five times on a five-attempt policy, and on an authenticated target that is five arrivals at a login for no chance of a different answer. Raised by browserUnreachable() when the launcher's own words say the browser is absent (must be specified, ENOENT, EACCES, Could not find Chrome); everything else stays the retryable transport code install a Chrome or Chromium and name it — localBrowser({ launcher, executablePath: env.CHROME_PATH }) — or attach to one somebody else is running with remoteBrowser({ cdpUrl: env.SCRAPE_CDP_URL })
X_SCRAPE_BROWSER_UNREACHABLE the browser went away mid-run retryable. The browser process or host died, or stopped answering, after the attach succeeded raise watchdog: { idleMs } on the scrape() definition if the site is genuinely slow, otherwise re-run once the browser host is back
X_SCRAPE_CAPTURE_INVALID the capture names a framing no picture can be taken with terminal — the rectangle is the caller's own literal, so attempt 2 passes the identical one, and it is refused before any driver runs, so there is no half-done capture to resume. Three shapes: fullPage beside a clip (CDP silently ignores one, so the caller gets a picture they did not ask for), a rectangle with no area or a non-finite number, and a clip on page.pdf(), which has no crop and would answer a whole-page PDF to a caller who asked for one component. A rectangle merely BELOW THE FOLD is accepted — this package sets and reads no viewport, and refusing it would reject the component-screenshot case it exists for drop fullPage or drop clip; measure the element with page.query('<selector>') and pass a rectangle with a positive width and height; a PDF has no crop — use page.screenshot({ clip })
X_SCRAPE_CDP_ATTACH_FAILED the CDP endpoint refused the attach retryable. A browser host still starting, a wrong cdpUrl, or an endpoint that is up and not speaking CDP. Also raised when a CdpResolver on remoteBrowser({ cdpUrl }) could not rent a browser. The endpoint is named by scheme and host only — meta.cdpUrl is wss://host:port — because a connect URL carries a token; the underlying throw is rendered through core's renderThrowable with URL parts stripped curl "$CDP_URL/json/version" to confirm the endpoint answers, then pass that webSocketDebuggerUrl to remoteBrowser({ cdpUrl })
X_SCRAPE_DOWNLOAD_TIMEOUT the download never landed retryable. A slow file, or a trigger that is not the element that starts the download raise the timeout on page.download({ timeout }), or confirm the trigger is the element that starts the download
X_SCRAPE_DRIVER_UNKNOWN no browser driver is installed for this run terminal. No scrapeDriver in apps/<app>/runtime.ts and the definition passed no driver:; cause lists what is installed, or none apps/<app>/runtime.ts: export const runtime = { scrapeDriver: localBrowser() } — or pass driver: fakeBrowser() on the scrape() definition
X_SCRAPE_EGRESS_IN_PAYLOAD the run exit carries a credential that is also in the job payload scrape({ egress }) answered a proxy URL whose password is readable out of the run's own input, so it rode the queue payload and is stored in x_jobs in the clear. Refused before the browser opens; terminal, because every remaining attempt reads the same row look the exit up by id inside the worker — egress: async ({ connectionId }) => (await repo.connectionById(connectionId))?.exit — and drop the exit field from the scrape input
X_SCRAPE_EGRESS_UNSUPPORTED this driver cannot give the session the exit the run asked for terminal. egress: named an exit for this run and the driver cannot dial it: remoteBrowser({ cdpUrl }) with a fixed URL attaches to a browser already bound to another exit, or the launcher has no page.authenticate() for an exit that carries credentials. The cause names scheme and host only, never the credentials pass remoteBrowser({ cdpUrl: resolver }) so each run rents a browser on its own exit, or drop egress: from the scrape() definition and set the exit once on the driver — remoteBrowser({ proxy })
X_SCRAPE_FIXTURE_MISSING the fixture driver has no recording for this request terminal. An offline driver never reaches the network, so an unrecorded URL has no answer to give add <fixtureDir>/<the recording file> holding { "url": "…", "html": "…" }, or point fixtureBrowser() at the directory that already holds it
X_SCRAPE_FIXTURE_STALE the recording is older than the fixture max age terminal. A test passing against a year-old copy of a page proves the test still parses the copy, not the site. A recordedAt that is not a date is stale too: its age cannot be shown to be inside maxAge re-record the fixture directory, or raise maxAge on fixtureBrowser({ maxAge }) with the reason an old recording still proves something
X_SCRAPE_FRAME_OUT_OF_PROCESS the frame runs in another renderer process the page session cannot read a cross-site iframe under site isolation renders in its own process, so the page session cannot read its accessibility tree localBrowser({ launcher, options: { args: ['--disable-site-isolation-trials'] } }) — keeps a cross-site frame in the page's process; or read it with page.query()
X_SCRAPE_HOST_BLOCKED the page asked for a host allowHosts does not list terminal. A third-party asset, an analytics beacon or a redirect off the allow-list. The allow-list is default deny, so silence blocks. Also raised on the HTTP leg when a host only a wildcard ('*', '*.example.com') admitted resolves to a loopback, private or link-local address — the cause names the address class, never the address add the host to allowHosts on the scrape() definition — cause names both the requested URL and the current list; for an inward host, name it exactly
X_SCRAPE_HTTP_FAILED the site answered the HTTP leg with a non-2xx status retryable by default — 429, 5xx and a deploy window are transient far more often than not. A 4xx that isRetryableStatus (@ultimat3/core) calls permanent is thrown with a per-instance terminal override, because the same code is both and only the throw site knows which it saw — 408, 409, 425 and 429 keep their attempts As of 2026-08-23. This row said "below 500 other than 429" until then, and a private copy of the table here called 408 and 425 terminal while the rest of the framework called them retryable; a terminal classification dead-letters the run on the attempt that failed rather than spending its declared attempts confirm the endpoint the browser leg calls is the one this request names — page.network() lists every URL the page actually fetched
X_SCRAPE_KEY_INVALID the key chord names no key a browser can press terminal — the chord is the caller's own literal, so attempt 2 passes the identical string. Refused by parseKeyChord BEFORE any key goes down, on every driver: press() holds each modifier, presses the key and releases in reverse, and a chord refused halfway through would leave Control held on the page for every later verb. Four shapes: an unknown modifier (Ctrl, Cmd, meta — the browser holds only Meta, Control, Alt, Shift), an empty key or a trailing +, an empty chord, and a modifier written twice page.press('Meta+K') — spell the chord as zero or more of Meta, Control, Alt, Shift joined by + and then the key — 'Meta+K', 'Escape', 'Shift+Tab' — using the browser's own key names; type a literal + with page.type()
X_SCRAPE_LAUNCHER_INCOMPLETE the launcher's page or browser lacks a method the driver forwards to the launcher hands back a page or browser missing a method every page verb forwards to, usually an old puppeteer-core bun add puppeteer-core@latest — then localBrowser({ launcher: puppeteer }) — a current puppeteer-core carries every one
X_SCRAPE_LAUNCH_ARGS_INVALID the launch options carry arguments this driver cannot hand the browser terminal. Refused before anything is launched: localBrowser({ options: { args } }) is not an array of strings, or it sets a proxy switch (--proxy-server, --proxy-bypass-list, --proxy-pac-url, --proxy-auto-detect, --no-proxy-server, either prefix, any case). The exit is the driver's proxy or the run's egress, appended after the caller's args; a second one in the args would let the browser leave by a route the session does not report. The fix lists every localBrowser( call and proxy switch; the cause names the edit — delete the switch and pass the exit as localBrowser({ proxy }) or scrape({ egress }), or pass args as a list of strings grep -rnE --include='*.ts' -e 'localBrowser\(' -e '--(no-)?proxy-' .
X_SCRAPE_NOT_ACTIONABLE the element is present and cannot be acted on terminal — something is in the way. Disabled, hidden, zero-sized, or covered by an overlay; cause names which. The separation from X_SCRAPE_SELECTOR_MISSING is the point — one is a rewrite, the other is a wait or a dismiss wait for the state that unblocks it — page.waitFor(selector, { state: 'enabled' }) — or dismiss whatever overlays it before the click
X_SCRAPE_OUTPUT_INVALID the extracted rows do not match the extract schema terminal. The selectors still match and what they yield no longer parses — a column that became empty, a number that became a range, a date that changed format align the extract schema with the page — the run's page.html artifact holds the markup the rows came from
X_SCRAPE_PAGE_CRASHED the renderer process died terminal, and it is the one liveness fault that is. A dead renderer takes its tab's state with it, so a re-run of a half-submitted form is the incident rather than the recovery. Out of memory far more often than a bug lower concurrency: on the scrape() definition, or raise the container memory limit in docker-compose.prod.yml
X_SCRAPE_PDF_BUSY every pdf() render slot stayed busy past the queue deadline more renders arrived at once than the renderer launches browsers for (maxConcurrent, default 2), and this one waited out queueTimeoutMs — each Chrome holds 150-300 MB, so the bound keeps a burst from exhausting the web role memory pdf({ renderer: chromePdfRenderer({ maxConcurrent: 4 }) }) — only on a host with the memory for it; a caller retries after the Retry-After it was answered
X_SCRAPE_PDF_OPTIONS_INVALID the PDF request names no single page size a print engine can lay out page.pdf() was given format beside widthMm/heightMm, one of widthMm/heightMm alone, a size beside preferCssPageSize, or a measure that is not a finite number page.pdf({ format: 'a4' }) — exactly one size: a format, or widthMm with heightMm, or preferCssPageSize alone
X_SCRAPE_PDF_ORIGIN_MISSING pdf() has no origin to render the route on APP_URL is unset in this process and the pdf() declaration names no origin, so there is no server for the browser to load the route from set APP_URL=https://your.app in the environment, or pdf({ origin: () => 'http://127.0.0.1:3000' }) in the declaring file
X_SCRAPE_PDF_ROUTE_FAILED the route pdf() printed answered an error status the page the browser loaded answered 4xx or 5xx — a missing record, a route the printing browser is not authorised for (no headers: given), or a server error — and printing it would store an error page as the document curl -i "$APP_URL<the path the cause names>" — then pass pdf({ headers }) for a private route, or fix the page
X_SCRAPE_PDF_ROUTE_INVALID a pdf() route is not a path on this app route() answered something other than a path beginning with one slash — a full URL, a protocol-relative //host path or a backslash — and the printing browser only ever loads the app own origin return a path from route(): pdf({ route: ({ input }) => '/invoices/' + input.id + '/print' })
X_SCRAPE_PROFILE_LOCKED another process holds this browser profile terminal. Two runs pointed at one profileDir, or a previous browser died without clearing its lock rm -f <profileDir>/SingletonLock once no browser is using it, or give this run its own localBrowser({ profileDir })
X_SCRAPE_PROMPT_UNANSWERED a login step asked for a code and nothing answered terminal, and in NEVER_RETRIED — an out-of-band code is not something a second attempt invents. Three raises: the scrape() declares no prompt; eventPrompt({ timeout }) saw no answer published within the timeout; or the event published for the prompt carries no sealed answer — a hand-written publishEvent() no prompt: add prompt: async (request) => … to the scrape() definition. Timed out or unsealed: answerPrompt({ runId, index, answer }) from a process on the same stored event bus, with the run id and index the cause's event name ends with
X_SCRAPE_RECOVER_REFUSED the recovery hook declined to recover this failure terminal. recover() was declared and returned no strategy for this failure, so the original failure stands and the run ends where the hook left it widen recover() to handle this failure, or drop recover: and let the job's retry policy own it
X_SCRAPE_REDIRECT_LOOP the HTTP leg followed its hop limit of redirects without an answer terminal. A chain that does not settle is a property of the endpoint and of the session reaching it — attempt 2 restores the same cookies and walks the same MAX_REDIRECT_HOPS (10) hops. The reachable case is a login redirecting to a page that redirects back to the login. Every hop is screened against allowHosts and robots and recorded in page.network(), so the chain is readable request the URL the chain settles on directly — page.network() lists every hop — or repair the credential in secrets:
X_SCRAPE_REMOTE_REQUIRED this driver needs a cdpUrl and was given none terminal. remoteBrowser() attaches to a browser somebody else started; there is nothing to attach to pass remoteBrowser({ cdpUrl: env.SCRAPE_CDP_URL }), or use localBrowser({ launcher }) to start one in this container
X_SCRAPE_ROBOTS_DISALLOWED robots.txt disallows this path terminal. The site's own file refuses this path for the declared user agent declare robots: { ignore: '<the written reason this run is permitted>' } on the scrape() definition, or scrape a path robots.txt allows
X_SCRAPE_SECRET_EXPOSED an artifact would have carried a secret this run typed terminal. A screenshot or a video was requested after a secret was typed into the page, and pixels cannot be redacted afterwards. Refused rather than captured-and-scrubbed call artifact.html() instead — page HTML is redacted by value and password fields are blanked — or take the capture before the secret is typed
X_SCRAPE_SELECTOR_MISSING the selector never appeared inside its window terminal — the page changed. The markup this scraper was written against is not what the site served. Deliberately distinct from X_SCRAPE_NOT_ACTIONABLE: nothing matched at all, so no wait and no retry can produce one open the run's page.html artifact and re-derive the selector, or raise the timeout on this waitFor({ timeout })
X_SCRAPE_SESSION_EXPIRED the restored session is no longer valid and nothing can renew it terminal, and it burns the session: the stored state failed its auth.validate() probe and the definition declares no auth.login, so there is nothing to replace it with add auth: { login } to the scrape() definition so an expired session can be replaced, or drop auth.validate and let the body handle the logged-out page
X_SCRAPE_TIMEOUT the step exceeded its wall-clock budget retryable. A slow site, a slow network, or a pass doing too much inside one budget raise timeout: on the scrape() definition, or split the pass so each step.run stays inside one budget
X_SCRAPE_WATCHDOG_STOPPED the wedge watchdog stopped measuring and the run was ended the clock the scrape definition (or apps/<app>/runtime.ts) supplied has a sleep() that rejects, so the guard's poll loop died and nothing was left watching for a wedged browser. Ended rather than run unwatched: a guard that stopped measuring is the first incident with nothing armed. A separate code from X_SCRAPE_WEDGED on purpose — that one means the page stopped answering, and sending a reader there to investigate a page that is fine is the whole cost of reusing it. Terminal, where its sibling is retryable: attempt 2 reaches the same clock identically, so retrying buys five browser launches that die at the first poll — and on an authenticated target, five arrivals at a login drop the custom clock: from the scrape() definition and any clock: line in apps/<app>/runtime.ts — @ultimat3/core's systemClock is the only SleepClock whose sleep() cannot reject — then re-run
X_SCRAPE_WEDGED the browser stopped answering and was killed retryable. No browser activity at all for watchdog.idleMs, so the watchdog killed the process and closed the browser — an attached one too. The session went with it raise watchdog: { idleMs } on the scrape() definition if the site is genuinely this slow, or call page.activity() from the channel the framework cannot see (your own CDP session, a warm-up), otherwise re-run — the browser was closed and its session is gone
X_SCRAPE_YIELD_COLLAPSED the run succeeded and returned far too little terminal, and the silent-green alarm. Two triggers: fewer rows than expect.minRows, or a fall past expect.maxDrop against the trailing median of the last expect.window runs (default 7, and no baseline below 3 samples). A run that succeeds and returns nothing is the worst failure a scraper has, because it stays green. A collapsed run is deliberately not recorded — check, then record, and record only a run that passed, or three broken runs at 2 rows make the median 2 and the alarm silences itself exactly when it was working open the run's page.html artifact and compare it with the extract selectors — a collapse is the page changing far more often than the data changing
X_SCRAPE_YIELD_HISTORY_MISSING a maxDrop is declared and no history store can supply its baseline terminal, raised at declaration time. expect.maxDrop measures a fall against a trailing baseline, and with no history: store expect.ts reads [] forever — so X_SCRAPE_YIELD_COLLAPSED can never fire and the alarm is silently inert. expect.minRows is an absolute floor checked before the history gate and stays legal on its own add history: memoryYieldHistory() for a test, or your own YieldHistory in production — or drop expect.maxDrop and keep expect.minRows

Notes

Scraping

@ultimat3/scraping owns its codes in SCRAPE_OWNED_ERROR_CODES and borrows two (X_NOT_IMPLEMENTED, X_ENV_MISSING) — the count is not written here because nothing derives it, and a number in prose beside a list that grows is a number that goes stale; bun run manifest re-derives the owner of every code. Every one is classified retryable or terminal once, in SCRAPE_ERROR_RETRY (packages/scraping/src/errors.ts), rather than re-decided by whichever catch saw it — a scraper's whole operational question is "was that the site being slow, or the site being different?"

The rule, stated once: retryable means the same code, run again, has a real chance of a different answer. Transport, budget and liveness faults qualify. Anything that says "the page is not the page this scraper was written against" does not — attempt 5 hammers a site with a request that cannot succeed, and on an authenticated target that is how three wrong attempts lock an account.

Two sets override the table, in failures.ts:

Set Codes Effect
NEVER_RETRIED X_SCRAPE_AUTH_FAILED, X_SCRAPE_PROMPT_UNANSWERED no retry policy, backoff or recovery hook can reach them
BURNS_SESSION X_SCRAPE_BLOCKED, X_SCRAPE_SESSION_EXPIRED the persisted identity is discarded before the retry — a flagged profile stays flagged, so reloading it re-trips the same block every time

X_NOT_IMPLEMENTED and X_ENV_MISSING are @ultimat3/core's and are documented on its page: the first is the honest stub the agent recovery seam throws, the second is what a declared secret with no value in the environment is — a missing environment variable, not a scraping concept needing its own code.

Clone this wiki locally