From 741c3b1e83733517c8af57cde7309556f7cf52f3 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 31 Jul 2026 11:10:09 +0000 Subject: [PATCH 01/17] Replace old product taxonomy with canonical primitives Update descriptive/label/help/comment text across the CLI, registry, skills, templates, workflows, and docs to use the four canonical primitives: - "Universal Scraper API" -> Fetch / Extract / "Fetch and Extract" (umbrella) - "Scraping Browser" -> Browser Sessions - "Residential Proxies" (standalone) -> folded into the primitives - "Batch Scraper API" -> "Batch" in prose/labels, "Batch API" in error strings Text-only: no command names, flags, endpoints, or hosts changed. Typecheck passes. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NsXfB6dR3rRX64K3NwyuoD --- README.md | 24 +++++++++---------- docs/capabilities.md | 8 +++---- registry/capabilities.json | 12 +++++----- registry/skills.json | 4 ++-- registry/templates.json | 4 ++-- skills/batch-jobs/SKILL.md | 4 ++-- skills/cost-control/SKILL.md | 2 +- skills/extract/SKILL.md | 2 +- skills/interact-browser/SKILL.md | 2 +- skills/protected-fetch/SKILL.md | 4 ++-- skills/zenrows/SKILL.md | 6 ++--- src/adapters/batch.ts | 4 ++-- src/adapters/extract.ts | 2 +- src/adapters/protected-fetch.ts | 4 ++-- src/cli/commands/batch.ts | 4 ++-- src/cli/commands/browser.ts | 10 ++++---- src/cli/commands/extract.ts | 6 ++--- src/cli/commands/fetch.ts | 4 ++-- src/cli/commands/usage.ts | 2 +- src/core/batch-api.ts | 18 +++++++------- src/core/config.ts | 4 ++-- src/core/http.ts | 4 ++-- src/core/usage.ts | 2 +- templates/batch-jsonl-pipeline/README.md | 4 ++-- templates/protected-fetch-node/README.md | 2 +- workflows/competitor-intelligence/WORKFLOW.md | 4 ++-- 26 files changed, 73 insertions(+), 73 deletions(-) diff --git a/README.md b/README.md index 02fb5a1..634d4e8 100644 --- a/README.md +++ b/README.md @@ -4,8 +4,8 @@ workflows, recipes, and evals layer for giving AI agents reliable access to protected web data through Zenrows cloud infrastructure.** -It makes the Zenrows Universal Scraper API, Scraping Browser, and Residential -Proxies installable and usable directly from AI agents and developer workflows — +It makes the four Zenrows primitives — Fetch, Extract, Batch, and Browser +Sessions — installable and usable directly from AI agents and developer workflows — so an agent can reliably access protected web data without hand-rolling anti-bot handling, proxies, or browser rendering. @@ -21,14 +21,14 @@ through AI agents, developers, and teams. ## 2. Why Zenrows Normal fetch fails. Generic scrapers fail. Browser-first tools are expensive. -The Zenrows **Universal Scraper API** retrieves protected pages reliably and -structures them, while the **Scraping Browser** is there for the rare cases that +Zenrows **Fetch** retrieves protected pages reliably and **Extract** structures +them, while **Browser Sessions** are there for the rare cases that need a real browser. Zenrows wins when the workflow runs over thousands or millions of URLs. ## 3. Product architecture -The core product is the Zenrows **Universal Scraper API** +The core product is Zenrows **Fetch and Extract** (`GET https://api.zenrows.com/v1/`). The CLI exposes it two ways: `zenrows fetch` retrieves a protected page, and `zenrows extract` turns it into structured data (JSON / CSS / Markdown). Both call the same endpoint — `extract` is just @@ -36,10 +36,10 @@ that endpoint with extraction parameters, not a separate product. | Command | What it does | Status (this build) | | --- | --- | --- | -| `zenrows fetch` | Universal Scraper API — retrieve a protected page | **available** — `GET https://api.zenrows.com/v1/` | -| `zenrows extract` | Universal Scraper API — structured extraction (Autoparse / CSS / Markdown) | **available** — same `/v1/` | -| `zenrows batch` | Batch Scraper API — fan out over many URLs | beta — cloud works with beta access; local validate/estimate always | -| `zenrows browser` | Scraping Browser (CDP) / MCP escalation | experimental | +| `zenrows fetch` | Fetch — retrieve a protected page | **available** — `GET https://api.zenrows.com/v1/` | +| `zenrows extract` | Extract — structured extraction (Autoparse / CSS / Markdown) | **available** — same `/v1/` | +| `zenrows batch` | Batch — fan out over many URLs | beta — cloud works with beta access; local validate/estimate always | +| `zenrows browser` | Browser Sessions (CDP) / MCP escalation | experimental | | `zenrows mcp` | MCP server config (remote + local) | **available** | | Zenrows CLI | this repo | available | @@ -117,7 +117,7 @@ zenrows extract --output markdown ## 9. Batch (beta) -The Zenrows **Batch Scraper API** (`https://async.api.zenrows.com/v1`) fans a +Zenrows **Batch** (`https://async.api.zenrows.com/v1`) fans a protected fetch/extract out over many URLs. It is a real product in **beta**: the cloud subcommands work once your API key has beta access; without it the API returns `BATCH_ACCESS_DENIED`. Local spec @@ -138,8 +138,8 @@ locally or fan out with `zenrows fetch` per URL. ## 10. Browser Sessions -Escalation only, gated by `policy.allow_browser`. Backed by the Zenrows Scraping -Browser (CDP) and the `@zenrows/mcp` `browser_*` tools. +Escalation only, gated by `policy.allow_browser`. Backed by Zenrows Browser +Sessions (CDP) and the `@zenrows/mcp` `browser_*` tools. ## 11. MCP diff --git a/docs/capabilities.md b/docs/capabilities.md index bc0637e..934b4c7 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -18,15 +18,15 @@ Classification is based on the public Zenrows documentation: | Capability | Backend evidence | Status | | --- | --- | --- | -| `protected_fetch` | Universal Scraper API `GET https://api.zenrows.com/v1/` with `mode`, `js_render`, `premium_proxy`, `proxy_country`, `wait`/`wait_for`, `js_instructions`, `response_type`, `screenshot`, `original_status`, … | available | +| `protected_fetch` | Fetch and Extract `GET https://api.zenrows.com/v1/` with `mode`, `js_render`, `premium_proxy`, `proxy_country`, `wait`/`wait_for`, `js_instructions`, `response_type`, `screenshot`, `original_status`, … | available | | `extract` | Same `/v1/` endpoint via `autoparse`, `css_extractor`, `response_type=markdown\|plaintext` | available | -| `batch` | Zenrows Batch Scraper API `https://async.api.zenrows.com/v1` (separate host, `X-API-Key` header) — real product in beta. Cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → `BATCH_ACCESS_DENIED`. Local JSONL spec validation + credit estimation work with no key. | beta | -| `browser` | Zenrows Scraping Browser (CDP) + `@zenrows/mcp` `browser_*` tools; no managed REST sessions API | experimental | +| `batch` | Zenrows Batch `https://async.api.zenrows.com/v1` (separate host, `X-API-Key` header) — real product in beta. Cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → `BATCH_ACCESS_DENIED`. Local JSONL spec validation + credit estimation work with no key. | beta | +| `browser` | Zenrows Browser Sessions (CDP) + `@zenrows/mcp` `browser_*` tools; no managed REST sessions API | experimental | | `mcp` | Hosted `https://mcp.zenrows.com/mcp` + local `npx -y @zenrows/mcp` | available | ## Important honesty note `protected_fetch` and `extract` are the **same** product: a single `/v1/` -Universal Scraper API. "Extract" is not a separate endpoint — it is parameters +Fetch and Extract API. "Extract" is not a separate endpoint — it is parameters on that endpoint (`autoparse` / `css_extractor` / `response_type`). The CLI keeps them as separate commands only for ergonomics. diff --git a/registry/capabilities.json b/registry/capabilities.json index 87e75d8..980613b 100644 --- a/registry/capabilities.json +++ b/registry/capabilities.json @@ -8,7 +8,7 @@ "command": "zenrows fetch", "backend": "GET https://api.zenrows.com/v1/", "requiresAuth": true, - "notes": "Universal Scraper API. Confirmed params: mode=auto (Adaptive Stealth), js_render, premium_proxy, proxy_country, wait, wait_for, js_instructions, custom_headers, session_id, original_status, allowed_status_codes, block_resources, response_type, screenshot." + "notes": "Fetch and Extract. Confirmed params: mode=auto (Adaptive Stealth), js_render, premium_proxy, proxy_country, wait, wait_for, js_instructions, custom_headers, session_id, original_status, allowed_status_codes, block_resources, response_type, screenshot." }, "extract": { "key": "extract", @@ -24,18 +24,18 @@ "label": "Batch (beta)", "status": "beta", "command": "zenrows batch", - "backend": "Batch Scraper API — https://async.api.zenrows.com/v1 (X-API-Key header; separate host from the scraper /v1/)", + "backend": "Batch — https://async.api.zenrows.com/v1 (X-API-Key header; separate host from the Fetch/Extract /v1/)", "requiresAuth": true, - "notes": "The Zenrows Batch Scraper API is a real product in beta. The cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → BATCH_ACCESS_DENIED. Local value always works with no key: `zenrows batch estimate` validates JSONL job specs and estimates credit cost." + "notes": "The Zenrows Batch is a real product in beta. The cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → BATCH_ACCESS_DENIED. Local value always works with no key: `zenrows batch estimate` validates JSONL job specs and estimates credit cost." }, "browser": { "key": "browser", - "label": "Browser (Scraping Browser)", + "label": "Browser Sessions", "status": "experimental", "command": "zenrows browser", - "backend": "Scraping Browser (CDP) + @zenrows/mcp browser tools", + "backend": "Browser Sessions (CDP) + @zenrows/mcp browser tools", "requiresAuth": true, - "notes": "Zenrows Scraping Browser and the @zenrows/mcp browser_* tools exist. There is no managed REST 'sessions' API in the public docs, so this is gated as experimental and escalation-only (policy.allow_browser=false by default)." + "notes": "Zenrows Browser Sessions and the @zenrows/mcp browser_* tools exist. There is no managed REST 'sessions' API in the public docs, so this is gated as experimental and escalation-only (policy.allow_browser=false by default)." }, "mcp": { "key": "mcp", diff --git a/registry/skills.json b/registry/skills.json index 7d19f2e..227b34d 100644 --- a/registry/skills.json +++ b/registry/skills.json @@ -36,7 +36,7 @@ { "name": "batch-jobs", "type": "skill", - "description": "Scale protected fetch/extract over many URLs with the Batch Scraper API (beta): submit/track/collect jobs with beta access; validate + estimate specs locally with no key.", + "description": "Scale protected fetch/extract over many URLs with Batch (beta): submit/track/collect jobs with beta access; validate + estimate specs locally with no key.", "status": "beta", "requires_backend_capabilities": ["batch"], "requires_auth": true, @@ -47,7 +47,7 @@ { "name": "interact-browser", "type": "skill", - "description": "Escalate to a browser (Scraping Browser / MCP) only when fetch/extract cannot do the job.", + "description": "Escalate to a browser (Browser Sessions / MCP) only when fetch/extract cannot do the job.", "status": "experimental", "requires_backend_capabilities": ["browser"], "requires_auth": true, diff --git a/registry/templates.json b/registry/templates.json index 4717f83..d65b411 100644 --- a/registry/templates.json +++ b/registry/templates.json @@ -3,7 +3,7 @@ { "name": "protected-fetch-node", "type": "template", - "description": "Minimal Node.js project calling the Zenrows Universal Scraper API.", + "description": "Minimal Node.js project calling Zenrows Fetch.", "status": "available", "requires_backend_capabilities": ["protected_fetch"], "requires_auth": true, @@ -25,7 +25,7 @@ { "name": "batch-jsonl-pipeline", "type": "template", - "description": "JSONL job-spec scaffold for high-scale workloads on the Batch Scraper API: submit/track/collect with beta access, validate + estimate locally with no key (beta).", + "description": "JSONL job-spec scaffold for high-scale workloads on Batch: submit/track/collect with beta access, validate + estimate locally with no key (beta).", "status": "beta", "requires_backend_capabilities": [], "requires_auth": false, diff --git a/skills/batch-jobs/SKILL.md b/skills/batch-jobs/SKILL.md index ee31df6..e6bc028 100644 --- a/skills/batch-jobs/SKILL.md +++ b/skills/batch-jobs/SKILL.md @@ -1,6 +1,6 @@ --- name: batch-jobs -description: Scale protected fetch/extract over many URLs via the Batch Scraper API (beta). Cloud create/status/results/cancel/wait/retry-failed work with beta access; estimate/validate run locally with no key. +description: Scale protected fetch/extract over many URLs via Batch (beta). Cloud create/status/results/cancel/wait/retry-failed work with beta access; estimate/validate run locally with no key. version: 0.1.0 requires_backend_capabilities: [batch] --- @@ -11,7 +11,7 @@ Process large workloads reliably and asynchronously. Batch is where Zenrows' high-scale anti-bot advantage becomes obvious — Zenrows wins when the workflow runs over thousands, millions, or recurring sets of URLs. -> Status: **beta**. The Zenrows Batch Scraper API is a real +> Status: **beta**. The Zenrows **Batch** is a real > product in beta and runs on a separate host > (`async.api.zenrows.com/v1`). The cloud subcommands work once your account has > beta access; without it the API returns 403 → `BATCH_ACCESS_DENIED`. The diff --git a/skills/cost-control/SKILL.md b/skills/cost-control/SKILL.md index 903a041..2ed6aef 100644 --- a/skills/cost-control/SKILL.md +++ b/skills/cost-control/SKILL.md @@ -9,7 +9,7 @@ requires_backend_capabilities: [] Prefer the **cheapest reliable** configuration; escalate only with evidence. -## Cost multipliers (Universal Scraper API) +## Cost multipliers (Fetch and Extract) - Basic request: **1×** - JS rendering (`js_render`): **5×** - Premium proxies (`premium_proxy`): **10×** diff --git a/skills/extract/SKILL.md b/skills/extract/SKILL.md index aad6239..215e561 100644 --- a/skills/extract/SKILL.md +++ b/skills/extract/SKILL.md @@ -11,7 +11,7 @@ Convert protected pages into structured data. The value is **protected page access + extraction**, not generic LLM parsing. > Honest note: there is no separate `/extract` endpoint. Extraction runs on the -> same Universal Scraper API (`/v1/`) via `autoparse`, `css_extractor`, and +> same Fetch and Extract API (`/v1/`) via `autoparse`, `css_extractor`, and > `response_type`. ## Methods (available today) diff --git a/skills/interact-browser/SKILL.md b/skills/interact-browser/SKILL.md index 3fbdba1..8ed6a64 100644 --- a/skills/interact-browser/SKILL.md +++ b/skills/interact-browser/SKILL.md @@ -14,7 +14,7 @@ or Extract is insufficient. > Status: **experimental**, gated behind `policy.allow_browser` (default false). > There is no managed REST "sessions" API; browser workflows run through the -> Zenrows **Scraping Browser** (CDP — connect Playwright/Puppeteer) and the +> Zenrows **Browser Sessions** (CDP — connect Playwright/Puppeteer) and the > `@zenrows/mcp` `browser_*` tools (navigate, click, fill, screenshot, …). ## Rules diff --git a/skills/protected-fetch/SKILL.md b/skills/protected-fetch/SKILL.md index aa21460..9af8a46 100644 --- a/skills/protected-fetch/SKILL.md +++ b/skills/protected-fetch/SKILL.md @@ -8,8 +8,8 @@ requires_backend_capabilities: [protected_fetch] # Protected Fetch Retrieve pages that normal fetch, generic scrapers, or naive browser tools -cannot. This is the **core primitive** — backed by the Zenrows Universal Scraper -API (`GET https://api.zenrows.com/v1/`). +cannot. This is the **core primitive** — backed by Zenrows **Fetch** +(`GET https://api.zenrows.com/v1/`). ## When to use - You have a known URL and want its content (HTML, Markdown, text, or a PDF). diff --git a/skills/zenrows/SKILL.md b/skills/zenrows/SKILL.md index e4b84ae..51c8d0d 100644 --- a/skills/zenrows/SKILL.md +++ b/skills/zenrows/SKILL.md @@ -66,14 +66,14 @@ Run `zenrows status` for the live capability matrix. As of this toolkit: | Protected Fetch | `zenrows fetch` | available (`GET /v1/`) | | Extract (Autoparse/CSS/Markdown) | `zenrows extract` | available (same `/v1/`) | | Batch | `zenrows batch` | beta (validate specs locally) | -| Browser | `zenrows browser` | experimental (Scraping Browser / MCP) | +| Browser | `zenrows browser` | experimental (Browser Sessions / MCP) | | MCP | `zenrows mcp` | available (remote + local server) | -Protected Fetch and Extract are the same Universal Scraper API used two ways. +Protected Fetch and Extract are the same API (`GET /v1/`) used two ways. The command consults the capability matrix before any cloud call — it never fakes success. -## Cost model (Universal Scraper API) +## Cost model (Fetch and Extract) Relative multipliers: basic **1×**, JS rendering **5×**, premium proxies **10×**, both **25×**. `mode=auto` charges only for the configuration that succeeds. diff --git a/src/adapters/batch.ts b/src/adapters/batch.ts index 1999d6e..78b0042 100644 --- a/src/adapters/batch.ts +++ b/src/adapters/batch.ts @@ -1,7 +1,7 @@ /** * Batch Jobs adapter. * - * Status: `beta` — the Zenrows Batch Scraper API is in beta. + * Status: `beta` — the Zenrows Batch API is in beta. * With beta access the cloud subcommands run for real (see `core/batch-api.ts`). * Without access the API returns 403 → BATCH_ACCESS_DENIED. This adapter owns * the local, no-network pieces: validating a JSONL job spec, estimating credit @@ -176,7 +176,7 @@ function assertProxyCountryPremium(params: Record, where: strin code: "PARAM_PROXY_COUNTRY_REQUIRES_PREMIUM", message: `proxy_country needs premium proxies (or mode=auto) — ${where}.`, likely_cause: - "The Batch Scraper API only geolocates the proxy when premium_proxy=true, or in Adaptive Stealth Mode (mode=auto).", + "Batch only geolocates the proxy when premium_proxy=true, or in Adaptive Stealth Mode (mode=auto).", next_action: "Add premium_proxy (10x cost) alongside proxy_country, or set mode=auto — geo-targeting works there without the flag.", suggested_commands: ["zenrows batch create jobs.jsonl --premium-proxy --proxy-country us"], diff --git a/src/adapters/extract.ts b/src/adapters/extract.ts index a39138a..feb4fad 100644 --- a/src/adapters/extract.ts +++ b/src/adapters/extract.ts @@ -2,7 +2,7 @@ * Extract adapter. * * IMPORTANT (honest mapping): there is no separate `/extract` endpoint. - * Structured extraction runs on the same `/v1/` Universal Scraper API via: + * Structured extraction runs on the same `/v1/` Fetch and Extract API via: * - autoparse=true → automatic structured JSON (available) * - css_extractor= → selector-based field extraction (available) * - outputs= → built-in output filters → JSON (available) diff --git a/src/adapters/protected-fetch.ts b/src/adapters/protected-fetch.ts index 6280b00..2e280d0 100644 --- a/src/adapters/protected-fetch.ts +++ b/src/adapters/protected-fetch.ts @@ -1,5 +1,5 @@ /** - * Protected Fetch adapter → Zenrows Universal Scraper API (`GET /v1/`). + * Protected Fetch adapter → Zenrows Fetch (`GET /v1/`). * * Maps toolkit options to confirmed API parameters and enforces the * auto/manual contract: in Adaptive Stealth Mode (mode=auto), `js_render` and @@ -34,7 +34,7 @@ export interface FetchOptions { cssExtractor?: string; autoparse?: boolean; /** - * Comma-separated Universal Scraper API output filters (e.g. "emails,links", + * Comma-separated Fetch and Extract output filters (e.g. "emails,links", * or "*" for all available fields). Returns structured JSON. Standalone: not combined with * autoparse / css_extractor / response_type. */ diff --git a/src/cli/commands/batch.ts b/src/cli/commands/batch.ts index 46a102d..99af8de 100644 --- a/src/cli/commands/batch.ts +++ b/src/cli/commands/batch.ts @@ -1,5 +1,5 @@ /** - * `zenrows batch` — Batch Scraper API (status: beta). + * `zenrows batch` — Batch API (status: beta). * * Local (no key, always works): `estimate`/`create --dry-run`-style spec * validation + credit estimate. Cloud (needs a key + Batch beta access): @@ -18,7 +18,7 @@ import { printError, writeOut } from "../output.ts"; export const batch: Command = { name: "batch", - summary: "Run JSONL batch jobs on the Zenrows Batch Scraper API (beta).", + summary: "Run JSONL batch jobs on Zenrows Batch (beta).", usage: "zenrows batch |status |results |cancel |wait |retry-failed >", help: [ "Local (no key):", diff --git a/src/cli/commands/browser.ts b/src/cli/commands/browser.ts index 1b36e83..d5639a5 100644 --- a/src/cli/commands/browser.ts +++ b/src/cli/commands/browser.ts @@ -2,7 +2,7 @@ * `zenrows browser` — Interact / Browser Sessions (status: experimental). * * Browser is an ESCALATION layer, not the default. There is no documented REST - * "sessions" API; Zenrows exposes the Scraping Browser (CDP) and browser_* + * "sessions" API; Zenrows exposes Browser Sessions (CDP) and browser_* * tools via the @zenrows/mcp server. So this command is gated behind * policy.allow_browser and points users to those surfaces rather than faking a * sessions API. @@ -15,9 +15,9 @@ import { printError } from "../output.ts"; export const browser: Command = { name: "browser", - summary: "Browser sessions — escalation only (experimental, via Scraping Browser / MCP).", + summary: "Browser sessions — escalation only (experimental, via Browser Sessions / MCP).", usage: "zenrows browser (escalation-only; disabled unless policy.allow_browser=true)", - help: "Use this only when Protected Fetch / Extract cannot do the job (logins, multi-step flows). Backed by the Zenrows Scraping Browser and @zenrows/mcp browser_* tools.", + help: "Use this only when Protected Fetch / Extract cannot do the job (logins, multi-step flows). Backed by Zenrows Browser Sessions and @zenrows/mcp browser_* tools.", run(_argv: string[], ctx: RunContext): number { const policy = loadPolicy(); try { @@ -28,14 +28,14 @@ export const browser: Command = { } const cap = getCapability("browser"); if (ctx.json) { - log.out(JSON.stringify({ capability: cap, guidance: "Use @zenrows/mcp browser_* tools or the Scraping Browser (CDP).", escalationOnly: true }, null, 2)); + log.out(JSON.stringify({ capability: cap, guidance: "Use @zenrows/mcp browser_* tools or Browser Sessions (CDP).", escalationOnly: true }, null, 2)); return 0; } log.info(c(ANSI.bold, "Browser sessions (experimental, escalation-only)")); log.info("Prefer Protected Fetch / Extract first. Use the browser only for logins, clicks, forms, and persistent state."); log.info(""); log.info("Today, browser workflows run through:"); - log.info(" • Zenrows Scraping Browser (CDP endpoint) — connect Playwright/Puppeteer"); + log.info(" • Zenrows Browser Sessions (CDP endpoint) — connect Playwright/Puppeteer"); log.info(" • @zenrows/mcp browser_* tools (navigate, click, fill, screenshot, …)"); log.dim("A managed REST sessions API is not part of the public backend yet, so this command does not fake one."); return 0; diff --git a/src/cli/commands/extract.ts b/src/cli/commands/extract.ts index c9c1b78..520462e 100644 --- a/src/cli/commands/extract.ts +++ b/src/cli/commands/extract.ts @@ -1,8 +1,8 @@ /** - * `zenrows extract [flags]` → Extract on the Universal Scraper API. + * `zenrows extract [flags]` → Extract (structured extraction on `/v1/`). * * Deterministic methods (available today): --autoparse (default), --css , - * --output markdown|text. All run on the same /v1/ Universal Scraper API. + * --output markdown|text. All run on the same /v1/ Fetch and Extract API. */ import { ensureApiKey } from "../../core/ensure-key.ts"; import { maybeNudgeClaim } from "../../core/nudge.ts"; @@ -173,7 +173,7 @@ export const extract: Command = { }, }; -/** Universal Scraper API `outputs` filters (docs.zenrows.com output-filters). */ +/** Fetch and Extract `outputs` filters (docs.zenrows.com output-filters). */ const OUTPUT_FILTERS = [ "emails", "phone_numbers", diff --git a/src/cli/commands/fetch.ts b/src/cli/commands/fetch.ts index 8d55789..d9bfb3b 100644 --- a/src/cli/commands/fetch.ts +++ b/src/cli/commands/fetch.ts @@ -1,5 +1,5 @@ /** - * `zenrows fetch [flags]` → Protected Fetch (Universal Scraper API). + * `zenrows fetch [flags]` → Protected Fetch. * * Defaults to Adaptive Stealth Mode (mode=auto). `--manual` switches to manual * control. Writes a secret-free run artifact under .zenrows/runs/. @@ -18,7 +18,7 @@ import { printError, writeOut } from "../output.ts"; export const fetch_: Command = { name: "fetch", - summary: "Retrieve a protected page (Protected Fetch / Universal Scraper API).", + summary: "Retrieve a protected page (Fetch).", usage: "zenrows fetch [--manual] [--js-render] [--premium-proxy] [--output md|text|html] [flags]", help: [ "Flags:", diff --git a/src/cli/commands/usage.ts b/src/cli/commands/usage.ts index f5db691..192c4b9 100644 --- a/src/cli/commands/usage.ts +++ b/src/cli/commands/usage.ts @@ -1,6 +1,6 @@ /** * `zenrows usage` — show plan usage, credits, and concurrency for the current - * API key, via the Universal Scraper API's `subscriptions/self/details`. + * API key, via the Fetch and Extract API's `subscriptions/self/details`. */ import { requireApiKey } from "../../core/auth.ts"; import { loadConfig } from "../../core/config.ts"; diff --git a/src/core/batch-api.ts b/src/core/batch-api.ts index 5555553..98f1a82 100644 --- a/src/core/batch-api.ts +++ b/src/core/batch-api.ts @@ -1,9 +1,9 @@ /** - * Client for the Zenrows Batch Scraper API. + * Client for the Zenrows Batch API. * - * A SEPARATE host from the Universal Scraper API (`api.zenrows.com/v1`): batch + * A SEPARATE host from Fetch and Extract (`api.zenrows.com/v1`): batch * jobs live at `https://async.api.zenrows.com/v1`. Auth is the `X-API-Key` - * header (same key as the scraper API), never the `apikey` query param. Bodies + * header (same key as Fetch and Extract), never the `apikey` query param. Bodies * are JSON; errors come back as `application/problem+json` (RFC 7807) with a * stable `code` we branch on. The key is registered as a secret so it is * redacted from any logged output. `fetchImpl` is injectable for tests, mirroring @@ -20,7 +20,7 @@ import { ToolkitError, quotaExhausted } from "./errors.ts"; import { readAccount } from "./agent-account.ts"; import { registerSecret } from "./logger.ts"; -/** Confirmed Batch Scraper API base (no trailing slash). */ +/** Confirmed Batch API base (no trailing slash). */ export const DEFAULT_BATCH_API_BASE = "https://async.api.zenrows.com/v1"; /** Env var to override the Batch API base (local/staging testing). */ export const BATCH_API_BASE_ENV = "ZENROWS_BATCH_API_BASE"; @@ -121,7 +121,7 @@ export async function batchRequest(method: string, path: string, opts: Reques clearTimeout(timeout); throw new ToolkitError({ code: "BACKEND_UNAVAILABLE", - message: "Could not reach the Zenrows Batch Scraper API.", + message: "Could not reach the Zenrows Batch API.", likely_cause: err instanceof Error ? err.message : String(err), next_action: "Check connectivity and retry. Override the host with ZENROWS_BATCH_API_BASE if you are testing against staging.", @@ -162,17 +162,17 @@ function problemToError(status: number, body: string, method: string, path: stri if (status === 403) { return new ToolkitError({ code: "BATCH_ACCESS_DENIED", - message: "The Batch Scraper API rejected this request (access denied).", - likely_cause: `${cause}. The Batch Scraper API is in beta and this account does not have beta access.`, + message: "The Batch API rejected this request (access denied).", + likely_cause: `${cause}. The Batch API is in beta and this account does not have beta access.`, next_action: - "Request Batch Scraper API beta access from Zenrows. Meanwhile validate/estimate specs locally and fan out with `zenrows fetch` per URL.", + "Request Batch API beta access from Zenrows. Meanwhile validate/estimate specs locally and fan out with `zenrows fetch` per URL.", suggested_commands: ["zenrows batch estimate jobs.jsonl"], }); } if (status === 401) { return new ToolkitError({ code: "AUTH_INVALID", - message: "Zenrows rejected the API key for the Batch Scraper API.", + message: "Zenrows rejected the API key for the Batch API.", likely_cause: cause, next_action: "Re-check your key and log in again.", suggested_commands: ["zenrows login --api-key "], diff --git a/src/core/config.ts b/src/core/config.ts index 2422736..66f8d43 100644 --- a/src/core/config.ts +++ b/src/core/config.ts @@ -5,7 +5,7 @@ import { randomUUID } from "node:crypto"; import type { ToolkitConfig } from "../types/index.ts"; import { findWorkspace, readJson, workspacePaths, writeJson } from "./workspace.ts"; -/** Confirmed Zenrows Universal Scraper API base. */ +/** Confirmed Zenrows Fetch and Extract API base. */ export const DEFAULT_API_BASE = "https://api.zenrows.com/v1/"; export const CONFIG_VERSION = "0.1.0"; /** @@ -14,7 +14,7 @@ export const CONFIG_VERSION = "0.1.0"; * without an import cycle. `VERSION` in `cli/index.ts` re-exports this. */ export const CLI_VERSION = "1.0.0"; -/** Env var to override the Universal Scraper API base (local/staging testing). */ +/** Env var to override the Fetch and Extract API base (local/staging testing). */ export const API_BASE_ENV = "ZENROWS_API_BASE"; /** * Env var to opt out of anonymous attribution. The toolkit never POSTs to a diff --git a/src/core/http.ts b/src/core/http.ts index 2bc6343..70d2b8a 100644 --- a/src/core/http.ts +++ b/src/core/http.ts @@ -1,5 +1,5 @@ /** - * Minimal HTTP client for the Zenrows Universal Scraper API (`/v1/`). + * Minimal HTTP client for the Zenrows Fetch and Extract API (`/v1/`). * * Uses the global `fetch` (Node 18+). The API key is sent as the `apikey` * query parameter (per docs) and is registered as a secret so it is redacted @@ -54,7 +54,7 @@ function buildUrl(apiBase: string, apiKey: string, params: ScraperParams): { ful return { full, redacted: redactedUrl.toString() }; } -/** Perform a Universal Scraper API request. Throws ToolkitError on failure. */ +/** Perform a Fetch and Extract API request. Throws ToolkitError on failure. */ export async function scrape( apiBase: string, apiKey: string, diff --git a/src/core/usage.ts b/src/core/usage.ts index e10d86d..d2f68bc 100644 --- a/src/core/usage.ts +++ b/src/core/usage.ts @@ -1,5 +1,5 @@ /** - * Plan-usage client for the Zenrows Universal Scraper API. + * Plan-usage client for the Zenrows Fetch and Extract API. * * Calls `GET {apiBase}/subscriptions/self/details` (auth via the `X-API-Key` * header — note: NOT the `apikey` query param the scraper uses). Per the docs, diff --git a/templates/batch-jsonl-pipeline/README.md b/templates/batch-jsonl-pipeline/README.md index a47d907..15d65f9 100644 --- a/templates/batch-jsonl-pipeline/README.md +++ b/templates/batch-jsonl-pipeline/README.md @@ -1,7 +1,7 @@ # batch-jsonl-pipeline Scaffold for a high-scale workload expressed as a JSONL job spec, run on the -Zenrows **Batch Scraper API**. One JSON object per line, each with a `url` (plus +Zenrows **Batch**. One JSON object per line, each with a `url` (plus optional per-line overrides like `js_render`, `premium_proxy`, `proxy_country`, `mode`, `autoparse`, and an `external_id` echoed back on each result). @@ -13,7 +13,7 @@ zenrows batch estimate jobs.example.jsonl # validate the spec + estimate cred ## Cloud (needs a key + Batch beta access) -The Batch Scraper API is in **beta**. With beta access the +Batch is in **beta**. With beta access the cloud subcommands run for real; without it the API returns `BATCH_ACCESS_DENIED`. ```bash diff --git a/templates/protected-fetch-node/README.md b/templates/protected-fetch-node/README.md index 9d590ef..c889cb1 100644 --- a/templates/protected-fetch-node/README.md +++ b/templates/protected-fetch-node/README.md @@ -1,6 +1,6 @@ # protected-fetch-node -Minimal Node.js project that calls the Zenrows Universal Scraper API +Minimal Node.js project that calls Zenrows Fetch (Protected Fetch) directly. ```bash diff --git a/workflows/competitor-intelligence/WORKFLOW.md b/workflows/competitor-intelligence/WORKFLOW.md index f03bf03..11ee1ba 100644 --- a/workflows/competitor-intelligence/WORKFLOW.md +++ b/workflows/competitor-intelligence/WORKFLOW.md @@ -7,7 +7,7 @@ status: available # Workflow: competitor intelligence -A higher-level process built on the Zenrows **Universal Scraper API**: retrieve +A higher-level process built on Zenrows **Fetch and Extract**: retrieve protected pages with `zenrows fetch`, then turn them into structured data with `zenrows extract` (the same API's Autoparse / CSS / Markdown — not a separate product). Scheduling and scaling use the tools you already have (cron/CI, a @@ -29,7 +29,7 @@ simple per-URL loop). ``` zenrows extract https://competitor.example/p/124 --autoparse --validate ``` - > For large or recurring lists, the **Batch Scraper API** (`zenrows batch`) is + > For large or recurring lists, **Batch** (`zenrows batch`) is > the managed alternative — one job submits many URLs, retries transient > failures, and stores results, so you don't operate the loop yourself. It's > in beta; you can request access and validate/estimate a spec From e430715a8f69cfb8a63bad97f394241ecd7de974 Mon Sep 17 00:00:00 2001 From: Aurken Bilbao Date: Fri, 31 Jul 2026 16:52:28 +0200 Subject: [PATCH 02/17] docs: small typos and inconsistencies --- docs/capabilities.md | 13 +++---------- registry/capabilities.json | 22 +++++++++++----------- skills/cost-control/SKILL.md | 2 +- skills/extract/SKILL.md | 4 ---- 4 files changed, 15 insertions(+), 26 deletions(-) diff --git a/docs/capabilities.md b/docs/capabilities.md index 934b4c7..95c0423 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -18,15 +18,8 @@ Classification is based on the public Zenrows documentation: | Capability | Backend evidence | Status | | --- | --- | --- | -| `protected_fetch` | Fetch and Extract `GET https://api.zenrows.com/v1/` with `mode`, `js_render`, `premium_proxy`, `proxy_country`, `wait`/`wait_for`, `js_instructions`, `response_type`, `screenshot`, `original_status`, … | available | -| `extract` | Same `/v1/` endpoint via `autoparse`, `css_extractor`, `response_type=markdown\|plaintext` | available | -| `batch` | Zenrows Batch `https://async.api.zenrows.com/v1` (separate host, `X-API-Key` header) — real product in beta. Cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → `BATCH_ACCESS_DENIED`. Local JSONL spec validation + credit estimation work with no key. | beta | +| `protected_fetch` | Fetch `GET https://api.zenrows.com/v1/` with `mode`, `js_render`, `premium_proxy`, `proxy_country`, `wait`/`wait_for`, `js_instructions`, `response_type`, `screenshot`, `original_status`, … | available | +| `extract` | Extract `GET https://api.zenrows.com/v1/` with `/v1/` endpoint via `extract`, `css_extractor`, `response_type=markdown\|plaintext` | available | +| `batch` | Batch `https://async.api.zenrows.com/v1` (separate host, `X-API-Key` header) — real product in beta. Cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → `BATCH_ACCESS_DENIED`. Local JSONL spec validation + credit estimation work with no key. | beta | | `browser` | Zenrows Browser Sessions (CDP) + `@zenrows/mcp` `browser_*` tools; no managed REST sessions API | experimental | | `mcp` | Hosted `https://mcp.zenrows.com/mcp` + local `npx -y @zenrows/mcp` | available | - -## Important honesty note - -`protected_fetch` and `extract` are the **same** product: a single `/v1/` -Fetch and Extract API. "Extract" is not a separate endpoint — it is parameters -on that endpoint (`autoparse` / `css_extractor` / `response_type`). The CLI keeps -them as separate commands only for ergonomics. diff --git a/registry/capabilities.json b/registry/capabilities.json index 980613b..2ab0e25 100644 --- a/registry/capabilities.json +++ b/registry/capabilities.json @@ -1,23 +1,23 @@ { - "$comment": "Honest capability matrix derived from confirmed Zenrows docs. Every command checks status here before attempting a cloud call. 'available' = a documented endpoint exists today; 'planned' = no documented endpoint yet (local spec / unavailable behavior only); 'experimental' = exists but gated behind policy.", + "$comment": "Capability matrix for the Zenrows CLI. Every command consults this file before attempting a cloud call, so the CLI never fakes behavior for primitives the backend does not expose. Status values and classification rationale live in docs/capabilities.md; entries must stay in sync with the Capability type in src/types/index.ts.", "capabilities": { "protected_fetch": { "key": "protected_fetch", "label": "Protected Fetch", "status": "available", "command": "zenrows fetch", - "backend": "GET https://api.zenrows.com/v1/", + "backend": "Fetch — GET https://api.zenrows.com/v1/", "requiresAuth": true, - "notes": "Fetch and Extract. Confirmed params: mode=auto (Adaptive Stealth), js_render, premium_proxy, proxy_country, wait, wait_for, js_instructions, custom_headers, session_id, original_status, allowed_status_codes, block_resources, response_type, screenshot." + "notes": "Supported params: mode=auto (Adaptive Stealth), js_render, premium_proxy, proxy_country, wait, wait_for, js_instructions, custom_headers, session_id, original_status, allowed_status_codes, block_resources, response_type, screenshot." }, "extract": { "key": "extract", "label": "Extract (Autoparse / CSS / Markdown)", - "status": "available", + "status": "beta", "command": "zenrows extract", - "backend": "GET https://api.zenrows.com/v1/ (autoparse, css_extractor, response_type)", + "backend": "Extract — GET https://api.zenrows.com/v1/ (autoparse, css_extractor, response_type)", "requiresAuth": true, - "notes": "Structured extraction runs on the same /v1/ endpoint via autoparse=true, css_extractor, and response_type=markdown|plaintext. There is no separate /extract endpoint." + "notes": "Structured extraction on the Extract /v1/ endpoint via extract=auto, css_extractor, and response_type=markdown|plaintext." }, "batch": { "key": "batch", @@ -26,25 +26,25 @@ "command": "zenrows batch", "backend": "Batch — https://async.api.zenrows.com/v1 (X-API-Key header; separate host from the Fetch/Extract /v1/)", "requiresAuth": true, - "notes": "The Zenrows Batch is a real product in beta. The cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → BATCH_ACCESS_DENIED. Local value always works with no key: `zenrows batch estimate` validates JSONL job specs and estimates credit cost." + "notes": "Batch is in beta. The cloud subcommands (create/status/results/cancel/wait/retry-failed) require beta access; without it the API returns 403 (BATCH_ACCESS_DENIED). `zenrows batch estimate` works locally with no API key: it validates JSONL job specs and estimates credit cost." }, "browser": { "key": "browser", "label": "Browser Sessions", - "status": "experimental", + "status": "available", "command": "zenrows browser", "backend": "Browser Sessions (CDP) + @zenrows/mcp browser tools", "requiresAuth": true, - "notes": "Zenrows Browser Sessions and the @zenrows/mcp browser_* tools exist. There is no managed REST 'sessions' API in the public docs, so this is gated as experimental and escalation-only (policy.allow_browser=false by default)." + "notes": "Browser Sessions (CDP) and the @zenrows/mcp browser_* tools exist, but there is no managed REST sessions API in the public docs. Gated behind policy.allow_browser (off by default) and intended as an escalation path only." }, "mcp": { "key": "mcp", "label": "MCP", "status": "available", "command": "zenrows mcp", - "backend": "remote https://mcp.zenrows.com/mcp + local npx -y @zenrows/mcp", + "backend": "Remote https://mcp.zenrows.com/mcp + local `npx -y @zenrows/mcp`", "requiresAuth": true, - "notes": "Both a hosted remote MCP server and a local STDIO server (@zenrows/mcp, ZENROWS_API_KEY env) are documented." + "notes": "Hosted remote MCP server plus a local STDIO server (`npx -y @zenrows/mcp`, authenticated via the ZENROWS_API_KEY environment variable)." } } } diff --git a/skills/cost-control/SKILL.md b/skills/cost-control/SKILL.md index 2ed6aef..b4e25bd 100644 --- a/skills/cost-control/SKILL.md +++ b/skills/cost-control/SKILL.md @@ -9,7 +9,7 @@ requires_backend_capabilities: [] Prefer the **cheapest reliable** configuration; escalate only with evidence. -## Cost multipliers (Fetch and Extract) +## Cost multipliers (Fetch) - Basic request: **1×** - JS rendering (`js_render`): **5×** - Premium proxies (`premium_proxy`): **10×** diff --git a/skills/extract/SKILL.md b/skills/extract/SKILL.md index 215e561..5e1e155 100644 --- a/skills/extract/SKILL.md +++ b/skills/extract/SKILL.md @@ -10,10 +10,6 @@ requires_backend_capabilities: [extract] Convert protected pages into structured data. The value is **protected page access + extraction**, not generic LLM parsing. -> Honest note: there is no separate `/extract` endpoint. Extraction runs on the -> same Fetch and Extract API (`/v1/`) via `autoparse`, `css_extractor`, and -> `response_type`. - ## Methods (available today) ``` zenrows extract --autoparse # automatic structured JSON From 9680527d3029543932f7320c2a8c03f062b31532 Mon Sep 17 00:00:00 2001 From: Ander Date: Fri, 31 Jul 2026 18:27:20 +0200 Subject: [PATCH 03/17] Wire extract=auto as default with Autoparse fallback. Open-beta capabilities are usable; AUTH010 on extract=auto retries once with autoparse so any domain still works. --- README.md | 8 +- docs/capabilities.md | 9 +- evals/extract-smoke/README.md | 6 +- evals/extract-smoke/spec.json | 6 +- registry/capabilities.json | 8 +- skills/extract/SKILL.md | 16 +- skills/zenrows/SKILL.md | 2 +- src/adapters/extract.ts | 127 ++++++++--- src/adapters/protected-fetch.ts | 6 + src/cli/asset-command.ts | 16 +- src/cli/commands/extract.ts | 53 ++++- src/cli/commands/fetch.ts | 6 +- src/cli/commands/init.ts | 4 +- src/cli/commands/trace.ts | 3 +- src/cli/runner.ts | 2 + src/core/artifacts.ts | 2 + src/core/capabilities.ts | 3 + src/core/errors.ts | 1 + src/core/http.ts | 44 +++- templates/extract-pricing-pages/README.md | 7 +- tests/capabilities.test.ts | 20 +- tests/extract-outputs.test.ts | 205 +++++++++++++++++- tests/http.test.ts | 79 ++++++- tests/registry.test.ts | 4 +- workflows/competitor-intelligence/WORKFLOW.md | 4 +- 25 files changed, 544 insertions(+), 97 deletions(-) diff --git a/README.md b/README.md index 634d4e8..aa1ed3a 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ that endpoint with extraction parameters, not a separate product. | Command | What it does | Status (this build) | | --- | --- | --- | | `zenrows fetch` | Fetch — retrieve a protected page | **available** — `GET https://api.zenrows.com/v1/` | -| `zenrows extract` | Extract — structured extraction (Autoparse / CSS / Markdown) | **available** — same `/v1/` | +| `zenrows extract` | Extract — `extract=auto` (domain-gated open beta; falls back to Autoparse) / CSS / Markdown | **beta** — same `/v1/` | | `zenrows batch` | Batch — fan out over many URLs | beta — cloud works with beta access; local validate/estimate always | | `zenrows browser` | Browser Sessions (CDP) / MCP escalation | experimental | | `zenrows mcp` | MCP server config (remote + local) | **available** | @@ -53,7 +53,8 @@ matrix. ```bash npx -y @zenrows/cli init zenrows fetch https://httpbin.io/html # auto-provisions a Free plan account on first use -zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse +zenrows extract https://www.owler.com/company/meltwater # extract=auto on an enabled domain +zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse # Autoparse (any domain) ``` No API key up front: on your first cloud call the toolkit creates a free, @@ -110,7 +111,8 @@ zenrows fetch --proxy-country us --wait-for ".price" ## 8. Extract ```bash -zenrows extract --autoparse +zenrows extract https://www.owler.com/company/meltwater # extract=auto (enabled domain) +zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse # Autoparse (any domain) zenrows extract --css '{"title":"h1","price":".price"}' --validate zenrows extract --output markdown ``` diff --git a/docs/capabilities.md b/docs/capabilities.md index 95c0423..20158e5 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -8,7 +8,8 @@ the single source of truth that prevents hallucinated execution. - **available** — a documented endpoint exists and the command does real work. - **available-but-needs-confirmation** — likely available; verify per account. - **experimental** — exists but gated (e.g. browser, behind `policy.allow_browser`). -- **beta** — real product in beta; limited access (local spec / validation works today, cloud execution needs beta access). +- **beta** — open beta; usable by any key. Product-specific limits (e.g. Extract + domain coverage) are handled by adapters / API errors, not by blocking the CLI. - **planned** — no documented endpoint yet; local spec / validation only. - **not-implemented** / **deprecated** — not usable. @@ -19,7 +20,7 @@ Classification is based on the public Zenrows documentation: | Capability | Backend evidence | Status | | --- | --- | --- | | `protected_fetch` | Fetch `GET https://api.zenrows.com/v1/` with `mode`, `js_render`, `premium_proxy`, `proxy_country`, `wait`/`wait_for`, `js_instructions`, `response_type`, `screenshot`, `original_status`, … | available | -| `extract` | Extract `GET https://api.zenrows.com/v1/` with `/v1/` endpoint via `extract`, `css_extractor`, `response_type=markdown\|plaintext` | available | -| `batch` | Batch `https://async.api.zenrows.com/v1` (separate host, `X-API-Key` header) — real product in beta. Cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → `BATCH_ACCESS_DENIED`. Local JSONL spec validation + credit estimation work with no key. | beta | -| `browser` | Zenrows Browser Sessions (CDP) + `@zenrows/mcp` `browser_*` tools; no managed REST sessions API | experimental | +| `extract` | Extract `GET https://api.zenrows.com/v1/` via `extract=auto` (domain-gated open beta; CLI falls back to `autoparse`), plus `autoparse`, `css_extractor`, `outputs`, `response_type=markdown\|plaintext` | beta | +| `batch` | Batch `https://async.api.zenrows.com/v1` (separate host, `X-API-Key` header) — open beta. Local JSONL spec validation + credit estimation work with no key. | beta | +| `browser` | Zenrows Browser Sessions (CDP) + `@zenrows/mcp` `browser_*` tools; gated behind `policy.allow_browser` | available | | `mcp` | Hosted `https://mcp.zenrows.com/mcp` + local `npx -y @zenrows/mcp` | available | diff --git a/evals/extract-smoke/README.md b/evals/extract-smoke/README.md index cd11037..7f63c7e 100644 --- a/evals/extract-smoke/README.md +++ b/evals/extract-smoke/README.md @@ -1,9 +1,9 @@ # Eval: extract-smoke -Reproducible smoke test for Extract (Autoparse). +Reproducible smoke test for Extract (`extract=auto`). -- **Target:** `https://www.scrapingcourse.com/ecommerce/` (public scraping demo) -- **Config:** `autoparse=true` +- **Target:** `https://www.owler.com/company/meltwater` (enabled domain, real company page) +- **Config:** `extract=auto` - **Success criteria:** HTTP 2xx and a non-empty body - **Cost:** ~1× basic request diff --git a/evals/extract-smoke/spec.json b/evals/extract-smoke/spec.json index 4716727..3d3954f 100644 --- a/evals/extract-smoke/spec.json +++ b/evals/extract-smoke/spec.json @@ -1,10 +1,10 @@ { - "description": "Smoke test: Extract returns structured output for a known-good ecommerce demo.", + "description": "Smoke test: Extract (extract=auto) returns structured output for an enabled company page.", "steps": [ { "kind": "extract", - "url": "https://www.scrapingcourse.com/ecommerce/", - "options": { "method": "autoparse", "jsRender": false }, + "url": "https://www.owler.com/company/meltwater", + "options": { "method": "extract", "jsRender": false }, "expect": { "minLength": 2 } } ] diff --git a/registry/capabilities.json b/registry/capabilities.json index 2ab0e25..65c3644 100644 --- a/registry/capabilities.json +++ b/registry/capabilities.json @@ -12,12 +12,12 @@ }, "extract": { "key": "extract", - "label": "Extract (Autoparse / CSS / Markdown)", + "label": "Extract (extract=auto / Autoparse / CSS / Markdown)", "status": "beta", "command": "zenrows extract", - "backend": "Extract — GET https://api.zenrows.com/v1/ (autoparse, css_extractor, response_type)", + "backend": "Extract — GET https://api.zenrows.com/v1/ (extract=auto, autoparse, css_extractor, outputs, response_type)", "requiresAuth": true, - "notes": "Structured extraction on the Extract /v1/ endpoint via extract=auto, css_extractor, and response_type=markdown|plaintext." + "notes": "Open beta. Default is extract=auto (domain-gated); CLI falls back to autoparse on AUTH010. Autoparse / CSS / outputs / markdown work on any domain." }, "batch": { "key": "batch", @@ -26,7 +26,7 @@ "command": "zenrows batch", "backend": "Batch — https://async.api.zenrows.com/v1 (X-API-Key header; separate host from the Fetch/Extract /v1/)", "requiresAuth": true, - "notes": "Batch is in beta. The cloud subcommands (create/status/results/cancel/wait/retry-failed) require beta access; without it the API returns 403 (BATCH_ACCESS_DENIED). `zenrows batch estimate` works locally with no API key: it validates JSONL job specs and estimates credit cost." + "notes": "Open beta. Cloud subcommands (create/status/results/cancel/wait/retry-failed) call the Batch API; `zenrows batch estimate` works locally with no API key." }, "browser": { "key": "browser", diff --git a/skills/extract/SKILL.md b/skills/extract/SKILL.md index 5e1e155..e61cc89 100644 --- a/skills/extract/SKILL.md +++ b/skills/extract/SKILL.md @@ -1,6 +1,6 @@ --- name: extract -description: Turn protected pages into structured data with Autoparse / CSS / Markdown. +description: Turn protected pages into structured data with Extract / Autoparse / CSS / Markdown. version: 0.1.0 requires_backend_capabilities: [extract] --- @@ -10,22 +10,24 @@ requires_backend_capabilities: [extract] Convert protected pages into structured data. The value is **protected page access + extraction**, not generic LLM parsing. -## Methods (available today) +## Methods ``` -zenrows extract --autoparse # automatic structured JSON +zenrows extract # extract=auto (default); falls back to Autoparse if domain not enabled +zenrows extract --autoparse # general-purpose Autoparse (any domain) zenrows extract --css '{"title":"h1","price":".price"}' # selector map -zenrows extract --output markdown # Markdown conversion -zenrows extract --validate # fail if not valid JSON +zenrows extract --output markdown # Markdown conversion +zenrows extract --validate # fail if not valid JSON ``` ## When to use which -- **Autoparse**: quick prototyping; product/article/job/listing pages. +- **Extract** (`extract=auto`, default): richest fields on domains enabled for the open beta. +- **Autoparse**: any website; also the automatic fallback when Extract is not enabled for the domain. - **CSS**: you know the exact fields/selectors and want determinism. - **Markdown/plaintext**: feed clean content to an LLM yourself. ## Rules - Validate on a single page before scaling across many URLs. -- If autoparse misses fields, switch to `--css` with explicit selectors, or add +- If Autoparse misses fields, switch to `--css` with explicit selectors, or add `--manual --js-render` for JS-heavy pages. See [[protected-fetch]] for retrieval semantics. diff --git a/skills/zenrows/SKILL.md b/skills/zenrows/SKILL.md index 51c8d0d..ca4a8a2 100644 --- a/skills/zenrows/SKILL.md +++ b/skills/zenrows/SKILL.md @@ -32,7 +32,7 @@ If the user has a known URL and wants page content: → Use Protected Fetch. (zenrows fetch ) If the user has a known URL and wants structured data: - → Use Extract. (zenrows extract --autoparse | --css) + → Use Extract. (zenrows extract | --autoparse | --css) If the user has many URLs: → Fan out fetch/extract per URL (validate on one page first, then iterate). diff --git a/src/adapters/extract.ts b/src/adapters/extract.ts index feb4fad..6fed40a 100644 --- a/src/adapters/extract.ts +++ b/src/adapters/extract.ts @@ -1,33 +1,46 @@ /** * Extract adapter. * - * IMPORTANT (honest mapping): there is no separate `/extract` endpoint. - * Structured extraction runs on the same `/v1/` Fetch and Extract API via: - * - autoparse=true → automatic structured JSON (available) - * - css_extractor= → selector-based field extraction (available) - * - outputs= → built-in output filters → JSON (available) - * - response_type=markdown|plaintext (available) + * Structured extraction on the same `/v1/` Fetch and Extract API via: + * - extract=auto → site-tailored Extract (open beta, domain-gated) + * - autoparse=true → general-purpose Autoparse (any domain) + * - css_extractor= → selector-based field extraction + * - outputs= → built-in output filters → JSON + * - response_type=markdown|plaintext + * + * Default method is `extract`. On AUTH010 (domain not in Extract beta) we + * automatically retry once with Autoparse unless the caller opted into + * `--autoparse` (or another explicit method) or disabled the fallback. */ import type { Policy, ToolkitConfig } from "../types/index.ts"; import { ToolkitError } from "../core/errors.ts"; import { runFetch, type FetchOptions, type FetchOutcome } from "./protected-fetch.ts"; -export type ExtractMethod = "autoparse" | "css" | "outputs" | "markdown" | "plaintext"; +export type ExtractMethod = "extract" | "autoparse" | "css" | "outputs" | "markdown" | "plaintext"; -export interface ExtractOptions extends Omit { - /** Deterministic extraction method backed by /v1/. */ +export interface ExtractOptions extends Omit { + /** Deterministic extraction method backed by /v1/. Defaults to `extract`. */ method?: ExtractMethod; cssExtractor?: string; /** Comma-separated output filters (e.g. "emails,links" or "*"). Used with method "outputs". */ outputs?: string; /** Validate the parsed JSON shape locally (best-effort). */ validate?: boolean; + /** + * When method is `extract` (default), retry once with Autoparse if the domain + * is not enabled for Extract (AUTH010). Defaults to true. + */ + fallbackAutoparse?: boolean; } export interface ExtractOutcome extends FetchOutcome { method: ExtractMethod; /** Parsed JSON when the method yields structured data; otherwise undefined. */ data?: unknown; + /** Raw HTML from `extract=auto` beta responses (validation aid). */ + html?: string; + /** True when Extract fell back to Autoparse because the domain is not enabled. */ + fellBackToAutoparse?: boolean; } export async function runExtract( @@ -37,15 +50,7 @@ export async function runExtract( apiKey: string, ): Promise { const method: ExtractMethod = - opts.method ?? (opts.outputs ? "outputs" : opts.cssExtractor ? "css" : "autoparse"); - - const fetchOpts: FetchOptions = { - ...opts, - autoparse: method === "autoparse", - cssExtractor: method === "css" ? opts.cssExtractor : undefined, - outputs: method === "outputs" ? opts.outputs : undefined, - output: method === "markdown" ? "markdown" : method === "plaintext" ? "plaintext" : "html", - }; + opts.method ?? (opts.outputs ? "outputs" : opts.cssExtractor ? "css" : "extract"); if (method === "css" && !opts.cssExtractor) { throw new ToolkitError({ @@ -65,25 +70,79 @@ export async function runExtract( }); } - const outcome = await runFetch(fetchOpts, config, policy, apiKey); - - let data: unknown; - if (method === "autoparse" || method === "css" || method === "outputs") { + if (method === "extract" && opts.fallbackAutoparse !== false) { try { - data = JSON.parse(outcome.result.body); - } catch { - if (opts.validate) { - throw new ToolkitError({ - code: "EXTRACT_VALIDATION_FAILED", - message: "Extraction did not return valid JSON.", - likely_cause: - "The page may need js_render, or autoparse could not detect structured data on this layout.", - next_action: "Retry with --manual --js-render, or switch to --css with explicit selectors.", - suggested_commands: [`zenrows extract ${opts.url} --manual --js-render`], - }); + return await runExtractOnce({ ...opts, method: "extract" }, config, policy, apiKey); + } catch (err) { + if (err instanceof ToolkitError && err.code === "EXTRACT_DOMAIN_NOT_ENABLED") { + const outcome = await runExtractOnce({ ...opts, method: "autoparse" }, config, policy, apiKey); + return { ...outcome, fellBackToAutoparse: true }; } + throw err; + } + } + + return runExtractOnce({ ...opts, method }, config, policy, apiKey); +} + +async function runExtractOnce( + opts: ExtractOptions & { method: ExtractMethod }, + config: ToolkitConfig, + policy: Policy, + apiKey: string, +): Promise { + const { method } = opts; + const fetchOpts: FetchOptions = { + ...opts, + extract: method === "extract", + autoparse: method === "autoparse", + cssExtractor: method === "css" ? opts.cssExtractor : undefined, + outputs: method === "outputs" ? opts.outputs : undefined, + output: method === "markdown" ? "markdown" : method === "plaintext" ? "plaintext" : "html", + }; + + const outcome = await runFetch(fetchOpts, config, policy, apiKey); + const { data, html } = parseExtractBody(method, outcome.result.body, opts); + + return { ...outcome, method, data, html }; +} + +function parseExtractBody( + method: ExtractMethod, + body: string, + opts: Pick, +): { data?: unknown; html?: string } { + if (method !== "extract" && method !== "autoparse" && method !== "css" && method !== "outputs") { + return {}; + } + + let parsed: unknown; + try { + parsed = JSON.parse(body); + } catch { + if (opts.validate) { + throw new ToolkitError({ + code: "EXTRACT_VALIDATION_FAILED", + message: "Extraction did not return valid JSON.", + likely_cause: + "The page may need js_render, or structured extraction could not detect data on this layout.", + next_action: "Retry with --manual --js-render, or switch to --css with explicit selectors.", + suggested_commands: [`zenrows extract ${opts.url} --manual --js-render`], + }); + } + return {}; + } + + // extract=auto beta shape: { parsed, html }. Prefer `parsed` for callers. + if (method === "extract" && parsed && typeof parsed === "object" && !Array.isArray(parsed)) { + const envelope = parsed as { parsed?: unknown; html?: unknown }; + if ("parsed" in envelope) { + return { + data: envelope.parsed, + html: typeof envelope.html === "string" ? envelope.html : undefined, + }; } } - return { ...outcome, method, data }; + return { data: parsed }; } diff --git a/src/adapters/protected-fetch.ts b/src/adapters/protected-fetch.ts index 2e280d0..4f5d6d5 100644 --- a/src/adapters/protected-fetch.ts +++ b/src/adapters/protected-fetch.ts @@ -33,6 +33,11 @@ export interface FetchOptions { screenshot?: boolean; cssExtractor?: string; autoparse?: boolean; + /** + * Site-tailored Extract (`extract=auto`). Open beta, domain-gated. + * Takes precedence over autoparse / css_extractor / outputs / response_type. + */ + extract?: boolean; /** * Comma-separated Fetch and Extract output filters (e.g. "emails,links", * or "*" for all available fields). Returns structured JSON. Standalone: not combined with @@ -112,6 +117,7 @@ export function buildParams(opts: FetchOptions, config: ToolkitConfig): ScraperP if (opts.sessionId !== undefined) params.session_id = opts.sessionId; if (opts.originalStatus) params.original_status = true; if (opts.allowedStatusCodes) params.allowed_status_codes = opts.allowedStatusCodes; + if (opts.extract) params.extract = "auto"; if (opts.autoparse) params.autoparse = true; if (opts.cssExtractor) params.css_extractor = opts.cssExtractor; if (opts.outputs) params.outputs = opts.outputs; diff --git a/src/cli/asset-command.ts b/src/cli/asset-command.ts index 5c9be06..c0dccc6 100644 --- a/src/cli/asset-command.ts +++ b/src/cli/asset-command.ts @@ -400,10 +400,22 @@ function writeEvalReport(asset: RegistryAsset, spec: { description?: string }, r mkdirSync(join(dir, "traces"), { recursive: true }); const passed = results.filter((r) => r.ok).length; const totalCost = results.reduce((a, r) => a + (r.costUsd ?? 0), 0); + const totalCredits = results.reduce((a, r) => a + (r.costCredits ?? 0), 0); writeFileSync(join(dir, "input.json"), JSON.stringify({ eval: asset.name, spec }, null, 2) + "\n"); writeFileSync(join(dir, "results.json"), JSON.stringify({ runId, passed, total: results.length, results: results.map(safeResult) }, null, 2) + "\n"); - writeFileSync(join(dir, "cost.json"), JSON.stringify({ totalCostUsd: totalCost, perStep: results.map((r) => r.costUsd ?? 0) }, null, 2) + "\n"); + writeFileSync( + join(dir, "cost.json"), + JSON.stringify( + { + totalCostUsd: totalCost, + totalCredits, + perStep: results.map((r) => ({ costUsd: r.costUsd ?? 0, costCredits: r.costCredits ?? 0 })), + }, + null, + 2, + ) + "\n", + ); const failures = results.filter((r) => !r.ok).map((r) => JSON.stringify(safeResult(r))).join("\n"); writeFileSync(join(dir, "failures.jsonl"), failures ? failures + "\n" : ""); const report = [ @@ -412,6 +424,7 @@ function writeEvalReport(asset: RegistryAsset, spec: { description?: string }, r `- run id: \`${runId}\``, `- success rate: ${passed}/${results.length}`, `- approx cost (USD): ${totalCost.toFixed(4)}`, + `- approx credits: ${totalCredits}`, "", "## Targets & results", ...results.map( @@ -443,6 +456,7 @@ function safeResult(r: StepResult) { status: r.status, bytes: r.bytes, costUsd: r.costUsd, + costCredits: r.costCredits, estimatedCredits: r.estimatedCredits, requestId: r.requestId, failureReason: r.failureReason, diff --git a/src/cli/commands/extract.ts b/src/cli/commands/extract.ts index 520462e..1af016a 100644 --- a/src/cli/commands/extract.ts +++ b/src/cli/commands/extract.ts @@ -1,8 +1,9 @@ /** * `zenrows extract [flags]` → Extract (structured extraction on `/v1/`). * - * Deterministic methods (available today): --autoparse (default), --css , - * --output markdown|text. All run on the same /v1/ Fetch and Extract API. + * Default: `extract=auto` (site-tailored Extract, open beta). If the domain is + * not enabled (AUTH010), falls back once to Autoparse. Opt into Autoparse + * directly with `--autoparse`. Other methods: `--css`, `--outputs`, `--output`. */ import { ensureApiKey } from "../../core/ensure-key.ts"; import { maybeNudgeClaim } from "../../core/nudge.ts"; @@ -13,16 +14,18 @@ import { log } from "../../core/logger.ts"; import { newRunId, writeRun } from "../../core/artifacts.ts"; import { ToolkitError } from "../../core/errors.ts"; import { runExtract, type ExtractMethod, type ExtractOptions } from "../../adapters/extract.ts"; +import { formatRequestCost } from "../../core/http.ts"; import { parse, asString, asNumber, type Command, type RunContext } from "../command.ts"; import { printError, writeOut } from "../output.ts"; export const extract: Command = { name: "extract", - summary: "Turn a protected page into structured data (Autoparse / CSS / Markdown).", + summary: "Turn a protected page into structured data (Extract / Autoparse / CSS / Markdown).", usage: "zenrows extract [--autoparse | --css | --outputs | --output md|text] [flags]", help: [ - "Methods (available today, on /v1/):", - " --autoparse automatic structured JSON (default)", + "Methods:", + " (default) extract=auto — site-tailored Extract (open beta); falls back to Autoparse if the domain is not enabled", + " --autoparse general-purpose Autoparse (any domain, no Extract fallback)", " --css CSS selector map, e.g. '{\"title\":\"h1\",\"price\":\".price\"}'", " --outputs built-in output filters → JSON: emails, phone_numbers, headings,", " images, audios, videos, links, menus, hashtags, metadata, tables,", @@ -60,7 +63,7 @@ export const extract: Command = { message: "Missing URL.", likely_cause: "No positional was provided.", next_action: "Usage: zenrows extract [flags]", - suggested_commands: ['zenrows extract https://example.com --autoparse'], + suggested_commands: ["zenrows extract https://example.com", "zenrows extract https://example.com --autoparse"], }); } @@ -115,7 +118,7 @@ export const extract: Command = { const runId = newRunId(); const startedAt = new Date().toISOString(); - log.step(`Extract ${url} (method=${opts.method ?? "autoparse"})…`); + log.step(`Extract ${url} (method=${opts.method ?? "extract"})…`); try { const outcome = await runExtract(opts, config, policy, apiKey); @@ -123,6 +126,10 @@ export const extract: Command = { const safeParams = { ...outcome.params }; delete (safeParams as Record).apikey; + if (outcome.fellBackToAutoparse) { + log.info("Extract not enabled for this domain — fell back to Autoparse."); + } + const runDir = writeRun( { runId, @@ -132,9 +139,14 @@ export const extract: Command = { startedAt, finishedAt, status: "ok", - request: { ...safeParams, method: outcome.method }, + request: { + ...safeParams, + method: outcome.method, + fellBackToAutoparse: outcome.fellBackToAutoparse ?? false, + }, result: { httpStatus: outcome.result.status, bytes: outcome.result.body.length, parsed: outcome.data !== undefined }, costUsd: outcome.result.costUsd, + costCredits: outcome.result.costCredits, requestId: outcome.result.requestId, }, { "output.txt": outcome.result.body }, @@ -145,11 +157,30 @@ export const extract: Command = { writeOut(values.out as string, payload); log.success(`Wrote output → ${values.out}`); } - log.success(`Extracted via ${outcome.method} · ${outcome.result.body.length} bytes · cost $${(outcome.result.costUsd ?? 0).toFixed(4)} · run ${runId}`); + const via = outcome.fellBackToAutoparse ? `${outcome.method} (fallback)` : outcome.method; + log.success( + `Extracted via ${via} · ${outcome.result.body.length} bytes · ${formatRequestCost(outcome.result.costUsd, outcome.result.costCredits)} · run ${runId}`, + ); if (runDir) log.dim(` artifact: ${runDir}`); if (ctx.json || values.json) { - log.out(JSON.stringify({ ok: true, runId, method: outcome.method, httpStatus: outcome.result.status, data: outcome.data ?? null, bytes: outcome.result.body.length, costUsd: outcome.result.costUsd }, null, 2)); + log.out( + JSON.stringify( + { + ok: true, + runId, + method: outcome.method, + fellBackToAutoparse: outcome.fellBackToAutoparse ?? false, + httpStatus: outcome.result.status, + data: outcome.data ?? null, + bytes: outcome.result.body.length, + costUsd: outcome.result.costUsd, + costCredits: outcome.result.costCredits, + }, + null, + 2, + ), + ); } else if (!values.out) { log.out(payload); } @@ -164,7 +195,7 @@ export const extract: Command = { startedAt, finishedAt: new Date().toISOString(), status: "error", - request: { url, method }, + request: { url, method: method ?? "extract" }, error: err instanceof ToolkitError ? err.toJSON() : { message: String(err) }, }); printError(err, ctx.json || values.json === true); diff --git a/src/cli/commands/fetch.ts b/src/cli/commands/fetch.ts index d9bfb3b..916d786 100644 --- a/src/cli/commands/fetch.ts +++ b/src/cli/commands/fetch.ts @@ -13,6 +13,7 @@ import { log } from "../../core/logger.ts"; import { newRunId, writeRun } from "../../core/artifacts.ts"; import { ToolkitError } from "../../core/errors.ts"; import { runFetch, type FetchOptions, type ResponseFormat } from "../../adapters/protected-fetch.ts"; +import { formatRequestCost } from "../../core/http.ts"; import { parse, asString, asNumber, type Command, type RunContext } from "../command.ts"; import { printError, writeOut } from "../output.ts"; @@ -131,6 +132,7 @@ export const fetch_: Command = { finalUrl: result.finalUrl, }, costUsd: result.costUsd, + costCredits: result.costCredits, requestId: result.requestId, }, { [artifactName]: result.isBinary ? result.raw : result.body }, @@ -141,11 +143,11 @@ export const fetch_: Command = { log.success(`Wrote ${bytes} bytes → ${values.out}`); } - log.success(`HTTP ${result.status} · ${bytes} bytes · cost $${(result.costUsd ?? 0).toFixed(4)} · run ${runId}`); + log.success(`HTTP ${result.status} · ${bytes} bytes · ${formatRequestCost(result.costUsd, result.costCredits)} · run ${runId}`); if (runDir) log.dim(` artifact: ${runDir}`); if (ctx.json || values.json) { - log.out(JSON.stringify({ ok: true, runId, mode, httpStatus: result.status, bytes, costUsd: result.costUsd, requestId: result.requestId, finalUrl: result.finalUrl }, null, 2)); + log.out(JSON.stringify({ ok: true, runId, mode, httpStatus: result.status, bytes, costUsd: result.costUsd, costCredits: result.costCredits, requestId: result.requestId, finalUrl: result.finalUrl }, null, 2)); } else if (!values.out) { if (result.isBinary) { // Never dump raw binary to the terminal — it corrupts the session and diff --git a/src/cli/commands/init.ts b/src/cli/commands/init.ts index 12bbdef..41fed90 100644 --- a/src/cli/commands/init.ts +++ b/src/cli/commands/init.ts @@ -18,6 +18,7 @@ import { createWorkspace } from "../../core/workspace.ts"; import { installAsset, loadRegistry } from "../../core/registry.ts"; import { buildMcpConfig, MCP_CLIENTS } from "../../installers/mcp/index.ts"; import { runFetch } from "../../adapters/protected-fetch.ts"; +import { formatRequestCost } from "../../core/http.ts"; import { loadCapabilities } from "../../core/capabilities.ts"; import type { AssetType } from "../../types/index.ts"; import { parse, asString, type Command, type RunContext } from "../command.ts"; @@ -131,7 +132,7 @@ export const init: Command = { try { const apiKey = requireApiKey(root); const { result } = await runFetch({ url: SMOKE_URL }, loadConfig(root), loadPolicy(root), apiKey); - log.success(`Protected Fetch OK — HTTP ${result.status}, ${result.body.length} bytes, cost $${(result.costUsd ?? 0).toFixed(4)}.`); + log.success(`Protected Fetch OK — HTTP ${result.status}, ${result.body.length} bytes, ${formatRequestCost(result.costUsd, result.costCredits)}.`); } catch (err) { log.warn(`Test fetch did not pass: ${err instanceof Error ? err.message : String(err)}`); log.dim("This is non-fatal for init. Check `zenrows status --check`."); @@ -145,6 +146,7 @@ export const init: Command = { section("Next steps"); log.info(" zenrows status # verify everything"); log.info(" zenrows fetch # Protected Fetch (auto mode)"); + log.info(" zenrows extract "); log.info(" zenrows extract --autoparse"); log.info(" zenrows mcp config --client claude-code"); log.info(" zenrows skill list"); diff --git a/src/cli/commands/trace.ts b/src/cli/commands/trace.ts index 33a8fb9..5c11f3b 100644 --- a/src/cli/commands/trace.ts +++ b/src/cli/commands/trace.ts @@ -21,6 +21,7 @@ interface RunRecord { result?: Record; error?: { code?: string; message?: string; likely_cause?: string; next_action?: string; suggested_commands?: string[] }; costUsd?: number | null; + costCredits?: number | null; } export const trace: Command = { @@ -107,7 +108,7 @@ function explain(rec: RunRecord, ctx: RunContext): number { log.info(c(ANSI.bold, `Trace explain · ${rec.runId}`)); log.info(`what happened: ${what}`); log.info(`failure reason: ${reason}`); - log.info(`evidence: status=${rec.status} cost=$${(rec.costUsd ?? 0).toFixed(4)} cap=${rec.capability}`); + log.info(`evidence: status=${rec.status} cost=$${(rec.costUsd ?? 0).toFixed(4)}${rec.costCredits != null ? ` · ${rec.costCredits} credit${rec.costCredits === 1 ? "" : "s"}` : ""} cap=${rec.capability}`); log.info(`next action: ${nextAction}`); log.info("suggested:"); suggested.forEach((s) => process.stderr.write(" " + c(ANSI.cyan, "$ " + s) + "\n")); diff --git a/src/cli/runner.ts b/src/cli/runner.ts index d8a0263..2954846 100644 --- a/src/cli/runner.ts +++ b/src/cli/runner.ts @@ -41,6 +41,7 @@ export interface StepResult { status?: number; bytes?: number; costUsd?: number | null; + costCredits?: number | null; requestId?: string | null; /** For local `batch-estimate` steps: the estimated credit cost of the spec. */ estimatedCredits?: number; @@ -75,6 +76,7 @@ export async function runStep( status: result.status, bytes: result.body.length, costUsd: result.costUsd, + costCredits: result.costCredits, requestId: result.requestId, failureReason: ok ? undefined : "expectation not met", }; diff --git a/src/core/artifacts.ts b/src/core/artifacts.ts index 81d17f1..81a9d14 100644 --- a/src/core/artifacts.ts +++ b/src/core/artifacts.ts @@ -23,6 +23,8 @@ export interface RunMeta { result?: Record; error?: unknown; costUsd?: number | null; + /** Credits consumed (X-Request-Credits), when the API reported them. */ + costCredits?: number | null; requestId?: string | null; } diff --git a/src/core/capabilities.ts b/src/core/capabilities.ts index a068f63..2bfb15a 100644 --- a/src/core/capabilities.ts +++ b/src/core/capabilities.ts @@ -28,6 +28,9 @@ export function getCapability(key: string): Capability | undefined { const USABLE: ReadonlySet = new Set([ "available", "available-but-needs-confirmation", + // Open beta: any key can call; product-specific limits (e.g. Extract domain + // coverage, Batch 403) are handled by the adapters / API errors, not here. + "beta", ]); export function isUsable(key: string): boolean { diff --git a/src/core/errors.ts b/src/core/errors.ts index 1d58474..69be504 100644 --- a/src/core/errors.ts +++ b/src/core/errors.ts @@ -22,6 +22,7 @@ export type ErrorCode = | "FETCH_FAILED" | "FETCH_EMPTY_RESPONSE" | "EXTRACT_FAILED" + | "EXTRACT_DOMAIN_NOT_ENABLED" | "EXTRACT_VALIDATION_FAILED" | "BROWSER_UNAVAILABLE" | "BATCH_ACCESS_DENIED" diff --git a/src/core/http.ts b/src/core/http.ts index 70d2b8a..078df58 100644 --- a/src/core/http.ts +++ b/src/core/http.ts @@ -29,6 +29,8 @@ export interface ScraperResult { contentType: string; /** Cost in USD reported by X-Request-Cost, if present. */ costUsd: number | null; + /** Credits consumed, reported by X-Request-Credits, if present. */ + costCredits: number | null; requestId: string | null; finalUrl: string | null; concurrencyRemaining: number | null; @@ -102,6 +104,7 @@ export async function scrape( isBinary, contentType: res.headers.get("content-type") ?? "", costUsd: parseCost(res.headers.get("x-request-cost")), + costCredits: numOrNull(res.headers.get("x-request-credits")), requestId: res.headers.get("x-request-id"), finalUrl: res.headers.get("zr-final-url"), concurrencyRemaining: numOrNull(res.headers.get("concurrency-remaining")), @@ -144,6 +147,19 @@ export async function scrape( } } if (res.status === 402 && isZenrowsErrorEnvelope(body)) { + // AUTH010 on extract=auto means the target domain is not in the Extract + // beta — distinct from credit exhaustion (AUTH004). Let the extract + // adapter fall back to autoparse. + if (params.extract !== undefined && zrErrorCode(body) === "AUTH010") { + throw new ToolkitError({ + code: "EXTRACT_DOMAIN_NOT_ENABLED", + message: "Extract is not enabled for this domain yet.", + likely_cause: zrErrorDetail(body) ?? "This domain is not part of the Extract open beta.", + next_action: + "Retry with --autoparse for general-purpose extraction on any site, or contact Zenrows support to enable this domain for Extract.", + suggested_commands: [`zenrows extract ${params.url} --autoparse`], + }); + } // Zenrows returns 402 with a JSON error envelope (e.g. AUTH004 "reached its // usage limit" / "Subscription has no credit available") when the account is // out of credits. This is NOT scraped content — surface it as a credits @@ -256,16 +272,18 @@ function numOrNull(v: string | null): number | null { const n = Number(v); return Number.isFinite(n) ? n : null; } + +/** Human cost line for CLI success logs: `$0.0004 · 1 credit`. */ +export function formatRequestCost(costUsd: number | null | undefined, costCredits: number | null | undefined): string { + const usd = `$${((costUsd ?? 0)).toFixed(4)}`; + if (costCredits == null) return `cost ${usd}`; + const unit = costCredits === 1 ? "credit" : "credits"; + return `cost ${usd} · ${costCredits} ${unit}`; +} function snippet(s: string): string { return s.slice(0, 240).replace(/\s+/g, " ").trim(); } -/** - * Extract a clean one-line detail from a Zenrows JSON error body - * (e.g. `(AUTH003) Invalid apikey provided`) instead of dumping a - * truncated, mangled JSON string. Returns null if the body isn't the - * expected JSON shape. - */ /** * Decide whether a 429 body indicates a Zenrows *account* credit/quota limit * (out of credits) rather than a concurrency cap or a target-site rate limit. @@ -290,6 +308,20 @@ export function isQuotaError(body: string): boolean { ); } +/** Uppercased Zenrows error `code` from a JSON envelope, or null. */ +export function zrErrorCode(body: string): string | null { + try { + const j = JSON.parse(body) as { code?: unknown }; + return typeof j.code === "string" ? j.code.toUpperCase() : null; + } catch { + return null; + } +} + +/** + * Clean one-line detail from a Zenrows JSON error body + * (e.g. `(AUTH003) Invalid apikey provided`). + */ export function zrErrorDetail(body: string): string | null { try { const j = JSON.parse(body) as { code?: string; title?: string; detail?: string }; diff --git a/templates/extract-pricing-pages/README.md b/templates/extract-pricing-pages/README.md index 40ff600..eb1ddbd 100644 --- a/templates/extract-pricing-pages/README.md +++ b/templates/extract-pricing-pages/README.md @@ -3,12 +3,15 @@ Extract structured pricing fields from product pages with Zenrows Extract. ```bash -# Autoparse (automatic structured JSON) +# Default: extract=auto on an enabled domain (site-tailored JSON) +zenrows extract https://www.owler.com/company/meltwater + +# Autoparse — general-purpose JSON on any domain zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse # Or a precise CSS selector map zenrows extract --css '{"title":"h1","price":".price"}' --validate ``` -Requires `extract` (available). Start on one page, confirm the fields, then +Requires `extract` (open beta). Start on one page, confirm the fields, then fan out `zenrows extract` across the rest of your URLs. diff --git a/tests/capabilities.test.ts b/tests/capabilities.test.ts index a0a9833..1b18d6e 100644 --- a/tests/capabilities.test.ts +++ b/tests/capabilities.test.ts @@ -6,23 +6,29 @@ import { ToolkitError } from "../src/core/errors.ts"; test("capability matrix loads with the honest classifications", () => { const caps = loadCapabilities(); assert.equal(caps.protected_fetch?.status, "available"); - assert.equal(caps.extract?.status, "available"); - assert.equal(caps.browser?.status, "experimental"); + assert.equal(caps.extract?.status, "beta"); + assert.equal(caps.browser?.status, "available"); assert.equal(caps.mcp?.status, "available"); - // Batch is a real product in beta. + // Batch is a real product in open beta. assert.equal(caps.batch?.status, "beta"); }); -test("isUsable true for available, false for absent capabilities", () => { +test("isUsable true for available + open beta, false for absent capabilities", () => { assert.equal(isUsable("protected_fetch"), true); assert.equal(isUsable("extract"), true); - assert.equal(isUsable("batch"), false); // beta is not usable (cloud needs beta access) + assert.equal(isUsable("batch"), true); // open beta is usable + assert.equal(isUsable("nope"), false); }); -test("assertUsable passes for available + experimental, throws for absent", () => { +test("assertUsable passes for available + experimental + beta", () => { assert.doesNotThrow(() => assertUsable("protected_fetch")); assert.doesNotThrow(() => assertUsable("browser")); // experimental is allowed (policy gates it) - assert.throws(() => assertUsable("batch"), (e: unknown) => e instanceof ToolkitError && e.code === "CAPABILITY_UNAVAILABLE"); // beta is not usable + assert.doesNotThrow(() => assertUsable("extract")); + assert.doesNotThrow(() => assertUsable("batch")); + assert.throws( + () => assertUsable("nope"), + (e: unknown) => e instanceof ToolkitError && e.code === "CAPABILITY_UNAVAILABLE", + ); }); test("protected_fetch maps to the confirmed /v1/ endpoint", () => { diff --git a/tests/extract-outputs.test.ts b/tests/extract-outputs.test.ts index 06b405b..1c71350 100644 --- a/tests/extract-outputs.test.ts +++ b/tests/extract-outputs.test.ts @@ -42,15 +42,47 @@ test("buildParams omits outputs when not requested", () => { }); // --- runExtract wiring (stubbed network) ------------------------------------ -function withFetch(body: string, fn: () => Promise): Promise { +const AUTH010 = JSON.stringify({ + code: "AUTH010", + detail: "This domain is not enabled for Extract.", + status: 402, + title: "Feature is not included in plan (AUTH010)", + type: "https://docs.zenrows.com/api-error-codes#AUTH010", +}); + +const AUTH004 = JSON.stringify({ + code: "AUTH004", + detail: "This account has reached its usage limit.", + status: 402, + title: "Usage exceeded (AUTH004)", + type: "https://docs.zenrows.com/api-error-codes#AUTH004", +}); + +function withFetch(body: string, fn: () => Promise, opts: { status?: number; headers?: Record } = {}): Promise { const orig = globalThis.fetch; globalThis.fetch = (async () => - new Response(body, { status: 200, headers: { "content-type": "application/json" } })) as typeof fetch; + new Response(body, { + status: opts.status ?? 200, + headers: { "content-type": "application/json", ...opts.headers }, + })) as typeof fetch; + return fn().finally(() => { + globalThis.fetch = orig; + }); +} + +/** Stub fetch with a URL-aware handler; restores the original on exit. */ +function withFetchHandler(handler: (url: string) => Response, fn: () => Promise): Promise { + const orig = globalThis.fetch; + globalThis.fetch = (async (input: string | URL | Request) => handler(String(input))) as typeof fetch; return fn().finally(() => { globalThis.fetch = orig; }); } +test("buildParams forwards extract=auto", () => { + assert.equal(buildParams({ url: "https://x.com", extract: true }, cfg).extract, "auto"); +}); + test("runExtract method 'outputs' sets the param, and parses the JSON result", async () => { await withFetch('{"emails":["a@b.com"],"links":["https://x"]}', async () => { const outcome = await runExtract( @@ -61,10 +93,177 @@ test("runExtract method 'outputs' sets the param, and parses the JSON result", a ); assert.equal(outcome.method, "outputs"); assert.equal(outcome.params.outputs, "emails,links"); - // outputs must NOT also request autoparse / css / a response_type + // outputs must NOT also request extract / autoparse / css / a response_type + assert.equal(outcome.params.extract, undefined); assert.equal(outcome.params.autoparse, undefined); assert.equal(outcome.params.css_extractor, undefined); assert.equal(outcome.params.response_type, undefined); assert.deepEqual(outcome.data, { emails: ["a@b.com"], links: ["https://x"] }); }); }); + +test("runExtract method 'css' sets css_extractor", async () => { + await withFetch('{"title":"Hi"}', async () => { + const outcome = await runExtract( + { url: "https://x.com", method: "css", cssExtractor: '{"title":"h1"}' }, + cfg, + defaultPolicy(), + "testkey", + ); + assert.equal(outcome.method, "css"); + assert.equal(outcome.params.css_extractor, '{"title":"h1"}'); + assert.equal(outcome.params.extract, undefined); + assert.deepEqual(outcome.data, { title: "Hi" }); + }); +}); + +test("runExtract method 'markdown' sets response_type", async () => { + await withFetch("# Hello", async () => { + const outcome = await runExtract( + { url: "https://x.com", method: "markdown" }, + cfg, + defaultPolicy(), + "testkey", + ); + assert.equal(outcome.method, "markdown"); + assert.equal(outcome.params.response_type, "markdown"); + assert.equal(outcome.params.extract, undefined); + assert.equal(outcome.data, undefined); + }); +}); + +test("runExtract default method is extract=auto and unwraps {parsed,html}", async () => { + await withFetch( + JSON.stringify({ parsed: { title: "X" }, html: "x" }), + async () => { + const outcome = await runExtract({ url: "https://x.com" }, cfg, defaultPolicy(), "testkey"); + assert.equal(outcome.method, "extract"); + assert.equal(outcome.params.extract, "auto"); + assert.equal(outcome.params.autoparse, undefined); + assert.deepEqual(outcome.data, { title: "X" }); + assert.equal(outcome.html, "x"); + assert.equal(outcome.fellBackToAutoparse, undefined); + }, + ); +}); + +test("runExtract extract success surfaces costCredits from response headers", async () => { + await withFetch( + JSON.stringify({ parsed: { ok: true }, html: "" }), + async () => { + const outcome = await runExtract({ url: "https://x.com" }, cfg, defaultPolicy(), "testkey"); + assert.equal(outcome.result.costUsd, 0.0003625); + assert.equal(outcome.result.costCredits, 1); + }, + { headers: { "x-request-cost": "0.0003625", "x-request-credits": "1" } }, + ); +}); + +test("runExtract --autoparse skips extract and does not unwrap envelope", async () => { + await withFetch('{"title":"Y"}', async () => { + const outcome = await runExtract( + { url: "https://x.com", method: "autoparse" }, + cfg, + defaultPolicy(), + "testkey", + ); + assert.equal(outcome.method, "autoparse"); + assert.equal(outcome.params.autoparse, true); + assert.equal(outcome.params.extract, undefined); + assert.deepEqual(outcome.data, { title: "Y" }); + }); +}); + +test("runExtract falls back to autoparse on AUTH010 for extract=auto", async () => { + const calls: string[] = []; + await withFetchHandler((url) => { + calls.push(url); + if (url.includes("extract=auto")) { + return new Response(AUTH010, { status: 402, headers: { "content-type": "application/json" } }); + } + return new Response('{"title":"fallback"}', { + status: 200, + headers: { + "content-type": "application/json", + "x-request-cost": "0.0003625", + "x-request-credits": "1", + }, + }); + }, async () => { + const outcome = await runExtract({ url: "https://x.com" }, cfg, defaultPolicy(), "testkey"); + assert.equal(outcome.method, "autoparse"); + assert.equal(outcome.fellBackToAutoparse, true); + assert.equal(outcome.params.autoparse, true); + assert.deepEqual(outcome.data, { title: "fallback" }); + assert.equal(outcome.result.costCredits, 1); + assert.equal(calls.length, 2); + assert.match(calls[0]!, /extract=auto/); + assert.match(calls[1]!, /autoparse=true/); + assert.doesNotMatch(calls[1]!, /extract=auto/); + }); +}); + +test("runExtract does not fall back when fallbackAutoparse is false", async () => { + await withFetch(AUTH010, async () => { + await assert.rejects( + () => + runExtract( + { url: "https://x.com", method: "extract", fallbackAutoparse: false }, + cfg, + defaultPolicy(), + "testkey", + ), + (e: unknown) => e instanceof ToolkitError && e.code === "EXTRACT_DOMAIN_NOT_ENABLED", + ); + }, { status: 402 }); +}); + +test("runExtract does not fall back on AUTH004 (credits exhausted)", async () => { + let calls = 0; + await withFetchHandler(() => { + calls += 1; + return new Response(AUTH004, { status: 402, headers: { "content-type": "application/json" } }); + }, async () => { + await assert.rejects( + () => runExtract({ url: "https://x.com" }, cfg, defaultPolicy(), "testkey"), + (e: unknown) => e instanceof ToolkitError && e.code === "POLICY_MAX_CREDITS_EXCEEDED", + ); + assert.equal(calls, 1, "must not retry autoparse when credits are exhausted"); + }); +}); + +test("runExtract surfaces autoparse failure after AUTH010 fallback", async () => { + await withFetchHandler((url) => { + if (url.includes("extract=auto")) { + return new Response(AUTH010, { status: 402, headers: { "content-type": "application/json" } }); + } + return new Response("upstream boom", { status: 500, headers: { "content-type": "text/plain" } }); + }, async () => { + await assert.rejects( + () => runExtract({ url: "https://x.com" }, cfg, defaultPolicy(), "testkey"), + (e: unknown) => e instanceof ToolkitError && e.code === "FETCH_FAILED", + ); + }); +}); + +test("runExtract --validate fails when body is not JSON", async () => { + await withFetch("not json", async () => { + await assert.rejects( + () => + runExtract( + { url: "https://x.com", method: "autoparse", validate: true }, + cfg, + defaultPolicy(), + "testkey", + ), + (e: unknown) => e instanceof ToolkitError && e.code === "EXTRACT_VALIDATION_FAILED", + ); + }); +}); + +test("runExtract --css without a selector map fails locally (no network)", async () => { + await assert.rejects( + () => runExtract({ url: "https://x.com", method: "css" }, cfg, defaultPolicy(), "testkey"), + (e: unknown) => e instanceof ToolkitError && e.code === "INVALID_USAGE", + ); +}); diff --git a/tests/http.test.ts b/tests/http.test.ts index 0846ed0..05efc29 100644 --- a/tests/http.test.ts +++ b/tests/http.test.ts @@ -1,6 +1,6 @@ import { test } from "node:test"; import assert from "node:assert/strict"; -import { scrape } from "../src/core/http.ts"; +import { scrape, formatRequestCost } from "../src/core/http.ts"; import { CLI_VERSION } from "../src/core/config.ts"; // The exact shape Zenrows returns when an account is out of credits / over its @@ -70,6 +70,31 @@ test("scrape marks ordinary text responses as non-binary", async () => { ); }); +test("scrape parses X-Request-Cost and X-Request-Credits headers", async () => { + await withFetch( + () => + new Response("ok", { + status: 200, + headers: { + "content-type": "text/html", + "x-request-cost": "0.0018125", + "x-request-credits": "5", + }, + }), + async () => { + const r = await scrape("https://api.zenrows.com/v1/", "k", { url: "https://x" }); + assert.equal(r.costUsd, 0.0018125); + assert.equal(r.costCredits, 5); + }, + ); +}); + +test("formatRequestCost includes credits when present", () => { + assert.equal(formatRequestCost(0.0004, 1), "cost $0.0004 · 1 credit"); + assert.equal(formatRequestCost(0.0091, 25), "cost $0.0091 · 25 credits"); + assert.equal(formatRequestCost(0.0004, null), "cost $0.0004"); +}); + test("scrape sends a User-Agent carrying the current CLI version", async () => { let seenUA: string | undefined; const orig = globalThis.fetch; @@ -85,6 +110,58 @@ test("scrape sends a User-Agent carrying the current CLI version", async () => { assert.equal(seenUA, `zenrows-cli/${CLI_VERSION}`); }); +test("scrape maps AUTH010 on extract=auto to EXTRACT_DOMAIN_NOT_ENABLED", async () => { + const AUTH010 = JSON.stringify({ + code: "AUTH010", + detail: "This domain is not enabled for Extract.", + status: 402, + title: "Feature is not included in plan (AUTH010)", + type: "https://docs.zenrows.com/api-error-codes#AUTH010", + }); + await withFetch( + () => new Response(AUTH010, { status: 402, headers: { "content-type": "application/json" } }), + async () => { + await assert.rejects( + () => + scrape("https://api.zenrows.com/v1/", "test-key", { + url: "https://example.net", + extract: "auto", + }), + (err: unknown) => { + assert.ok(err && typeof err === "object" && "code" in err); + assert.equal((err as { code: string }).code, "EXTRACT_DOMAIN_NOT_ENABLED"); + return true; + }, + ); + }, + ); +}); + +test("scrape treats AUTH010 without extract as a credits/plan error, not EXTRACT_DOMAIN", async () => { + // AUTH010 is also used for other plan-gated features; only map it to + // EXTRACT_DOMAIN_NOT_ENABLED when the request asked for extract=auto. + const AUTH010 = JSON.stringify({ + code: "AUTH010", + detail: "Feature is not included in plan.", + status: 402, + title: "Feature is not included in plan (AUTH010)", + type: "https://docs.zenrows.com/api-error-codes#AUTH010", + }); + await withFetch( + () => new Response(AUTH010, { status: 402, headers: { "content-type": "application/json" } }), + async () => { + await assert.rejects( + () => scrape("https://api.zenrows.com/v1/", "test-key", { url: "https://example.net" }), + (err: unknown) => { + assert.ok(err && typeof err === "object" && "code" in err); + assert.equal((err as { code: string }).code, "POLICY_MAX_CREDITS_EXCEEDED"); + return true; + }, + ); + }, + ); +}); + test("scrape treats a 402 credit/quota response as an error, not success", async () => { await withFetch( () => new Response(AUTH004, { status: 402, headers: { "content-type": "application/json" } }), diff --git a/tests/registry.test.ts b/tests/registry.test.ts index 01fbcf4..b09c135 100644 --- a/tests/registry.test.ts +++ b/tests/registry.test.ts @@ -47,5 +47,7 @@ test("assetRunnable reflects backend capability status", () => { const fetchRecipe = loadRegistry("recipe").find((r) => r.name === "fetch-protected-page")!; assert.equal(assetRunnable(fetchRecipe), true); // needs protected_fetch (available) const batchSkill = loadRegistry("skill").find((s) => s.name === "batch-jobs")!; - assert.equal(assetRunnable(batchSkill), false); // needs batch (beta = not cloud-runnable) + assert.equal(assetRunnable(batchSkill), true); // needs batch (open beta = usable) + const extractSkill = loadRegistry("skill").find((s) => s.name === "extract")!; + assert.equal(assetRunnable(extractSkill), true); // needs extract (open beta = usable) }); diff --git a/workflows/competitor-intelligence/WORKFLOW.md b/workflows/competitor-intelligence/WORKFLOW.md index 11ee1ba..5e1a58f 100644 --- a/workflows/competitor-intelligence/WORKFLOW.md +++ b/workflows/competitor-intelligence/WORKFLOW.md @@ -22,12 +22,12 @@ simple per-URL loop). ``` 2. **Validate extraction** on one page (available). ``` - zenrows extract https://competitor.example/p/123 --autoparse --validate + zenrows extract https://competitor.example/p/123 --validate ``` 3. **Scale** across the collected URLs — loop the validated extract step over each URL (from your own script, cron, or CI): ``` - zenrows extract https://competitor.example/p/124 --autoparse --validate + zenrows extract https://competitor.example/p/124 --validate ``` > For large or recurring lists, **Batch** (`zenrows batch`) is > the managed alternative — one job submits many URLs, retries transient From c02176a9bffa6ce636f31578d14de43a50953185 Mon Sep 17 00:00:00 2001 From: Ander Date: Sat, 1 Aug 2026 14:01:28 +0200 Subject: [PATCH 04/17] feat(browser): Browser Sessions REST client MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add src/core/browser-api.ts — a thin REST client for the managed Browser Sessions API (https://mcp.zenrows.com/browser/sessions/*, Authorization: Bearer), modeled on batch-api.ts (separate host, header auth, injectable fetch). Covers session create/verb/close, base64 capture decode (screenshot/PDF), the CDP connect-URL builder, and normalizeSelectValue (the backend matches