diff --git a/README.md b/README.md index 02fb5a1..037f6b1 100644 --- a/README.md +++ b/README.md @@ -4,8 +4,8 @@ workflows, recipes, and evals layer for giving AI agents reliable access to protected web data through Zenrows cloud infrastructure.** -It makes the Zenrows Universal Scraper API, Scraping Browser, and Residential -Proxies installable and usable directly from AI agents and developer workflows — +It makes the four Zenrows primitives — Fetch, Extract, Batch, and Browser +Sessions — installable and usable directly from AI agents and developer workflows — so an agent can reliably access protected web data without hand-rolling anti-bot handling, proxies, or browser rendering. @@ -21,14 +21,14 @@ through AI agents, developers, and teams. ## 2. Why Zenrows Normal fetch fails. Generic scrapers fail. Browser-first tools are expensive. -The Zenrows **Universal Scraper API** retrieves protected pages reliably and -structures them, while the **Scraping Browser** is there for the rare cases that +Zenrows **Fetch** retrieves protected pages reliably and **Extract** structures +them, while **Browser Sessions** are there for the rare cases that need a real browser. Zenrows wins when the workflow runs over thousands or millions of URLs. ## 3. Product architecture -The core product is the Zenrows **Universal Scraper API** +The core product is Zenrows **Fetch and Extract** (`GET https://api.zenrows.com/v1/`). The CLI exposes it two ways: `zenrows fetch` retrieves a protected page, and `zenrows extract` turns it into structured data (JSON / CSS / Markdown). Both call the same endpoint — `extract` is just @@ -36,10 +36,10 @@ that endpoint with extraction parameters, not a separate product. | Command | What it does | Status (this build) | | --- | --- | --- | -| `zenrows fetch` | Universal Scraper API — retrieve a protected page | **available** — `GET https://api.zenrows.com/v1/` | -| `zenrows extract` | Universal Scraper API — structured extraction (Autoparse / CSS / Markdown) | **available** — same `/v1/` | -| `zenrows batch` | Batch Scraper API — fan out over many URLs | beta — cloud works with beta access; local validate/estimate always | -| `zenrows browser` | Scraping Browser (CDP) / MCP escalation | experimental | +| `zenrows fetch` | Fetch — retrieve a protected page | **available** — `GET https://api.zenrows.com/v1/` | +| `zenrows extract` | Extract — `extract=auto` (domain-gated open beta; falls back to Autoparse) / CSS / Markdown | **beta** — same `/v1/` | +| `zenrows batch` | Batch — fan out over many URLs | beta — cloud works with beta access; local validate/estimate always | +| `zenrows browser` | Browser Sessions REST API (same backend as MCP `browser_*`) | **available** — escalation-only; bills by bandwidth + time | | `zenrows mcp` | MCP server config (remote + local) | **available** | | Zenrows CLI | this repo | available | @@ -53,7 +53,8 @@ matrix. ```bash npx -y @zenrows/cli init zenrows fetch https://httpbin.io/html # auto-provisions a Free plan account on first use -zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse +zenrows extract https://www.owler.com/company/meltwater # extract=auto on an enabled domain +zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse # Autoparse (any domain) ``` No API key up front: on your first cloud call the toolkit creates a free, @@ -110,14 +111,16 @@ zenrows fetch --proxy-country us --wait-for ".price" ## 8. Extract ```bash -zenrows extract --autoparse +zenrows extract https://www.owler.com/company/meltwater # extract=auto (enabled domain) +zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse # Autoparse (any domain) zenrows extract --css '{"title":"h1","price":".price"}' --validate +zenrows extract --outputs emails,links # built-in output filters → JSON zenrows extract --output markdown ``` ## 9. Batch (beta) -The Zenrows **Batch Scraper API** (`https://async.api.zenrows.com/v1`) fans a +Zenrows **Batch** (`https://async.api.zenrows.com/v1`) fans a protected fetch/extract out over many URLs. It is a real product in **beta**: the cloud subcommands work once your API key has beta access; without it the API returns `BATCH_ACCESS_DENIED`. Local spec @@ -138,8 +141,17 @@ locally or fan out with `zenrows fetch` per URL. ## 10. Browser Sessions -Escalation only, gated by `policy.allow_browser`. Backed by the Zenrows Scraping -Browser (CDP) and the `@zenrows/mcp` `browser_*` tools. +Escalation only — **prefer `fetch`/`extract` for the vast majority of cases**; +they cost less. Use the browser for logins, forms, and multi-step JS flows that +Protected Fetch can't handle. Drives the managed Browser Sessions REST API +(`mcp.zenrows.com/browser/sessions/*`, same backend as the `@zenrows/mcp` +`browser_*` tools). `zenrows browser connect` prints the raw CDP wss URL for +bring-your-own Playwright/Puppeteer. + +**Billing & lifecycle:** sessions bill by **bandwidth + session time** and +**auto-terminate after 15 minutes** — `run ` closes automatically; +close interactive sessions with `zenrows browser close`. On by default; opt out +with `zenrows policy set allow_browser false`. ## 11. MCP @@ -216,8 +228,8 @@ Reports write `input.json`, `results.json`, `report.md`, `failures.jsonl`, - `.zenrows/account.json` holds no secret — only the accountId, Free-period info, and claim link. - `.zenrows/policy.json` enforces credit/page/concurrency limits and domain allow/deny. -- Destructive `uninstall` requires `--yes`. Browser and experimental are off by - default. +- Destructive `uninstall` requires `--yes`. Browser is on by default; opt out with + `zenrows policy set allow_browser false`. Experimental features are off by default. ## 19. Capability matrix diff --git a/docs/capabilities.md b/docs/capabilities.md index bc0637e..fd7939b 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -7,8 +7,9 @@ the single source of truth that prevents hallucinated execution. - **available** — a documented endpoint exists and the command does real work. - **available-but-needs-confirmation** — likely available; verify per account. -- **experimental** — exists but gated (e.g. browser, behind `policy.allow_browser`). -- **beta** — real product in beta; limited access (local spec / validation works today, cloud execution needs beta access). +- **experimental** — exists but not yet promoted to a stable status. +- **beta** — open beta; usable by any key. Product-specific limits (e.g. Extract + domain coverage) are handled by adapters / API errors, not by blocking the CLI. - **planned** — no documented endpoint yet; local spec / validation only. - **not-implemented** / **deprecated** — not usable. @@ -18,15 +19,8 @@ Classification is based on the public Zenrows documentation: | Capability | Backend evidence | Status | | --- | --- | --- | -| `protected_fetch` | Universal Scraper API `GET https://api.zenrows.com/v1/` with `mode`, `js_render`, `premium_proxy`, `proxy_country`, `wait`/`wait_for`, `js_instructions`, `response_type`, `screenshot`, `original_status`, … | available | -| `extract` | Same `/v1/` endpoint via `autoparse`, `css_extractor`, `response_type=markdown\|plaintext` | available | -| `batch` | Zenrows Batch Scraper API `https://async.api.zenrows.com/v1` (separate host, `X-API-Key` header) — real product in beta. Cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → `BATCH_ACCESS_DENIED`. Local JSONL spec validation + credit estimation work with no key. | beta | -| `browser` | Zenrows Scraping Browser (CDP) + `@zenrows/mcp` `browser_*` tools; no managed REST sessions API | experimental | +| `protected_fetch` | Fetch `GET https://api.zenrows.com/v1/` with `mode`, `js_render`, `premium_proxy`, `proxy_country`, `wait`/`wait_for`, `js_instructions`, `response_type`, `screenshot`, `original_status`, … | available | +| `extract` | Extract `GET https://api.zenrows.com/v1/` via `extract=auto` (domain-gated open beta; CLI falls back to `autoparse`), plus `autoparse`, `css_extractor`, `outputs`, `response_type=markdown\|plaintext` | beta | +| `batch` | Batch `https://async.api.zenrows.com/v1` (separate host, `X-API-Key` header) — open beta. Local JSONL spec validation + credit estimation work with no key. | beta | +| `browser` | Browser Sessions REST API `https://mcp.zenrows.com/browser/sessions/*` (Bearer; same backend as `@zenrows/mcp` `browser_*`); CDP via `zenrows browser connect`. Escalation-only (prefer fetch/extract); on by default, opt out via `policy.allow_browser`; bills by bandwidth + session time (15-min max) | available | | `mcp` | Hosted `https://mcp.zenrows.com/mcp` + local `npx -y @zenrows/mcp` | available | - -## Important honesty note - -`protected_fetch` and `extract` are the **same** product: a single `/v1/` -Universal Scraper API. "Extract" is not a separate endpoint — it is parameters -on that endpoint (`autoparse` / `css_extractor` / `response_type`). The CLI keeps -them as separate commands only for ergonomics. diff --git a/evals/extract-smoke/README.md b/evals/extract-smoke/README.md index cd11037..7f63c7e 100644 --- a/evals/extract-smoke/README.md +++ b/evals/extract-smoke/README.md @@ -1,9 +1,9 @@ # Eval: extract-smoke -Reproducible smoke test for Extract (Autoparse). +Reproducible smoke test for Extract (`extract=auto`). -- **Target:** `https://www.scrapingcourse.com/ecommerce/` (public scraping demo) -- **Config:** `autoparse=true` +- **Target:** `https://www.owler.com/company/meltwater` (enabled domain, real company page) +- **Config:** `extract=auto` - **Success criteria:** HTTP 2xx and a non-empty body - **Cost:** ~1× basic request diff --git a/evals/extract-smoke/spec.json b/evals/extract-smoke/spec.json index 4716727..3d3954f 100644 --- a/evals/extract-smoke/spec.json +++ b/evals/extract-smoke/spec.json @@ -1,10 +1,10 @@ { - "description": "Smoke test: Extract returns structured output for a known-good ecommerce demo.", + "description": "Smoke test: Extract (extract=auto) returns structured output for an enabled company page.", "steps": [ { "kind": "extract", - "url": "https://www.scrapingcourse.com/ecommerce/", - "options": { "method": "autoparse", "jsRender": false }, + "url": "https://www.owler.com/company/meltwater", + "options": { "method": "extract", "jsRender": false }, "expect": { "minLength": 2 } } ] diff --git a/registry/capabilities.json b/registry/capabilities.json index 87e75d8..231d875 100644 --- a/registry/capabilities.json +++ b/registry/capabilities.json @@ -1,50 +1,50 @@ { - "$comment": "Honest capability matrix derived from confirmed Zenrows docs. Every command checks status here before attempting a cloud call. 'available' = a documented endpoint exists today; 'planned' = no documented endpoint yet (local spec / unavailable behavior only); 'experimental' = exists but gated behind policy.", + "$comment": "Capability matrix for the Zenrows CLI. Every command consults this file before attempting a cloud call, so the CLI never fakes behavior for primitives the backend does not expose. Status values and classification rationale live in docs/capabilities.md; entries must stay in sync with the Capability type in src/types/index.ts.", "capabilities": { "protected_fetch": { "key": "protected_fetch", "label": "Protected Fetch", "status": "available", "command": "zenrows fetch", - "backend": "GET https://api.zenrows.com/v1/", + "backend": "Fetch — GET https://api.zenrows.com/v1/", "requiresAuth": true, - "notes": "Universal Scraper API. Confirmed params: mode=auto (Adaptive Stealth), js_render, premium_proxy, proxy_country, wait, wait_for, js_instructions, custom_headers, session_id, original_status, allowed_status_codes, block_resources, response_type, screenshot." + "notes": "Supported params: mode=auto (Adaptive Stealth), js_render, premium_proxy, proxy_country, wait, wait_for, js_instructions, custom_headers, session_id, original_status, allowed_status_codes, block_resources, response_type, screenshot." }, "extract": { "key": "extract", - "label": "Extract (Autoparse / CSS / Markdown)", - "status": "available", + "label": "Extract (extract=auto / Autoparse / CSS / Markdown)", + "status": "beta", "command": "zenrows extract", - "backend": "GET https://api.zenrows.com/v1/ (autoparse, css_extractor, response_type)", + "backend": "Extract — GET https://api.zenrows.com/v1/ (extract=auto, autoparse, css_extractor, outputs, response_type)", "requiresAuth": true, - "notes": "Structured extraction runs on the same /v1/ endpoint via autoparse=true, css_extractor, and response_type=markdown|plaintext. There is no separate /extract endpoint." + "notes": "Open beta. Default is extract=auto (domain-gated); CLI falls back to autoparse on AUTH010. Autoparse / CSS / outputs / markdown work on any domain." }, "batch": { "key": "batch", "label": "Batch (beta)", "status": "beta", "command": "zenrows batch", - "backend": "Batch Scraper API — https://async.api.zenrows.com/v1 (X-API-Key header; separate host from the scraper /v1/)", + "backend": "Batch — https://async.api.zenrows.com/v1 (X-API-Key header; separate host from the Fetch/Extract /v1/)", "requiresAuth": true, - "notes": "The Zenrows Batch Scraper API is a real product in beta. The cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → BATCH_ACCESS_DENIED. Local value always works with no key: `zenrows batch estimate` validates JSONL job specs and estimates credit cost." + "notes": "Open beta. Cloud subcommands (create/status/results/cancel/wait/retry-failed) call the Batch API; `zenrows batch estimate` works locally with no API key." }, "browser": { "key": "browser", - "label": "Browser (Scraping Browser)", - "status": "experimental", + "label": "Browser Sessions", + "status": "available", "command": "zenrows browser", - "backend": "Scraping Browser (CDP) + @zenrows/mcp browser tools", + "backend": "Browser Sessions REST API — https://mcp.zenrows.com/browser/sessions/* (Bearer auth; same backend as @zenrows/mcp browser_* tools)", "requiresAuth": true, - "notes": "Zenrows Scraping Browser and the @zenrows/mcp browser_* tools exist. There is no managed REST 'sessions' API in the public docs, so this is gated as experimental and escalation-only (policy.allow_browser=false by default)." + "notes": "Browser Sessions is a GA Zenrows product (formerly Scraping Browser). Drives its managed REST API directly (create/verb/close over HTTP with Authorization: Bearer) — no CDP client or browser dependency needed. Escalation-only: prefer fetch/extract (they cost less) for the vast majority of cases. On by default; opt out with policy.allow_browser=false. Sessions bill by bandwidth + session time and auto-terminate after 15 minutes. For raw CDP control, `zenrows browser connect` prints the wss://browser.zenrows.com endpoint for your own Playwright/Puppeteer." }, "mcp": { "key": "mcp", "label": "MCP", "status": "available", "command": "zenrows mcp", - "backend": "remote https://mcp.zenrows.com/mcp + local npx -y @zenrows/mcp", + "backend": "Remote https://mcp.zenrows.com/mcp + local `npx -y @zenrows/mcp`", "requiresAuth": true, - "notes": "Both a hosted remote MCP server and a local STDIO server (@zenrows/mcp, ZENROWS_API_KEY env) are documented." + "notes": "Hosted remote MCP server plus a local STDIO server (`npx -y @zenrows/mcp`, authenticated via the ZENROWS_API_KEY environment variable)." } } } diff --git a/registry/skills.json b/registry/skills.json index 7d19f2e..458cea6 100644 --- a/registry/skills.json +++ b/registry/skills.json @@ -26,7 +26,7 @@ "name": "extract", "type": "skill", "description": "Turn protected pages into structured data with Autoparse / CSS / Markdown.", - "status": "available", + "status": "beta", "requires_backend_capabilities": ["extract"], "requires_auth": true, "version": "0.1.0", @@ -36,7 +36,7 @@ { "name": "batch-jobs", "type": "skill", - "description": "Scale protected fetch/extract over many URLs with the Batch Scraper API (beta): submit/track/collect jobs with beta access; validate + estimate specs locally with no key.", + "description": "Scale protected fetch/extract over many URLs with Batch (beta): submit/track/collect jobs with beta access; validate + estimate specs locally with no key.", "status": "beta", "requires_backend_capabilities": ["batch"], "requires_auth": true, @@ -47,8 +47,8 @@ { "name": "interact-browser", "type": "skill", - "description": "Escalate to a browser (Scraping Browser / MCP) only when fetch/extract cannot do the job.", - "status": "experimental", + "description": "Escalate to Browser Sessions (REST API / MCP browser_*) only when fetch/extract cannot do the job.", + "status": "available", "requires_backend_capabilities": ["browser"], "requires_auth": true, "version": "0.1.0", diff --git a/registry/templates.json b/registry/templates.json index 4717f83..d65b411 100644 --- a/registry/templates.json +++ b/registry/templates.json @@ -3,7 +3,7 @@ { "name": "protected-fetch-node", "type": "template", - "description": "Minimal Node.js project calling the Zenrows Universal Scraper API.", + "description": "Minimal Node.js project calling Zenrows Fetch.", "status": "available", "requires_backend_capabilities": ["protected_fetch"], "requires_auth": true, @@ -25,7 +25,7 @@ { "name": "batch-jsonl-pipeline", "type": "template", - "description": "JSONL job-spec scaffold for high-scale workloads on the Batch Scraper API: submit/track/collect with beta access, validate + estimate locally with no key (beta).", + "description": "JSONL job-spec scaffold for high-scale workloads on Batch: submit/track/collect with beta access, validate + estimate locally with no key (beta).", "status": "beta", "requires_backend_capabilities": [], "requires_auth": false, diff --git a/skills/batch-jobs/SKILL.md b/skills/batch-jobs/SKILL.md index ee31df6..e6bc028 100644 --- a/skills/batch-jobs/SKILL.md +++ b/skills/batch-jobs/SKILL.md @@ -1,6 +1,6 @@ --- name: batch-jobs -description: Scale protected fetch/extract over many URLs via the Batch Scraper API (beta). Cloud create/status/results/cancel/wait/retry-failed work with beta access; estimate/validate run locally with no key. +description: Scale protected fetch/extract over many URLs via Batch (beta). Cloud create/status/results/cancel/wait/retry-failed work with beta access; estimate/validate run locally with no key. version: 0.1.0 requires_backend_capabilities: [batch] --- @@ -11,7 +11,7 @@ Process large workloads reliably and asynchronously. Batch is where Zenrows' high-scale anti-bot advantage becomes obvious — Zenrows wins when the workflow runs over thousands, millions, or recurring sets of URLs. -> Status: **beta**. The Zenrows Batch Scraper API is a real +> Status: **beta**. The Zenrows **Batch** is a real > product in beta and runs on a separate host > (`async.api.zenrows.com/v1`). The cloud subcommands work once your account has > beta access; without it the API returns 403 → `BATCH_ACCESS_DENIED`. The diff --git a/skills/compliance-policy/SKILL.md b/skills/compliance-policy/SKILL.md index 02f804f..23d651e 100644 --- a/skills/compliance-policy/SKILL.md +++ b/skills/compliance-policy/SKILL.md @@ -15,16 +15,16 @@ zenrows policy show zenrows policy set max_credits_per_run 5000 zenrows policy set blocked_domains "example.com,foo.test" zenrows policy set allowed_domains "mysite.com" # non-empty = allow-list mode -zenrows policy set allow_browser true # enable escalation +zenrows policy set allow_browser false # opt OUT of browser (on by default) zenrows policy set allow_experimental true ``` ## Hard rules - **Never** print or commit API keys. Keys live in `.zenrows/secrets.json` (0600, gitignored) and are redacted from logs and run artifacts. -- Respect `allowed_domains` / `blocked_domains` (→ `POLICY_BLOCKED_DOMAIN`). -- Stay under `max_credits_per_run` / `max_pages_per_run` / `max_concurrency`. +- Respect `allowed_domains` / `blocked_domains` — enforced by the CLI (→ `POLICY_BLOCKED_DOMAIN`). +- `max_credits_per_run` / `max_pages_per_run` / `max_concurrency` are **advisory budgets** surfaced by `zenrows status` (not hard-enforced by the CLI today) — self-limit against them. - Confirm destructive `uninstall` with `--yes`. -- Browser and experimental commands are **off by default**. +- **Experimental** commands are off by default (`allow_experimental`). **Browser is on by default** (opt out with `allow_browser=false`). See [[cost-control]] and [[trace-debug]]. diff --git a/skills/cost-control/SKILL.md b/skills/cost-control/SKILL.md index 903a041..b4e25bd 100644 --- a/skills/cost-control/SKILL.md +++ b/skills/cost-control/SKILL.md @@ -9,7 +9,7 @@ requires_backend_capabilities: [] Prefer the **cheapest reliable** configuration; escalate only with evidence. -## Cost multipliers (Universal Scraper API) +## Cost multipliers (Fetch) - Basic request: **1×** - JS rendering (`js_render`): **5×** - Premium proxies (`premium_proxy`): **10×** diff --git a/skills/extract/SKILL.md b/skills/extract/SKILL.md index aad6239..c5d4672 100644 --- a/skills/extract/SKILL.md +++ b/skills/extract/SKILL.md @@ -1,6 +1,6 @@ --- name: extract -description: Turn protected pages into structured data with Autoparse / CSS / Markdown. +description: Turn protected pages into structured data with Extract / Autoparse / CSS / Markdown. version: 0.1.0 requires_backend_capabilities: [extract] --- @@ -10,26 +10,25 @@ requires_backend_capabilities: [extract] Convert protected pages into structured data. The value is **protected page access + extraction**, not generic LLM parsing. -> Honest note: there is no separate `/extract` endpoint. Extraction runs on the -> same Universal Scraper API (`/v1/`) via `autoparse`, `css_extractor`, and -> `response_type`. - -## Methods (available today) +## Methods ``` -zenrows extract --autoparse # automatic structured JSON +zenrows extract # extract=auto (default); falls back to Autoparse if domain not enabled +zenrows extract --autoparse # general-purpose Autoparse (any domain) zenrows extract --css '{"title":"h1","price":".price"}' # selector map -zenrows extract --output markdown # Markdown conversion -zenrows extract --validate # fail if not valid JSON +zenrows extract --outputs emails,links # built-in output filters → JSON +zenrows extract --output markdown # Markdown conversion +zenrows extract --validate # fail if not valid JSON ``` ## When to use which -- **Autoparse**: quick prototyping; product/article/job/listing pages. +- **Extract** (`extract=auto`, default): richest fields on domains enabled for the open beta. +- **Autoparse**: any website; also the automatic fallback when Extract is not enabled for the domain. - **CSS**: you know the exact fields/selectors and want determinism. - **Markdown/plaintext**: feed clean content to an LLM yourself. ## Rules - Validate on a single page before scaling across many URLs. -- If autoparse misses fields, switch to `--css` with explicit selectors, or add +- If Autoparse misses fields, switch to `--css` with explicit selectors, or add `--manual --js-render` for JS-heavy pages. See [[protected-fetch]] for retrieval semantics. diff --git a/skills/interact-browser/SKILL.md b/skills/interact-browser/SKILL.md index 3fbdba1..ea2eaaf 100644 --- a/skills/interact-browser/SKILL.md +++ b/skills/interact-browser/SKILL.md @@ -12,12 +12,18 @@ persistent cookies) on JS-heavy or interactive pages that Protected Fetch cannot handle. This is **escalation only** — use it only when Protected Fetch or Extract is insufficient. -> Status: **experimental**, gated behind `policy.allow_browser` (default false). -> There is no managed REST "sessions" API; browser workflows run through the -> Zenrows **Scraping Browser** (CDP — connect Playwright/Puppeteer) and the -> `@zenrows/mcp` `browser_*` tools (navigate, click, fill, screenshot, …). +> Browser Sessions is a **GA** Zenrows product (formerly Scraping Browser), +> **on by default** (opt out with `policy.allow_browser=false`). It is +> escalation-only and costs more than fetch/extract: sessions bill by +> **bandwidth + session time** and auto-terminate after **15 minutes**. +> Drive sessions with `zenrows browser` against the managed REST API +> (`https://mcp.zenrows.com/browser/sessions/*`, same backend as `@zenrows/mcp` +> `browser_*`). For raw CDP, `zenrows browser connect` prints the wss endpoint. ## Rules -- Try [[protected-fetch]] / [[extract]] first; escalate only with evidence. -- Enable in policy explicitly: `zenrows policy set allow_browser true`. -- Keep sessions short; capture a trace for debugging hard targets. +- Try [[protected-fetch]] / [[extract]] first; escalate only with evidence — they cost less. +- On by default; if a workspace opted out, re-enable with `zenrows policy set allow_browser true`. +- Prefer `zenrows browser run ` for multi-step flows (auto-closes). +- Always `close` interactive sessions — billed by bandwidth + session time (15-min hard cap). +- `select --value` matches the option's `value` attribute (bare value or a CSS + selector like `option[value="2"]`). Label text / index are not supported. diff --git a/skills/protected-fetch/SKILL.md b/skills/protected-fetch/SKILL.md index aa21460..9af8a46 100644 --- a/skills/protected-fetch/SKILL.md +++ b/skills/protected-fetch/SKILL.md @@ -8,8 +8,8 @@ requires_backend_capabilities: [protected_fetch] # Protected Fetch Retrieve pages that normal fetch, generic scrapers, or naive browser tools -cannot. This is the **core primitive** — backed by the Zenrows Universal Scraper -API (`GET https://api.zenrows.com/v1/`). +cannot. This is the **core primitive** — backed by Zenrows **Fetch** +(`GET https://api.zenrows.com/v1/`). ## When to use - You have a known URL and want its content (HTML, Markdown, text, or a PDF). diff --git a/skills/zenrows/SKILL.md b/skills/zenrows/SKILL.md index e4b84ae..e4da444 100644 --- a/skills/zenrows/SKILL.md +++ b/skills/zenrows/SKILL.md @@ -32,13 +32,13 @@ If the user has a known URL and wants page content: → Use Protected Fetch. (zenrows fetch ) If the user has a known URL and wants structured data: - → Use Extract. (zenrows extract --autoparse | --css) + → Use Extract. (zenrows extract | --autoparse | --css) If the user has many URLs: → Fan out fetch/extract per URL (validate on one page first, then iterate). If the user needs login, clicks, forms, sessions, or persistent state: - → Use Interact / Browser Sessions.(zenrows browser) [experimental, escalation-only] + → Use Interact / Browser Sessions.(zenrows browser) [escalation-only] If the user wants to integrate with an agent/coding environment: → Use the Zenrows CLI: install / plugin / mcp / skills / templates / @@ -54,7 +54,8 @@ If the user wants to integrate with an agent/coding environment: (`zenrows trace explain `) before adding `--js-render` / `--premium-proxy`. - **Do not use the browser unless required.** Browser is an escalation layer, - not the default. It is gated behind `policy.allow_browser`. + not the default — prefer fetch/extract (they cost less). It is on by default; + opt out with `policy.allow_browser=false`. Sessions bill by bandwidth + time. - **Do not scale before validating** the workflow on a small sample. ## What is available today @@ -64,16 +65,16 @@ Run `zenrows status` for the live capability matrix. As of this toolkit: | Primitive | Command | Status | | --- | --- | --- | | Protected Fetch | `zenrows fetch` | available (`GET /v1/`) | -| Extract (Autoparse/CSS/Markdown) | `zenrows extract` | available (same `/v1/`) | +| Extract (extract=auto / Autoparse/CSS/Markdown) | `zenrows extract` | beta (same `/v1/`; extract=auto falls back to autoparse) | | Batch | `zenrows batch` | beta (validate specs locally) | -| Browser | `zenrows browser` | experimental (Scraping Browser / MCP) | +| Browser | `zenrows browser` | available (Browser Sessions REST API / MCP) | | MCP | `zenrows mcp` | available (remote + local server) | -Protected Fetch and Extract are the same Universal Scraper API used two ways. +Protected Fetch and Extract are the same API (`GET /v1/`) used two ways. The command consults the capability matrix before any cloud call — it never fakes success. -## Cost model (Universal Scraper API) +## Cost model (Fetch and Extract) Relative multipliers: basic **1×**, JS rendering **5×**, premium proxies **10×**, both **25×**. `mode=auto` charges only for the configuration that succeeds. diff --git a/src/adapters/batch.ts b/src/adapters/batch.ts index 1999d6e..78b0042 100644 --- a/src/adapters/batch.ts +++ b/src/adapters/batch.ts @@ -1,7 +1,7 @@ /** * Batch Jobs adapter. * - * Status: `beta` — the Zenrows Batch Scraper API is in beta. + * Status: `beta` — the Zenrows Batch API is in beta. * With beta access the cloud subcommands run for real (see `core/batch-api.ts`). * Without access the API returns 403 → BATCH_ACCESS_DENIED. This adapter owns * the local, no-network pieces: validating a JSONL job spec, estimating credit @@ -176,7 +176,7 @@ function assertProxyCountryPremium(params: Record, where: strin code: "PARAM_PROXY_COUNTRY_REQUIRES_PREMIUM", message: `proxy_country needs premium proxies (or mode=auto) — ${where}.`, likely_cause: - "The Batch Scraper API only geolocates the proxy when premium_proxy=true, or in Adaptive Stealth Mode (mode=auto).", + "Batch only geolocates the proxy when premium_proxy=true, or in Adaptive Stealth Mode (mode=auto).", next_action: "Add premium_proxy (10x cost) alongside proxy_country, or set mode=auto — geo-targeting works there without the flag.", suggested_commands: ["zenrows batch create jobs.jsonl --premium-proxy --proxy-country us"], diff --git a/src/adapters/extract.ts b/src/adapters/extract.ts index a39138a..a20485a 100644 --- a/src/adapters/extract.ts +++ b/src/adapters/extract.ts @@ -1,33 +1,60 @@ /** * Extract adapter. * - * IMPORTANT (honest mapping): there is no separate `/extract` endpoint. - * Structured extraction runs on the same `/v1/` Universal Scraper API via: - * - autoparse=true → automatic structured JSON (available) - * - css_extractor= → selector-based field extraction (available) - * - outputs= → built-in output filters → JSON (available) - * - response_type=markdown|plaintext (available) + * Structured extraction on the same `/v1/` Fetch and Extract API via: + * - extract=auto → site-tailored Extract (open beta, domain-gated) + * - autoparse=true → general-purpose Autoparse (any domain) + * - css_extractor= → selector-based field extraction + * - outputs= → built-in output filters → JSON + * - response_type=markdown|plaintext + * + * Default method is `extract`. On AUTH010 (domain not in Extract beta) we + * automatically retry once with Autoparse unless the caller opted into + * `--autoparse` (or another explicit method) or disabled the fallback. */ import type { Policy, ToolkitConfig } from "../types/index.ts"; import { ToolkitError } from "../core/errors.ts"; import { runFetch, type FetchOptions, type FetchOutcome } from "./protected-fetch.ts"; -export type ExtractMethod = "autoparse" | "css" | "outputs" | "markdown" | "plaintext"; +export type ExtractMethod = "extract" | "autoparse" | "css" | "outputs" | "markdown" | "plaintext"; -export interface ExtractOptions extends Omit { - /** Deterministic extraction method backed by /v1/. */ +export interface ExtractOptions extends Omit { + /** Deterministic extraction method backed by /v1/. Defaults to `extract`. */ method?: ExtractMethod; cssExtractor?: string; /** Comma-separated output filters (e.g. "emails,links" or "*"). Used with method "outputs". */ outputs?: string; /** Validate the parsed JSON shape locally (best-effort). */ validate?: boolean; + /** + * When method is `extract` (default), retry once with Autoparse if the domain + * is not enabled for Extract (AUTH010). Defaults to true. + */ + fallbackAutoparse?: boolean; } export interface ExtractOutcome extends FetchOutcome { method: ExtractMethod; /** Parsed JSON when the method yields structured data; otherwise undefined. */ data?: unknown; + /** Raw HTML from `extract=auto` beta responses (validation aid). */ + html?: string; + /** True when Extract fell back to Autoparse because the domain is not enabled. */ + fellBackToAutoparse?: boolean; + /** + * True when a structured method returned no data (null / empty object / empty + * array / non-JSON). Lets callers warn instead of reporting a silent success. + */ + empty?: boolean; +} + +/** True when a parsed extraction result carries no usable structured data. */ +function isEmptyData(data: unknown): boolean { + if (data === null || data === undefined) return true; + if (Array.isArray(data)) return data.length === 0; + if (typeof data === "object") return Object.keys(data as object).length === 0; + if (typeof data === "string") return data.trim() === ""; + return false; } export async function runExtract( @@ -37,15 +64,7 @@ export async function runExtract( apiKey: string, ): Promise { const method: ExtractMethod = - opts.method ?? (opts.outputs ? "outputs" : opts.cssExtractor ? "css" : "autoparse"); - - const fetchOpts: FetchOptions = { - ...opts, - autoparse: method === "autoparse", - cssExtractor: method === "css" ? opts.cssExtractor : undefined, - outputs: method === "outputs" ? opts.outputs : undefined, - output: method === "markdown" ? "markdown" : method === "plaintext" ? "plaintext" : "html", - }; + opts.method ?? (opts.outputs ? "outputs" : opts.cssExtractor ? "css" : "extract"); if (method === "css" && !opts.cssExtractor) { throw new ToolkitError({ @@ -65,25 +84,95 @@ export async function runExtract( }); } - const outcome = await runFetch(fetchOpts, config, policy, apiKey); - - let data: unknown; - if (method === "autoparse" || method === "css" || method === "outputs") { + if (method === "extract" && opts.fallbackAutoparse !== false) { try { - data = JSON.parse(outcome.result.body); - } catch { - if (opts.validate) { - throw new ToolkitError({ - code: "EXTRACT_VALIDATION_FAILED", - message: "Extraction did not return valid JSON.", - likely_cause: - "The page may need js_render, or autoparse could not detect structured data on this layout.", - next_action: "Retry with --manual --js-render, or switch to --css with explicit selectors.", - suggested_commands: [`zenrows extract ${opts.url} --manual --js-render`], - }); + return await runExtractOnce({ ...opts, method: "extract" }, config, policy, apiKey); + } catch (err) { + if (err instanceof ToolkitError && err.code === "EXTRACT_DOMAIN_NOT_ENABLED") { + const outcome = await runExtractOnce({ ...opts, method: "autoparse" }, config, policy, apiKey); + return { ...outcome, fellBackToAutoparse: true }; } + throw err; + } + } + + return runExtractOnce({ ...opts, method }, config, policy, apiKey); +} + +async function runExtractOnce( + opts: ExtractOptions & { method: ExtractMethod }, + config: ToolkitConfig, + policy: Policy, + apiKey: string, +): Promise { + const { method } = opts; + const fetchOpts: FetchOptions = { + ...opts, + extract: method === "extract", + autoparse: method === "autoparse", + cssExtractor: method === "css" ? opts.cssExtractor : undefined, + outputs: method === "outputs" ? opts.outputs : undefined, + output: method === "markdown" ? "markdown" : method === "plaintext" ? "plaintext" : "html", + }; + + const outcome = await runFetch(fetchOpts, config, policy, apiKey); + const { data, html, empty } = parseExtractBody(method, outcome.result.body, opts); + + return { ...outcome, method, data, html, empty }; +} + +function parseExtractBody( + method: ExtractMethod, + body: string, + opts: Pick, +): { data?: unknown; html?: string; empty?: boolean } { + if (method !== "extract" && method !== "autoparse" && method !== "css" && method !== "outputs") { + return {}; + } + + let parsed: unknown; + try { + parsed = JSON.parse(body); + } catch { + if (opts.validate) { + throw new ToolkitError({ + code: "EXTRACT_VALIDATION_FAILED", + message: "Extraction did not return valid JSON.", + likely_cause: + "The page may need js_render, or structured extraction could not detect data on this layout.", + next_action: "Retry with --manual --js-render, or switch to --css with explicit selectors.", + suggested_commands: [`zenrows extract ${opts.url} --manual --js-render`], + }); + } + return { empty: true }; + } + + let data: unknown = parsed; + let html: string | undefined; + // extract=auto beta shape: { parsed, html }. Prefer `parsed` for callers. + if (method === "extract" && parsed && typeof parsed === "object" && !Array.isArray(parsed)) { + const envelope = parsed as { parsed?: unknown; html?: unknown }; + if ("parsed" in envelope) { + data = envelope.parsed; + html = typeof envelope.html === "string" ? envelope.html : undefined; } } - return { ...outcome, method, data }; + const empty = isEmptyData(data); + // ACT-1514: an empty result must not read as a silent success. Under + // --validate it's an error; otherwise the command surfaces a warning. + if (opts.validate && empty) { + throw new ToolkitError({ + code: "EXTRACT_VALIDATION_FAILED", + message: "Extraction returned no structured data.", + likely_cause: + "The domain returned an empty result — the page may render content via JavaScript, or this layout isn't auto-extractable.", + next_action: "Retry with --manual --js-render, or use --css with explicit selectors.", + suggested_commands: [ + `zenrows extract ${opts.url} --manual --js-render`, + `zenrows extract ${opts.url} --css '{"title":"h1"}'`, + ], + }); + } + return { data, html, empty }; } diff --git a/src/adapters/protected-fetch.ts b/src/adapters/protected-fetch.ts index 6280b00..4f5d6d5 100644 --- a/src/adapters/protected-fetch.ts +++ b/src/adapters/protected-fetch.ts @@ -1,5 +1,5 @@ /** - * Protected Fetch adapter → Zenrows Universal Scraper API (`GET /v1/`). + * Protected Fetch adapter → Zenrows Fetch (`GET /v1/`). * * Maps toolkit options to confirmed API parameters and enforces the * auto/manual contract: in Adaptive Stealth Mode (mode=auto), `js_render` and @@ -34,7 +34,12 @@ export interface FetchOptions { cssExtractor?: string; autoparse?: boolean; /** - * Comma-separated Universal Scraper API output filters (e.g. "emails,links", + * Site-tailored Extract (`extract=auto`). Open beta, domain-gated. + * Takes precedence over autoparse / css_extractor / outputs / response_type. + */ + extract?: boolean; + /** + * Comma-separated Fetch and Extract output filters (e.g. "emails,links", * or "*" for all available fields). Returns structured JSON. Standalone: not combined with * autoparse / css_extractor / response_type. */ @@ -112,6 +117,7 @@ export function buildParams(opts: FetchOptions, config: ToolkitConfig): ScraperP if (opts.sessionId !== undefined) params.session_id = opts.sessionId; if (opts.originalStatus) params.original_status = true; if (opts.allowedStatusCodes) params.allowed_status_codes = opts.allowedStatusCodes; + if (opts.extract) params.extract = "auto"; if (opts.autoparse) params.autoparse = true; if (opts.cssExtractor) params.css_extractor = opts.cssExtractor; if (opts.outputs) params.outputs = opts.outputs; diff --git a/src/cli/asset-command.ts b/src/cli/asset-command.ts index 5c9be06..834b0cf 100644 --- a/src/cli/asset-command.ts +++ b/src/cli/asset-command.ts @@ -173,7 +173,7 @@ function explainCmd(type: AssetType, argv: string[], ctx: RunContext): number { if (!name) throw usageErr(type, "explain "); const asset = requireAsset(type, name); if (ctx.json) { - log.out(JSON.stringify({ ...asset, runnable: assetRunnable(asset) }, null, 2)); + log.out(JSON.stringify({ ok: true, ...asset, runnable: assetRunnable(asset) }, null, 2)); return 0; } log.info(c(ANSI.bold, `${asset.name} (${asset.type}, ${asset.status})`)); @@ -257,7 +257,7 @@ function validateSkill(argv: string[], ctx: RunContext): number { if (/apikey=|ZENROWS_API_KEY=[A-Za-z0-9]/.test(body)) errors.push("SKILL.md may contain a secret"); } if (ctx.json) { - log.out(JSON.stringify({ name, valid: errors.length === 0, errors }, null, 2)); + log.out(JSON.stringify({ ok: errors.length === 0, name, valid: errors.length === 0, errors }, null, 2)); } else if (errors.length === 0) { log.success(`Skill "${name}" is valid.`); } else { @@ -400,10 +400,22 @@ function writeEvalReport(asset: RegistryAsset, spec: { description?: string }, r mkdirSync(join(dir, "traces"), { recursive: true }); const passed = results.filter((r) => r.ok).length; const totalCost = results.reduce((a, r) => a + (r.costUsd ?? 0), 0); + const totalCredits = results.reduce((a, r) => a + (r.costCredits ?? 0), 0); writeFileSync(join(dir, "input.json"), JSON.stringify({ eval: asset.name, spec }, null, 2) + "\n"); writeFileSync(join(dir, "results.json"), JSON.stringify({ runId, passed, total: results.length, results: results.map(safeResult) }, null, 2) + "\n"); - writeFileSync(join(dir, "cost.json"), JSON.stringify({ totalCostUsd: totalCost, perStep: results.map((r) => r.costUsd ?? 0) }, null, 2) + "\n"); + writeFileSync( + join(dir, "cost.json"), + JSON.stringify( + { + totalCostUsd: totalCost, + totalCredits, + perStep: results.map((r) => ({ costUsd: r.costUsd ?? 0, costCredits: r.costCredits ?? 0 })), + }, + null, + 2, + ) + "\n", + ); const failures = results.filter((r) => !r.ok).map((r) => JSON.stringify(safeResult(r))).join("\n"); writeFileSync(join(dir, "failures.jsonl"), failures ? failures + "\n" : ""); const report = [ @@ -412,6 +424,7 @@ function writeEvalReport(asset: RegistryAsset, spec: { description?: string }, r `- run id: \`${runId}\``, `- success rate: ${passed}/${results.length}`, `- approx cost (USD): ${totalCost.toFixed(4)}`, + `- approx credits: ${totalCredits}`, "", "## Targets & results", ...results.map( @@ -443,6 +456,7 @@ function safeResult(r: StepResult) { status: r.status, bytes: r.bytes, costUsd: r.costUsd, + costCredits: r.costCredits, estimatedCredits: r.estimatedCredits, requestId: r.requestId, failureReason: r.failureReason, diff --git a/src/cli/commands/account.ts b/src/cli/commands/account.ts index 4ea1a5d..39f7b85 100644 --- a/src/cli/commands/account.ts +++ b/src/cli/commands/account.ts @@ -69,6 +69,7 @@ export const account: Command = { if (ctx.json) { log.out(JSON.stringify({ + ok: true, hasKey: auth.hasKey, source: auth.source, account: acct ?? null, diff --git a/src/cli/commands/batch.ts b/src/cli/commands/batch.ts index 46a102d..6d55e03 100644 --- a/src/cli/commands/batch.ts +++ b/src/cli/commands/batch.ts @@ -1,5 +1,5 @@ /** - * `zenrows batch` — Batch Scraper API (status: beta). + * `zenrows batch` — Batch API (status: beta). * * Local (no key, always works): `estimate`/`create --dry-run`-style spec * validation + credit estimate. Cloud (needs a key + Batch beta access): @@ -11,6 +11,10 @@ import { join } from "node:path"; import { log, ANSI, c } from "../../core/logger.ts"; import { estimateCredits, toJobBody, validateJsonl } from "../../adapters/batch.ts"; import { requireApiKey } from "../../core/auth.ts"; +import { ensureApiKey } from "../../core/ensure-key.ts"; +import { assertUsable } from "../../core/capabilities.ts"; +import { assertDomainAllowed, assertWithinLimits, loadPolicy } from "../../core/policy.ts"; +import { newRunId, writeRun } from "../../core/artifacts.ts"; import { createJob, downloadResults, getJob, listResults, rerunJob, stopJob, waitForJob, type Job } from "../../core/batch-api.ts"; import { asNumber, asString, parse, type Command, type RunContext } from "../command.ts"; import { ToolkitError } from "../../core/errors.ts"; @@ -18,7 +22,7 @@ import { printError, writeOut } from "../output.ts"; export const batch: Command = { name: "batch", - summary: "Run JSONL batch jobs on the Zenrows Batch Scraper API (beta).", + summary: "Run JSONL batch jobs on Zenrows Batch (beta).", usage: "zenrows batch |status |results |cancel |wait |retry-failed >", help: [ "Local (no key):", @@ -28,8 +32,9 @@ export const batch: Command = { " --js-render job-level: render JavaScript", " --premium-proxy job-level: use residential IPs", " --proxy-country job-level: geo-target (needs --premium-proxy)", - " --output job-level response_type (markdown|plaintext|pdf)", - " --wait poll until the run finishes", + " --output job-level response_type (markdown|plaintext|pdf|html)", + " --follow poll until the run finishes (alias: --wait)", + " --no-signup do not auto-create a Free plan account if no key exists", " status show run status + stats", " results [--status s] list results (successful|failed|all); paginated", " --out write results as JSONL instead of printing", @@ -79,7 +84,7 @@ function estimateCmd(rest: string[], ctx: RunContext): number { const v = validateJsonl(file); const est = estimateCredits(v.jobs); if (ctx.json) { - log.out(JSON.stringify({ ...v, estimatedCredits: est.credits }, null, 2)); + log.out(JSON.stringify({ ok: v.errors.length === 0, ...v, estimatedCredits: est.credits }, null, 2)); } else { log.info(c(ANSI.bold, `Job spec: ${file}`)); log.info(`valid jobs: ${v.validJobs}/${v.totalLines}`); @@ -100,13 +105,21 @@ async function createCmd(rest: string[], ctx: RunContext): Promise { "premium-proxy": { type: "boolean" }, "proxy-country": { type: "string" }, output: { type: "string" }, - wait: { type: "boolean" }, + follow: { type: "boolean" }, + wait: { type: "boolean" }, // back-compat alias for --follow + "no-signup": { type: "boolean" }, json: { type: "boolean" }, }); + // `--follow` is the clear name (poll until the run finishes); `--wait` is kept + // as an alias because fetch/extract use `--wait ` for a different meaning. + const follow = values.follow === true || values.wait === true; const json = ctx.json || values.json === true; const file = positionals[0]; if (!file) throw needFile(); + // Capability gate first — a beta-disabled backend never gets a spec read. + assertUsable("batch"); + const v = validateJsonl(file); if (v.errors.length) { log.warn(`${v.errors.length} invalid line(s) — fix them before submitting:`); @@ -120,6 +133,15 @@ async function createCmd(rest: string[], ctx: RunContext): Promise { }); } + // Pre-flight governance — all local, before any network call: + // 1. every task URL must pass the allow/deny domain policy, + // 2. the run must fit the batch-only page/credit caps (a batch is the one + // primitive that fans out into many requests, so the caps bind here). + const policy = loadPolicy(); + for (const job of v.jobs) assertDomainAllowed(job.url, policy); + const est = estimateCredits(v.jobs); + assertWithinLimits({ pages: v.validJobs, credits: est.credits }, policy, "batch"); + const jobParams: Record = {}; if (values["js-render"] === true) jobParams.js_render = true; if (values["premium-proxy"] === true) jobParams.premium_proxy = true; @@ -130,18 +152,54 @@ async function createCmd(rest: string[], ctx: RunContext): Promise { // toJobBody validates proxy_country/premium_proxy BEFORE any HTTP call. const body = toJobBody(v.jobs, jobParams); - const apiKey = requireApiKey(); + const apiKey = await ensureApiKey( + values["no-signup"] ? { ...policy, auto_signup: false } : policy, + { + onProvision: (a) => { + log.info("No API key found — created a Zenrows Free plan account for you."); + log.dim(`Claim it anytime (keeps your usage): ${a.claimUrl}`); + }, + }, + ); - log.step(`Submitting batch job (${body.tasks.length} tasks)…`); - const job = await createJob(body, { apiKey }); - const finished = values.wait === true ? await waitForJob(job.job_id, { apiKey }) : job; - printJob(finished, json, `Submitted job ${job.job_id}`); - return 0; + const runId = newRunId(); + const startedAt = new Date().toISOString(); + log.step(`Submitting batch job (${body.tasks.length} tasks, ~${est.credits} credits)…`); + try { + const job = await createJob(body, { apiKey }); + const finished = follow ? await waitForJob(job.job_id, { apiKey }) : job; + const runDir = writeRun({ + runId, + command: "zenrows batch create", + capability: "batch", + startedAt, + finishedAt: new Date().toISOString(), + status: "ok", + request: { file, tasks: body.tasks.length, estimatedCredits: est.credits, jobParams }, + result: { jobId: job.job_id, status: finished.latest_run?.status ?? "unknown" }, + }); + printJob(finished, json, `Submitted job ${job.job_id}`); + if (runDir && !json) log.dim(` artifact: ${runDir}`); + return 0; + } catch (err) { + writeRun({ + runId, + command: "zenrows batch create", + capability: "batch", + startedAt, + finishedAt: new Date().toISOString(), + status: "error", + request: { file, tasks: body.tasks.length, estimatedCredits: est.credits, jobParams }, + error: err instanceof ToolkitError ? err.toJSON() : { message: String(err) }, + }); + throw err; + } } async function statusCmd(rest: string[], ctx: RunContext): Promise { const { positionals } = parse(rest, {}); const id = requireId(positionals[0]); + assertUsable("batch"); const apiKey = requireApiKey(); const job = await getJob(id, { apiKey }); printJob(job, ctx.json, `Job ${id}`); @@ -158,6 +216,7 @@ async function resultsCmd(rest: string[], ctx: RunContext): Promise { const json = ctx.json || values.json === true; const id = requireId(positionals[0]); const status = normalizeResultStatus(asString(values.status)); + assertUsable("batch"); const apiKey = requireApiKey(); const rows = await listResults(id, { apiKey, status }); @@ -220,6 +279,7 @@ async function resultsCmd(rest: string[], ctx: RunContext): Promise { async function cancelCmd(rest: string[], ctx: RunContext): Promise { const { positionals } = parse(rest, {}); const id = requireId(positionals[0]); + assertUsable("batch"); const apiKey = requireApiKey(); const job = await stopJob(id, { apiKey }); printJob(job, ctx.json, `Stopped job ${id}`); @@ -230,6 +290,7 @@ async function waitCmd(rest: string[], ctx: RunContext): Promise { const { values, positionals } = parse(rest, { timeout: { type: "string" }, json: { type: "boolean" } }); const json = ctx.json || values.json === true; const id = requireId(positionals[0]); + assertUsable("batch"); const apiKey = requireApiKey(); const job = await waitForJob(id, { apiKey, timeoutMs: asNumber(values.timeout) }); printJob(job, json, `Job ${id} finished`); @@ -239,6 +300,7 @@ async function waitCmd(rest: string[], ctx: RunContext): Promise { async function retryCmd(rest: string[], ctx: RunContext): Promise { const { positionals } = parse(rest, {}); const id = requireId(positionals[0]); + assertUsable("batch"); const apiKey = requireApiKey(); // "Reruns and retrying failures": POST /jobs/{id}/rerun?status=failed replays // only the failures; already-successful tasks carry over. @@ -284,8 +346,19 @@ function normalizeOutput(v?: string): string | undefined { pdf: "pdf", html: "", // raw HTML is the default; no response_type }; - const mapped = map[v.toLowerCase()]; - return mapped ? mapped : undefined; + const key = v.toLowerCase(); + if (!(key in map)) { + // Fail loudly rather than silently dropping an unrecognized format (house rule). + throw new ToolkitError({ + code: "INVALID_USAGE", + message: `Unknown --output format: ${v}.`, + likely_cause: "Only markdown | plaintext | pdf | html are supported for batch --output.", + next_action: "Use --output md|markdown | text|plaintext | pdf | html (html = raw HTML, the default).", + suggested_commands: ["zenrows batch create jobs.jsonl --output markdown"], + }); + } + const mapped = map[key]; + return mapped ? mapped : undefined; // html → undefined (no response_type) } function requireId(id: string | undefined): string { diff --git a/src/cli/commands/browser.ts b/src/cli/commands/browser.ts index 1b36e83..b6d6d8a 100644 --- a/src/cli/commands/browser.ts +++ b/src/cli/commands/browser.ts @@ -1,43 +1,626 @@ /** - * `zenrows browser` — Interact / Browser Sessions (status: experimental). + * `zenrows browser` — Browser Sessions (status: available). * - * Browser is an ESCALATION layer, not the default. There is no documented REST - * "sessions" API; Zenrows exposes the Scraping Browser (CDP) and browser_* - * tools via the @zenrows/mcp server. So this command is gated behind - * policy.allow_browser and points users to those surfaces rather than faking a - * sessions API. + * Browser is an ESCALATION layer, not the default. It drives the Zenrows managed + * REST session API (https://mcp.zenrows.com/browser/sessions/*, Bearer auth) — + * the same backend the official @zenrows/mcp server uses — so no CDP client or + * browser dependency is needed. On by default; opt out via policy.allow_browser=false. + * + * Session model: `open` creates a server-side session (short TTL + idle + * timeout) and prints a session_id; subsequent verb subcommands take + * `--session `; `close` frees it. For multi-step flows prefer + * `run ` (one process, auto-closes). `connect` prints the direct + * CDP wss URL for users who want to drive it with their own Playwright/Puppeteer. + * + * Sessions bill by bandwidth + session time, so always close them. */ -import { loadPolicy, assertBrowserAllowed } from "../../core/policy.ts"; -import { getCapability } from "../../core/capabilities.ts"; +import { readFileSync } from "node:fs"; +import { ensureApiKey } from "../../core/ensure-key.ts"; +import { assertUsable } from "../../core/capabilities.ts"; +import { loadPolicy, assertBrowserAllowed, assertDomainAllowed } from "../../core/policy.ts"; import { log, ANSI, c } from "../../core/logger.ts"; -import { type Command, type RunContext } from "../command.ts"; -import { printError } from "../output.ts"; +import { newRunId, writeRun } from "../../core/artifacts.ts"; +import { ToolkitError } from "../../core/errors.ts"; +import { parse, asString, asNumber, type Command, type RunContext } from "../command.ts"; +import { printError, writeOut } from "../output.ts"; +import { + createSession, + sessionCall, + closeSession, + decodeBinary, + connectUrl, + normalizeSelectValue, + type SessionProxyOpts, +} from "../../core/browser-api.ts"; + +type ParseOpts = Record; + +/** + * Generic session verbs: subcommand → REST verb. Each declares its extra flags, + * required flags, a body builder, the HTTP method (default POST), and — for + * text-returning verbs — the field to print raw in non-JSON mode. + */ +interface VerbSpec { + verb: string; + method?: "GET" | "POST" | "DELETE"; + opts?: ParseOpts; + required?: string[]; + body?: (v: Record) => Record | undefined; + textField?: string; +} + +const VERBS: Record = { + navigate: { verb: "navigate", opts: { url: { type: "string" } }, required: ["url"], body: (v) => ({ url: asString(v.url) }) }, + "go-back": { verb: "go_back" }, + "go-forward": { verb: "go_forward" }, + reload: { verb: "reload" }, + click: { verb: "click", opts: { selector: { type: "string" } }, required: ["selector"], body: (v) => ({ selector: asString(v.selector) }) }, + hover: { verb: "hover", opts: { selector: { type: "string" } }, required: ["selector"], body: (v) => ({ selector: asString(v.selector) }) }, + check: { verb: "check", opts: { selector: { type: "string" } }, required: ["selector"], body: (v) => ({ selector: asString(v.selector) }) }, + uncheck: { verb: "uncheck", opts: { selector: { type: "string" } }, required: ["selector"], body: (v) => ({ selector: asString(v.selector) }) }, + focus: { verb: "focus", opts: { selector: { type: "string" } }, required: ["selector"], body: (v) => ({ selector: asString(v.selector) }) }, + type: { + verb: "type", + opts: { selector: { type: "string" }, text: { type: "string" }, "clear-first": { type: "boolean" } }, + required: ["selector", "text"], + body: (v) => ({ selector: asString(v.selector), text: asString(v.text), clear_first: v["clear-first"] === true }), + }, + fill: { + verb: "fill", + opts: { selector: { type: "string" }, value: { type: "string" } }, + required: ["selector", "value"], + body: (v) => ({ selector: asString(v.selector), value: asString(v.value) }), + }, + select: { + verb: "select", + opts: { selector: { type: "string" }, value: { type: "string" } }, + required: ["selector", "value"], + // Backend matches() each