v1.0.0

PageSift 1.0.0 Release Notes

Release Version: v1.0.0
License: AGPL-3.0-only (Commercial licensing available)

Changes since 0.91.1. Measurements are in the README and benchmarks.

---

Changes that can break existing use

HTTP, MCP and Jina

Surface	Before	Now
MCP, all tools	Unknown arguments accepted	Rejected, with every name listed in details.unknownFields
MCP pagesift_scrape and others	Declared screenshot, branding, query, audio, queryOptions, screenshotOptions, pdfOptions, redactPII and similar	Removed where the engine had no matching behaviour
MCP pagesift_crawl	prompt, webhookHeaders, deduplicateSimilarURLs	Removed; adds scope, maxDepth, and webhook as an object
MCP formats objects	Any key accepted	additionalProperties: false; adds intent and modes
/v1, /v2	Unknown fields ignored silently	Still accepted, listed as ignored_fields: a, b in warnings
Extraction schema	Unknown keys ignored	Unknown keys at the top level and in fields return INVALID_REQUEST
Jina headers x-md-*, x-no-gfm, x-keep-img-data-url, x-remove-overlay, x-page, x-base, x-robots-txt	Ignored silently	Values pagesift cannot honor return UNSUPPORTED_OPTION
Warnings without a code	request waited …, response body exceeded …, CSV truncated …, sheet truncated …, columns beyond …, spine item …	Prefixed with queue_wait:, body_truncated:, cells_truncated:, columns_dropped:, epub_item_missing:; the text after the code is unchanged
Scanned PDF warning	scanned PDF: … for a mostly scanned file	scanned_pages_without_ocr: … for the same files; a text PDF with some image-only pages gets some_pages_scanned: …, which does not enter data.warning
Jina headers x-respond-with, x-engine, x-retain-*, x-markdown-chunking, x-respond-timing	Invalid values ignored	Invalid values return an error
Jina headers x-proxy-url with a URL, x-with-generated-alt, x-detach-invisibles	Ignored	Return an error

Python, Node, CLI and configuration

Surface	Before	Now
Python Scraper, scrape_many	request_interval, seconds, float	request_interval_ms, milliseconds, integer
Python ScrapeOptions.replace, **overrides	TypeError naming the first unknown option	TypeError naming all of them
Node, all option objects	Unknown keys dropped; fractions in integer options truncated	Both rejected
CLI relocate	status values candidate, low_confidence, ambiguous, budget_exceeded; fields score, runner_up	status values exact, anchor, fingerprint, missing, ambiguous; fields score, posterior, p, selector, text, html; new --identity ATTR
[[rules]]	Bad include_tags, exclude_tags or schema failed on the first matching request	Fails at startup

Rust

Crate	Before	Now
pagesift_extract	relocation::{ElementFingerprint, relocate, RelocationOptions, RelocateOutcome}, FuzzyAnchor, FuzzyMatch, FuzzyOptions	Removed. Use locate::{Target, Located, locate, locate_list, locate_each, robust_selector, find_similar, Fingerprint} and extract_adaptive
Engine	Scraper::act reported success without executing the action; element had selector	Runs on a real page and returns element with role and name; UNSUPPORTED_OPTION without Chrome
pagesift-llm	ChatResponse { content, finish_reason }; Chinese error messages	Adds usage, so hand-written struct literals need it; English error messages

New

• OCR for scanned PDFs, off by default. [ocr] enabled, quality, formulas, models_dir, [gpu] mode, PAGESIFT_GPU;
  --ocr --ocr-formulas --gpu --models-dir; pagesift models fetch [--quality]; parsers[].mode: "ocr";
  metadata.pagesift.ocr; warnings scanned_pages_without_ocr, some_pages_scanned, gpu_fallback, ocr_failed,
  ocr_incomplete. Results missing OCR text are not cached. Python and Node:
  ocr, ocr_quality / ocrQuality, ocr_formulas / ocrFormulas, models_dir / modelsDir, gpu, and the
  parsers option. Models download on first use and carry open weights under Apache-2.0.
• GPU (NVIDIA): OCR text models run on the GPU, falling back to the CPU with gpu_fallback.
• Selector extraction with memory. {"type": "extract", "schema": …}, {"type": "extract", "intent": …};
  pagesift extract, pagesift similar; POST /v2/similar, POST /v2/observe; Python
  Scraper(store_path=…), Scraper.extract, extract_iter, Document.extract; Node storePath, extractSchema,
  scraper.extract, extractStream, page.extract. Warnings relocated, extract_missing, extract_ambiguous,
  selector_healed, selector_resolved, extract_schema, llm_usage.
• Page cache control in the bindings: Python max_age, Node maxAge.
• Change monitoring. /v2/monitor (alias /v2/monitors), /{id}, /{id}/checks, /{id}/checks/{checkId},
  POST /{id}/run.
• Jina-compatible reader. /r/ maps the Jina request headers (see docs), /s/<query>,
  SSE, chunking, 422 ASSERTION_FAILED, x-pagesift-warnings.
• Browser actions and sessions. actions on HTTP, MCP and Jina; keepSession, POST and
  DELETE /v2/scrape/{id}/interact, pagesift_interact, pagesift_interact_stop; errors FORBIDDEN, SESSION_GONE;
  warnings act_plan, live_view_unavailable, session_replayed.
• Agent. POST /v2/agent, GET and DELETE /v2/agent/{id}, pagesift agent, pagesift_agent,
  pagesift_agent_status; warnings agent_robots_disallowed, agent_unverified_fields, agent_incomplete.
• Browser fingerprint, off by default: [fetch] browser_fingerprint, --browser-fingerprint,
  metadata.pagesift.browserFingerprint, warning fingerprint_fallback.
• WARC input. pagesift scrape x.warc[.gz], pagesift crawl --from-warc FILE, warning archive_records_skipped.
• Hooks and rules. pagesift-hooks, Scraper::builder(config).hook(&hooks).build(); [[rules]],
  [secrets] NAME = { env = "VAR", hosts = ["example.com"] }.
• respondTiming (html … network-idle). --ignore-robots-txt and --max-depth on the CLI; --ignore-robots
  and --depth still work.

Fixed

• Monitor: crawl targets were lost after a restart; a failed crawl reported every old page as removed; a 404
  reported removed on every check; failed checks stretched the interval; background checks did not count against
  the page quota.
• Jina: x-preset: reader always returned 400 unless x-retain-media was set; SSE dropped output options; chunking
  dropped front matter and summaries; /s/ ignored chunking and was not rate limited; x-with-iframe: false still
  forced a browser.
• Extraction: the CLI ran without a store although it advertised relocation; semantic fields called the
  model on every request; act could send a max_tokens above the model's limit.
• data.warning now also carries scanned_pages_without_ocr, ocr_failed, columns_dropped and epub_item_missing.

Known limits

OCR and GPU. An image above about 36.6 megapixels (A4 at 1200 dpi) is not sent to OCR and reports ocr_failed and
ocr_incomplete. Without a request timeout, OCR is bounded per image (120 s to decode, 300 s without a GPU reply), not
per document. On a long-running server the first scanned document returns without OCR text while models download. Only
display formulas are recognized; inline formulas are not. In auto mode a page with a text
layer is not sent to OCR, even when that layer holds little text, so some scans need mode: "ocr". Layout and formula
models run on the CPU, because routing them to the GPU scored lower and ran slower and the cause was not found. The
sm_70 kernels have not run on a real card; only the sm_61 build was exercised, on a GTX 1070. Falling back to the
CPU was tested for a driver that fails to load and for full GPU memory, not for a driver failing in the middle of a run.

Extraction. Relocation covers baseSelector and top-level fields with a selector. Sub-fields of nested, list
and nested_list, source fields and baseFields are not relocated. On a page with very few elements a changed
selector reports extract_ambiguous instead of guessing. MCP over stdio runs without a store, so it has no selector memory; use pagesift mcp http. The bindings cannot configure a model, and find_similar and
observe have no MCP or binding entry. A cached page never triggers relocation; pass maxAge: 0 for the current page.

Actions. A cached plan repeats the element chosen the first time, so an instruction like "click the cheapest" keeps
clicking that element after the page changes. /interact is not available through the cluster gateway and has no live
view. It was not tested with a real model, the Firecrawl SDK, or under real memory pressure.

Security. The browser resolves a host again after pagesift checks it, so DNS rebinding can still reach a private
address through a browser page. Pop-ups and WebSocket connections opened by a page were not tested against the same
checks. Run the server where private addresses are not reachable if you browse untrusted pages. A monitor keeps running
after its API key is removed. headers and x-set-cookie are sent by the HTTP engine only; a browser page reports headers_not_sent.
/interact output does not pass through [[rules]] drop_fields, rename_fields, include_tags or exclude_tags.

Agent. Once a run has read web content, it opens only URLs from the caller, a search, a map or a page it read, plus
URLs without a query string on hosts it already knows, so a page can still have the model request a path on its own
host. model, threadId, mode, location, safe, exchange and zeroDataRetention are ignored and listed in
warnings. Results are stored for 24 hours like any other job, including with zeroDataRetention: true.

Monitor. Monitors are not replicated to a warm standby and are loaded once at startup, so a standby that takes over
has none. Crawl targets make no conditional requests. Unknown fields in targets[].scrapeOptions and schedule are not
listed in ignored_fields. Cron schedules use only the period; the time of day and a non-UTC timezone are ignored and
reported with monitor_time_ignored and monitor_timezone_ignored. The published monitor simulation runs its own copy
of the scheduling logic, not the server's scheduler.

Fingerprint. There is one profile, Linux Chrome. It does not resume TLS sessions, cannot be combined with a proxy,
and fails the handshake if a server selects ML-DSA. It matches Chrome on the JA4 and HTTP/2 fingerprints that were
compared; it was not tested against real anti-bot systems.

Other. WARC export writes no response records. pagesift_policy::session_pool is a library and is not part of
the default fetch path. SinkHook runs only where the writer calls it, which among built-in entries is CLI crawl.
media-idle behaves like network-idle. --from-warc converts pages by host without following links and ignores
maxDepth.

Measurements

PageSift was re-measured on 2026-11-04; other tools keep their recorded numbers, with the date of each. Against 0.91.1,
taking turns on the same machine, quality scores are identical. Local crawl, HTTP and chunking times are within the
spread between runs. Each process uses about 2.5 MB more memory and the binary grew from 30.6 MB to 38.0 MB, because of
the OCR and agent code. Crawl, chunking and Office figures are medians of four rounds. Figures were taken just before
the last robustness fixes to PDF images, OCR and the server; a sample of 30 scanned files gave byte-identical output
before and after those fixes. The live-web stages (A3, wild
crawl) ran over a phone hotspot on 4G. The 10M-request scale run was not repeated for this release.