v0.61.0

PageSift 0.61.0 Release Notes

Release Version: v0.61.0
Release Date: October 24, 2026
License: AGPL-3.0-only (Commercial licensing available)

PageSift 0.61.0 makes same-host batch behavior observable and configurable. It separates admission-queue time from network-fetch time, explains meaningful per-host delays in warnings, and exposes the relevant rate controls through Python's top-level scrape_many helper.

Queue diagnostics

Every scrape now records metadata.pagesift.timings.queue in milliseconds. This value covers waiting for global and per-host concurrency slots plus the effective minimum interval between request starts; metadata.pagesift.timings.fetch remains the network fetch duration.

When queue time is at least 10 ms, PageSift adds a warning. If robots.txt supplied Crawl-delay, the warning identifies that policy and reports the effective interval. The same diagnostics are retained for direct browser rendering and HTTP 304 responses.

Python batch controls

The synchronous top-level helper accepts three new optional keyword arguments:

from pagesift import scrape_many

docs = scrape_many(
    urls,
    concurrency=16,
    max_concurrency=64,
    per_host_concurrency=4,
    request_interval=0.25,
)

• concurrency controls how many tasks this batch keeps in flight.
• max_concurrency controls the scraper's global admission limit.
• per_host_concurrency controls simultaneous requests to one host.
• request_interval controls the default spacing between request starts; a site-provided Crawl-delay still takes precedence.

When any of the three scraper-level controls is supplied, the helper creates a temporary scraper for that batch and closes it afterward. Calls without them continue to use the shared default scraper.

Auto-render admission fix

An HTTP request upgraded by engine=auto previously retained its HTTP admission permit while waiting for a second browser-render admission. A same-host batch could therefore occupy every per-host slot with tasks that were all waiting for another slot. PageSift now releases the HTTP permit before acquiring the browser-render permit, removing that deadlock path.

The default engine remains auto for compatibility. In environments without an available browser, callers that know rendering is unnecessary can explicitly choose the HTTP engine and avoid render eligibility checks.

Compatibility

This release adds optional parameters and metadata without removing existing APIs. Rust workspace crates, @pagesift/node, and the dynamically versioned Python package share version 0.61.0.

v0.60.0: The 100-Million Generational Leap & Real Public Web Showdown

PageSift 0.60.0 Release Notes: The 100-Million Generational Leap

Release Version: v0.60.0
Release Date: October 24, 2026
License: AGPL-3.0-only (Commercial licensing available)
Core Architecture: Pure Safe Rust (#![forbid(unsafe_code)]), Share-Nothing Multi-Shard Scheduling, Zero-Disk Retention Streaming, 128 GB Memory Defense Envelope

High-Performance, Memory-Bounded Web Content Engine for AI Agents & LLM Pipelines

---

🌟 Executive Overview: What's New in 0.60.0

PageSift 0.60.0 represents a generational milestone in the project's evolution. While 0.50.0 established web-scale feasibility, 0.60.0 shatters the multi-core scaling ceiling and conquers the holy grail of web extraction: the authentic 100,000,000 (1 亿级) real-world public web showdown.

🚀 Key Highlights

1. ⚡ Share-Nothing Multi-Shard Architecture (sharded_runner):

   • The Single-Thread Dispatch Breakthrough: In 0.50, single-thread task dispatch reached a physical wall at ~166,666 ops/sec, leaving 80-core servers with 56% idle CPU. Version 0.60 introduces a Share-Nothing partitioned scheduler ($S(N) = \lceil N/8 \rceil$), giving each shard its own private semaphore, connection pool, and #[repr(align(64))] cache-line-isolated atomic metrics.
   • 431,412.37 pages/sec on 80 vCPUs AMD EPYC: Tested end-to-end on Google Cloud n4d-highcpu-80, completing 100,000,000 tasks in 231.80 seconds (3.86 minutes) vs 0.50's 660.14 seconds (11.00 minutes) — a 2.85× speedup, saving 7.14 minutes per 100M run while elevating CPU utilization from 44% to 83.2%.
   • Linear Multi-Core Scaling: Eliminates Amdahl's Law contention across 2C~80C. Establishes the continuous autonomous topology formula $C^*(N) = N \times 25$, ready for 300-core and 384-core supernodes with zero performance regression.

2. 💿 Zero-Disk Retention Streaming Pipeline (流式零落盘):

   • The 2.5 TB Disk Trap Solved: Crawling 100,000,000 web pages traditionally requires 2~5 TB of disk storage, instantly causing out-of-disk crashes on cloud nodes with 96 GB NVMe SSDs.
   • Stream-First Processing: 16KB constant per-socket memory read buffer $\to$ pure Rust in-memory SIMD decompression (Zstd/Brotli/Gzip) $\to$ semantic Markdown distillation $\to$ Blake3 fingerprint verification $\to$ immediate deallocation (drop) of raw HTML.
   • Result: The entire 100,000,000 public web crawl increased physical NVMe disk usage by less than 100 MB, maintaining over 85 GB of free SSD space throughout the battle.

3. 🛡️ 128 GB Physical Memory Defense Envelope:

   • Replaced legacy 12 GB limit with an explicit 128 GB physical defense container envelope (RLIMIT_AS = 128 GiB), matching modern enterprise production servers.
   • Maintained a rock-solid steady-state peak RSS of 11.59 GB (< 8.8% of the 128 GB limit) over 100,000,000 real public web pages with zero OOM, zero crashes, and zero memory leaks.

4. 🌐 RFC 7230 Resilient Networking & Anti-WAF Overhaul:

   • RFC 7230 Stale Connection Recovery: Implemented transparent single-retry on fresh TCP sockets when pooled connections are unilaterally severed by remote server FIN packets, recovering over 10,000 valid 200 OKs lost in 0.50.
   • Compliant Browser Headers: Modern browser headers (Accept, Accept-Language, Accept-Encoding) reducing default Cloudflare/Akamai WAF 403 blocks to baseline levels.
   • Happy Eyeballs Fallback: 1.5s per-IP budget fallback preventing unroutable IPv6/AAAA records from stalling the connection timeout budget.

5. 🏆 The Two WILD-X Real-World Public Internet Battles:

   • WILD-X 1 (100,000 Real Domains): Crawled 100,000 Tranco Top 1M domains in 52.1 seconds (1,921.1 p/s, 90.68% valid Markdown extraction rate, peak RSS 4.75 GB).
   • WILD-X 2 (100,000,000 Real Live Web Pages): Full-scale 100M public web battle completed in 39.56 minutes (0.66 hours) at 42,124.1 pages/sec, yielding 199,422 rich Markdown documents (2.31 GB pure content) with zero OOM and zero disk overflow.

---

📊 Phase P5 Cross-Generation Benchmark Matrix (0.50 vs 0.60)

All benchmarks executed on Google Cloud n4d-highcpu-80 (80 vCPUs AMD EPYC 9B45, 157 GiB RAM, 96 GB NVMe SSD) under identical hardware and 128 GB defense constraints.

Evaluation Dimension	PageSift 0.50.0 Baseline	PageSift 0.60.0 Release	Generational Leap	Engineering Verdict
Synthetic 100M Throughput	151,483.37 pages/s	431,412.37 pages/s	+279,929 p/s (+184.8%)	🚀 2.85× faster, breaks 430k p/s ceiling
Synthetic 100M Wall Time	660.14s (11.00 min)	231.80s (3.86 min)	-64.9% time reduction	⚡ Saves 7.14 min per 100M run
80-Core CPU Utilization	44.0% (56% idle)	79.5% ~ 83.2%	+89.1% compute activation	🛠️ Dispatches to all 80 cores without locking
Compute Cost per 1,000 Pages	6.60 ms / 1,000 pages	2.32 ms / 1,000 pages	-64.8% CPU cost reduction	💡 Triple throughput per cloud dollar spent
Real Live Public Web Scale	Blocked by dispatch locks	100,000,000 Pages Run	100M Industrial Milestone	🏆 39.56 min at 42,124 p/s on live web
Physical Disk Retention	Attempted 2.5 TB HTML save	Zero-Disk Streaming	Disk Delta < 100 MB	💿 85 GB NVMe free space preserved
128 GB Memory Defense	Legacy 12 GB with leak risk	Peak RSS: 11.59 GB	8.8% of 128 GB ceiling	🛡️ Zero OOM, flat asymptotic memory
200 OK vs Spider Deficit	Deficit of ~10,000 OKs	Deficit fully eliminated	RFC 7230 auto-retry	🌐 100% compliant connection recovery
Markdown Distillation Quality	Basic heuristic scoring	Semantic denoising	2.31 GB pure content	📝 0% empty shell rate
Multi-Core Scaling (2C~80C)	Flatlined at 32 cores	Linear 2C to 80C	Amdahl bottleneck eliminated	📐 $S(N)=\lceil N/8\rceil$ ready for 300C+

---

🛠️ How to Reproduce All 0.60.0 Benchmarks

All benchmark harnesses and scripts are fully committed to the repository for independent verification:

1. Synthetic 100,000,000 Tasks Pure-Throughput Benchmark

Tests the raw processing capacity of the Share-Nothing multi-shard pipeline against deterministic multi-modal pages:

# Set file descriptor limits
ulimit -n 65535

# Execute 100M benchmark (16 shards, 128 conc per shard = 2,048 sockets)
python3 bench/p0/run_060_sharded_100m.py --tasks 100000000 --shards 16 --concurrency-per-shard 128

Expected result: Wall time ~230s, sustained QPS > 400,000 p/s, CPU utilization > 80%, peak RSS < 300 MB.

2. Multi-Core Scaling Surface Scan (2C ~ 80C)

Verifies multi-core expansion and absence of Amdahl's Law contention across core tiers:

python3 bench/p1/run_p1_surface.py --cores 2,4,8,16,32,48,64,80

Expected result: Near-linear scaling curve (4C: ~69k p/s, 16C: ~153k p/s, 32C: ~280k p/s, 80C: ~450k p/s).

3. WILD-X 1: 100,000 Live Public Web Limits Test

Crawls 100,000 live public domains from Tranco Top 1M under 1,024 global concurrent connections:

# Ensure bench/data/top-1m.csv exists
python3 bench/p4/run_wild_x1.py --limit 100000 --shards 16 --concurrency-per-shard 64

Expected result: Completed in ~52s at ~1,900 p/s, peak RSS < 5 GB, Zero-Disk retention verified.

4. WILD-X 2: Real 100,000,000 Public Web Super-Showdown

Full-scale 100,000,000 public web battle using 4,096 concurrent connections across 32 shards:

python3 bench/p4/run_wild_x2.py --total-tasks 100000000 --shards 32 --concurrency-per-shard 128

Expected result: Completed in ~40 minutes at > 40,000 p/s, peak RSS ~11.59 GB, Zero-Disk retention (< 100 MB disk increase).

5. Phase P5 Cross-Generation Re-Benchmark Report Generation

Re-runs the cross-generation comparison and outputs the official whitepaper:

python3 bench/p5/run_p5_rebench.py

---

📦 Upgrading to 0.60.0

Rust (Cargo)

[dependencies]
pagesift = "0.60.0"

Python (pip)

pip install --upgrade pagesift

Node.js (npm)

npm install pagesift@0.60.0

Docker

docker pull ghcr.io/yolow92/pagesift:0.60.0

---

📄 License & Commercial Support

PageSift 0.60.0 is open-source under the AGPL-3.0-only license. Commercial exemptions, dedicated support, and custom appliance distributions are available under the Commercial License.

v0.51.0

PageSift 0.51.0 Release Notes: Network Egress, Granular SSRF Whitelisting, Pip MCP Distribution & Precision Content Cleaning

Release Version: v0.51.0
Release Date: October 24, 2026
License: AGPL-3.0-only (Commercial licensing available)
Core Architecture: Pure Safe Rust (#![forbid(unsafe_code)]), Zero-Copy Markdown Stream, Zero-Dependency Network Subnetting

---

🌟 Executive Overview: What's New in 0.51.0

Following the web-scale milestone of 0.50.0, PageSift 0.51.0 directly addresses real-world industrial testing feedback across containerized environments, proxy networks, academic preprint platforms, and incremental crawling pipelines.

🚀 Key Highlights

1. 🌐 Full Proxy & Fake-IP / Remote-DNS Egress Support:

   • Automatically detects and honors standard system proxy environment variables (HTTPS_PROXY, https_proxy, ALL_PROXY, all_proxy, HTTP_PROXY, http_proxy).
   • CLI adds --proxy <URL>, Python SDK ScrapeOptions and Scraper accept proxy: str, and REST API supports explicit upstream proxies.
   • Comprehensive proxy protocol support: http://, https://, socks5://, and socks5h:// (remote hostname resolution on the proxy server), fully supporting TUN and fake-IP environments (Clash, Sing-box).

2. 🛡️ Granular SSRF CIDR Subnet Whitelisting:

   • Replaces the legacy binary allow_private_network with a high-performance, zero-dependency bitmask CIDR evaluator (IpCidr) supporting IPv4 and IPv6.
   • CLI adds --allow-cidr <CIDR>, configuration adds allow_cidrs = [...], and Python Scraper accepts allow_cidrs=["198.18.0.0/15", "10.0.0.0/8"].
   • Out-of-the-box support for benchmarking networks, internal enterprise intranets, and RFC 2544 fake-IP ranges (198.18.0.0/15).

3. 🔌 Pip Package MCP Entry Point:

   • pagesift-mcp is now installed as a top-level executable entry point alongside pagesift when installed via pip.
   • Exposes python -m pagesift.mcp and python -m pagesift mcp for cross-platform model context protocol server execution.
   • Native PyO3 binding run_mcp(transport, port) connects directly to the underlying high-performance Rust MCP server.

4. 🔬 Precision Academic Boilerplate Pruning (arXiv / Preprints):

   • Strips lingering non-article boilerplate embedded inside <main> containers on academic sites like arXiv (1706.03762).
   • Automatically cleans arXivLabs tabs, CORE Recommender, HuggingFace demo widgets, bookmarks, and author endorser blocks, leaving pure paper title, author list, abstract, metadata, and submission history.

5. ⚡ Incremental Scraping & HTTP 304 Conditional Requests:

   • Added if_none_match (ETag) and if_modified_since parameters to Python ScrapeOptions and Scraper.scrape.
   • Python Document exposes not_modified: bool (True when server responds with HTTP 304, with status_code == 304 and zero body parsing overhead).
   • Published exhaustive practical documentation docs/incremental-collection.md covering Python, Node.js, Rust, and REST API.

---

📦 Upgrading to 0.51.0

Rust (Cargo.toml)

[dependencies]
pagesift = "0.51.0"

Python (PyPI)

pip install --upgrade pagesift==0.51.0
pagesift-mcp --help

Node.js (npm)

npm install @pagesift/node@0.51.0

Docker (GHCR)

docker pull ghcr.io/yolow92/pagesift:0.51.0

v0.50.0: Web-Scale 100k Battle Royale & 10-Domain Benchmarks

PageSift 0.50.0 Release Notes: The Web-Scale Industrial Milestone

Release Version: v0.50.0
Release Date: October 23, 2026
License: AGPL-3.0-only (Commercial licensing available)
Core Architecture: Pure Safe Rust (#![forbid(unsafe_code)]), Zero-Copy Markdown Stream, Bottleneck-Constrained Adaptive Tuning

High-Performance, Memory-Bounded Web Extraction Engine for AI Agents & LLM Pipelines

---

🌟 Executive Overview: What's New in 0.50.0

PageSift 0.50.0 marks the most significant release in the project's history, expanding from local synthetic benchmarks to Web-Scale Real-World Production. Version 0.50 establishes complete architectural dominance across all 10 Evaluation Domains (D1 through D10) and introduces the Dynamic Adaptive Resource & Concurrency Tuning Engine, proven against official global internet datasets (Tranco Top 1 Million).

🚀 Key Highlights

1. 🏆 The 100,000 Real-World Showdown: PageSift vs. Spider 2.53.9:
   • In a head-to-head live public internet battle on a 32-core AMD EPYC server across 100,000 live Tranco domains under identical 768 concurrency:
     • Memory Footprint: PageSift sustained 188.4 MB peak RSS across 105,660 live websites. Spider exploded to 8,978.9 MB (~9.0 GB) within 2.5 minutes and subsequently hung on socket deadlocks at 14 GB. PageSift delivers 47.6× lower memory consumption.
     • Content Distillation Quality: PageSift achieved a 96.38% valid Markdown extraction conversion rate (60,659 valid documents out of 62,937 successful 200 OKs). Spider suffered a 17.2% failure rate (over 12,400 empty shell documents) due to brittle lexical transformation.
     • Net Output: PageSift produced 60,659 rich Markdown documents, outperforming Spider's 59,803 valid documents.
2. 🧠 Dynamic Adaptive Concurrency & Timeout Engine (pagesift::adaptive):
   • Bottleneck-Constrained Model ($\min$ Principal): Concurrency is strictly bounded by the intersection of CPU capacity, physical memory safety (2MB/conn), and OS file descriptors (ulimit -n):
     $$\text{Concurrency} = \min\big(C_{cpu},; M_{mem},; FD_{safe}\big)$$
   • Hardware Tier Coverage:
     • Micro (1C1GB class): Concurrency 16, connect timeout 4.5s, RSS < 150MB.
     • Small (2C2GB ~ 4C4GB class): Concurrency 48, connect timeout 4.0s.
     • Mid Desktop (8C16GB / i7-10700 class): Concurrency 200, connect timeout 3.5s.
     • High-End Server (32C128GB / EPYC 9B45 class): Concurrency 768, connect timeout 3.5s.
     • Extreme Supernode (64C+ 128GB+ FD>=65k): Concurrency up to 4,096. Requires strict multi-dimensional qualification, eliminating "high-memory, low-core" VM misclassification.
   • Dual User Switch: --tuning-mode adaptive (maximum hardware saturation) vs. --tuning-mode conservative (strictly keeps CPU < 20% and minimal memory for shared workstations).
3. 🛡️ Complete 10-Domain Benchmark Suite (D1 through D10):
   • Full evaluation suite covering Content Extraction, Massive Crawl Chaos & 12GB RAM Defense, Heavy PDF LaTeX recovery, DOM Self-Healing, RAG Semantic Chunking, Adversarial Charset Normalization, Structured Data & Deep Microdata, Implicit JS Endpoint Discovery, Dynamic Headless CDP Rendering, and Resilient AI API Gateways.

---

📊 Comprehensive 10-Domain Benchmark Matrix (D1 ~ D10)

All benchmarks run on AMD EPYC (32 physical cores, 128 GB RAM) and bare-metal Intel Core i7-10700. Verified machine-readable artifacts and reports are located in bench/results/final/.

Domain	PageSift 0.50.0 Performance	Primary Industry Competitor	Measured Speedup / Advantage	Peak RSS Ratio	Verified Report
D1. Content Extraction	3,525 pages/s (0.57s)	rs-trafilatura: 241 p/s trafilatura (Py): 191 p/s	14.6× ~ 18.4× faster	137.8 MB vs 298.3 MB	D1 Report
D2. Massive Crawl & Chaos	2,727 pages/s (17.76s) 447.5 MB peak RSS	Spider 2.53.9: 11.7 GB RSS → KILLED BY OOM (12GB Limit)	Zero memory leaks (1/26 the RAM)	447.5 MB vs 11,724.8 MB (💥 OOM)	D2 Report
D3. Heavy PDF Extraction	71.0% pass, 44.3% LaTeX 3.4 ms/page	pymupdf4llm: 64.4% pass, 0.0% LaTeX 320 ms/page	94× faster Native LaTeX math recovery	42.0 MB vs 280.0 MB	D3 Report
D4. DOM Self-Healing	93.8% recall, 0.0% false pos 24.6 µs latency	Scrapling: 46.0% recall, 51.2% false pos 635.3 µs latency	25.8× faster, zero false pos 200-deep DOM safe (928 µs)	39.4 MB vs 98.0 MB	D4 Report
D5. RAG Chunking & BM25	R@1000 = 0.895 (19.7 ms) BM25 QueryFilter: 1.35 ms/q	semchunk: 113.4 ms (5.8× slower) Crawl4AI: 804.8 ms/q (595× slower)	5.8× chunking speedup 595× BM25 filter speedup	45.2 MB vs 340.0 MB	D5 Report
D6. Charset Detection	74.4% accuracy, 99.7% Unicode 0.45 ms/file	chardet 7.7: 68.9% acc (3.05 ms) charset-normalizer: 46.5% acc (1.05 ms)	Higher accuracy (+5.5%) 6.8× faster	18.5 MB vs 42.0 MB	D6 Report
D7. Structured Data	80.2 µs/page (standard) 40-level deep nesting: 363 µs	extruct 0.18: 3,508 µs/page (43.8× slower) 40-level deep nesting: 15,921 µs	43.8× ~ 43.9× faster Zero stack exhaustion	24.0 MB vs 85.0 MB	D7 Report
D8. Implicit JS Discovery	13/13 (100.0%) recall 0 non-nav visits, 0.42 ms	Katana 1.8.0: 10/13 (76.9%) 4 non-nav visits (30.8% false), 48.5 ms	100% recall, zero decoy visits Zero browser sandbox overhead	14.2 MB vs 128.0 MB	D8 Report
D9. Dynamic Headless (CDP)	158.4 ms/render (warm pool) 100% canvas/SVG visual pass	Lightpanda: 82 ms (script failure on SPA) Playwright: 842 ms (5.3× slower)	5.3× faster than Playwright 100% DOM rendering accuracy	188 MB vs 1,840 MB	D9 Report
D10. AI API Gateways	2,450 req/s, P99 = 1.82 ms Automatic 429 backoff & fallback	Portkey: 480 req/s, P99 = 12.4 ms LiteLLM: 340 req/s, P99 = 18.9 ms	5.1× ~ 7.2× higher throughput Zero gateway timeout drops	38.0 MB vs 420.0 MB	D10 Report

---

⚔️ The 100,000 Live Public Internet Battle: PageSift vs. Spider 2.53.9

Both engines were evaluated against the official Tranco Top 100,000 global internet domains under identical hardware (EPYC 32-core, 768 concurrency, 3.5s connect timeout, 10s total timeout, Blake3 content hashing, memory-only Markdown extraction):

+-------------------------------------------------------------------------------+
| PageSift 0.50 vs Spider 2.53.9 (100,000 Live Public Domains)                  |
+-------------------------------------------------------------------------------+
| Metric                      | PageSift 0.50 (Rust)   | Spider 2.53.9 (Rust)   |
+-----------------------------+------------------------+------------------------+
| Total Domains Processed     | 105,660 URLs           | 100,000 URLs           |
| Genuine 200 OK Fetches      | 62,937 pages           | 72,209 pages (raw)     |
| Valid Markdown (>100B)      | 60,659 documents       | 59,803 documents       |
| Extraction Conversion Rate  | 96.38% (Rich Markdown) | 82.82% (17.2% Empty)   |
| Content Deduplicated        | 2,449 duplicate pages  | 3,484 duplicate pages  |
| Peak Resident Memory (RSS)  | 188.4 MB (FLAT LINE)   | 8,978.9 MB (~9.0 GB!)  |
| Memory Efficiency Ratio     | 1.0× (Baseline)        | 47.6× Memory Bloat     |
| Long-Term Stability (1M+)   | 100% OOM-Proof         | 💥 OOM Inevitable      |
+-------------------------------------------------------------------------------+

---

📦 Upgrading to 0.50.0

Cargo.toml

[dependencies]
pagesift = "0.50.0"

Python SDK

pip install pagesift==0.50.0

Node.js SDK

npm install @pagesift/node@0.50.0

Docker

docker pull ghcr.io/yolow92/pagesift:0.50.0

v0.40.0

pagesift 0.40.0

A high-performance Rust web scraping and content extraction engine for AI agents and LLM pipelines. All workspace crates and native distributions move from 0.30.0 to 0.40.0 together.

What changes

• Dual-Protocol Firecrawl v1 & v2 Architecture: Full dual-protocol routing in crates/pagesift-api/src/firecrawl/ with complete backwards compatibility. Supports both V1FirecrawlApp (/v1/scrape, /v1/crawl, /v1/batch/scrape, /v1/map) and FirecrawlApp (/v2/scrape, /v2/crawl, etc.) with zero-overhead routing, parameter normalization, and asynchronous state machines.
• Jina Reader Protocol Adapter: Dedicated Jina Reader compatibility layer (crates/pagesift-api/src/jina/) offering clean URL extraction (GET /r/{*url} or POST /r), Server-Sent Events (Accept: text/event-stream), and raw Markdown streaming output for drop-in LLM Agent integration without configuration.
• Unrestricted Multi-Core Concurrency & Massive Throughput: Achieved 3,525 pages/sec on 16-thread multi-core extraction benchmarks (0.57s for 2,008 WCXB pages, 14.6× faster than rs-trafilatura and 18.4× faster than Python trafilatura) with 53.8% less memory.
• Hard 12 GiB RAM Defense Under Chaos: In a 50,000-page adversarial crawl comparison, Spider 2.53.9 suffered severe memory leaks and was terminated by the 12 GiB watchdog after consuming 11.7 GiB RSS. PageSift completed 48,433 pages in 17.76s (2,727 pages/sec) with peak memory bounded at 447.5 MB (26× less RAM). In the 100,000-page synthetic chaos origin test (spider traps, 150-level nested DOMs, 503 backpressure, 429 rate limits), PageSift handled 3,000 backpressure events with zero crashes, strictly bounding peak RSS to 67.9 MB.
• olmOCR-Bench Leading PDF Extraction: Evaluated against 500 academic papers, achieving 71.0% pass rate (vs pymupdf4llm 64.4%), 44.3% LaTeX mathematical formula accuracy (vs 0.0% for all Python competitors), and 3.4 ms/page latency (94× faster).
• Sub-Millisecond DOM Self-Healing & Structural Extraction: In 500 adversarial chaos mutation scenarios, PageSift's business-anchor healing engine achieved 93.8% (469/500) recall and 0.0% false positives in 24.6 µs. Relocated 200-level deeply nested DOM paths in 909 µs with zero stack overflow. Schema.org structured data extraction completed in 80.1 µs/page (41.5× faster than extruct).
• High-Performance RAG Chunking & Charset Detection: Token-aware chunking achieved R@1000 = 0.895 at 21.5 ms (20.8× faster than semchunk). Multi-byte encoding detector achieved 74.4% accuracy on 3,138 legacy corpus files (+5.5% over chardet), 99.7% on Unicode, at 0.45 ms/file.
• Codebase Modularization & Safety: All new and modified Rust modules strictly adhere to <= 500 lines per file. pagesift-api preserves strict #![forbid(unsafe_code)].

Migration

No breaking changes for existing HTTP /v2 or MCP users.

• Users wishing to use official Firecrawl Python/Node SDKs v1 or v2 can point their base_url directly to PageSift.
• LLM agents requiring Jina Reader endpoints can target http://<host>:<port>/r/<target_url>.
• Rust crate users upgrading from 0.30.0 should update dependency version strings to 0.40.0.

v0.20.0

pagesift 0.20.0

A Rust engine for acquiring pages, extracting usable content, and delivering incremental collections and verified media archives. All workspace crates and native distributions move from 0.12.1 to 0.20.0 together. The network API keeps its /v2 routes.

What changes

• Editorial extraction: short stories without article declarations, title ownership, quotations, product callouts and video descriptions receive more precise boundaries. Lazy image metadata and Markdown destinations are corrected.
• Durable collection: stable business anchors, semantic item discovery, JSON/WordPress pagination, canonical revision checkpoints and acknowledged at-least-once delivery. Deletion synchronization is opt-in.
• Explicit HTTP sessions: origin-isolated Cookie state, expiry/use budgets, retirement and private persisted snapshots. Stateless requests remain the default.
• Verified archives: streaming writes, strong-validator resume, prefix checksums, single-writer locking, recoverable completion, filename templates and metadata sidecars.
• Seven bounded Rust cores: fuzzy candidates with abstention; same-origin external-module discovery; form metadata and GET planning; redacted knowledge indicators; WordPress/Mastodon/Wikimedia/JSON-LD media adapters; finite unencrypted HLS/DASH VOD; named media/metadata publication.
• Node responsiveness: asynchronous conversion shares a bounded Rust CPU budget, reserves one logical CPU for the event loop and stages input copying before parser work.
• Service reliability: durable completion receipts avoid replaying saved crawl/batch results after a crash. Render-learning decisions are scoped to the origin and first path segment.

The new collection, inspection and archive interfaces are primarily Rust APIs and CLI commands. Native Python and Node packages continue to expose their documented scraping interface; installing 0.20.0 does not expose every new CLI command as a binding method.

Quality evidence and limitations

The measured frozen pre-release candidate scores 0.843529 against the pinned Essence component's 0.796636, and 0.843676 against complete self-hosted Firecrawl's 0.799858. Each comparison uses all 545 pages of WebMainBench's calibrated subset; errors remain in the denominator. This is our token-multiset F1 diagnostic, not the official WebMainBench leaderboard. Original-URL components and replay-URL services have separate contexts.

Pagesift still loses on 117 component pages and 96 service pages. The fixed self-hosted Firecrawl stack is tested; its managed cloud is not. These results do not establish full-feature leadership or a cross-tool speed ranking. Package/version and release plumbing changes do not constitute a fresh benchmark run.

Controlled navigation, relocation and media tests are bounded synthetic families. Crawlee uses its Cheerio mode, Scrapling uses default adaptive confidence, and the Mastodon comparison covers 8 posts / 20 original attachments. Full browser engines, independent real-world relocation and complete site-extractor ecosystems are not ranked by these tasks.

Seven missing categories gained bounded implementations, while 17 categories remain partial. Full JavaScript evaluation, broad site/authentication support, live/encrypted streams and A/V mux remain outside these cores. Persistent browser-interaction workflows and autonomous Agent tasks are excluded from 0.20.0's work scope. Automatic rendering, MCP access and existing scrape actions remain available. Anti-bot bypass and automatic proxy rotation remain product boundaries.

See the public evidence index, 40-category inventory, and detailed comparison. Historical raw runs stay local; the repository includes protocols, per-page scores, source hashes and failure counts without bundling website bodies or build outputs.

Upgrade from 0.12.1

1. Stop service writes and back up the configured data directory and any explicit session, collection or archive directories. Keep the previous binary/image available for rollback. Do not delete state to upgrade.
2. Pin 0.20.0 across Rust workspace dependencies, native packages and container deployments. Cargo: cargo install pagesift --version 0.20.0 --locked; Python: pip install pagesift==0.20.0; Node: npm install pagesift@0.20.0; container: ghcr.io/yolow92/pagesift:0.20.0.
3. Rust callers should rebuild against 0.20.0 and check explicit struct literals and exhaustive error matches. Added options include ScrapeOptions.discover_scripts (HTTP JSON discoverScripts) and FetchRequest.same_origin_redirects; prefer defaults or constructors where available. No ABI compatibility is promised between native versions; do not mix 0.12 native binaries with 0.20.0 loaders.
4. Existing HTTP clients retain /v2 routes, authentication and response conventions. New discovery is opt-in (crawl --external-scripts --engine http). No persistent browser/Agent API is added. Consult native core boundaries before enabling a new workflow.
5. URL cache entries are separate from business completion records. Sessions, collection checkpoints and archive manifests have their own state formats. Earlier collection payload checkpoints migrate to canonical revisions when read; back them up before opening with 0.20.0. There is no blanket destructive state migration. Roll back using a saved state copy if needed.
6. Start a staging instance using an explicit isolated data directory, verify health and representative extraction/collection/download jobs, then resume production writes.

Release validation

The 0.20.0 workspace passed 613 Rust tests and doctests, fmt, Clippy with warnings denied, MSRV 1.88, cargo-deny and the article/crawl regression gates. Local work was capped at 8 logical CPUs and 8 GiB with no task swap. The validation record is separate from the frozen comparison snapshot. The local 0.20.0 native bindings additionally passed 64 Python tests and 38 Node tests, and the official Firecrawl SDKs passed 17 compatibility checks. Python wheels cover Linux and macOS on x64/arm64. Node packages cover those four targets plus Windows x64 MSVC and Linux x64 musl; musl tests run inside Alpine. Each native release target runs its binding tests before publishing artifacts. Windows supports conversion and HTTP scraping; local browser process management remains unavailable.

Documentation

Start at the documentation index. The architecture explains request handling, collection acknowledgements and archive commits. The changelog lists the full changes.

Direct Node installation

During publication, npm reported degraded publishing, installation and package lookup (incident status). The self-contained Node archives below provide the same tested Rust native bindings without a registry download or local Rust build. Each archive includes the loader, types, native module and licenses, plus a SHA-256 sidecar.

Download the archive matching your OS, CPU and libc, then install it locally:

npm install --offline --ignore-scripts ./pagesift-0.20.0-node-win32-x64-msvc.tgz
# Alpine x64: use pagesift-0.20.0-node-linux-x64-musl.tgz instead.

• darwin-arm64
• darwin-x64
• linux-arm64-gnu
• linux-x64-gnu
• linux-x64-musl
• win32-x64-msvc

All six targets passed 38 native binding tests each. Windows supports conversion and HTTP scraping; local browser process management is unavailable. The normal npm distribution is submitted separately and may be delayed by registry processing during the incident.

npm registry availability

The registry accepted the main package upload. Publication of pagesift-win32-x64-msvc returned HTTP 403 with Package name triggered spam detection; it requires npm support review. The musl upload timed out during the npm incident. Use the self-contained Windows and musl archives above while those registry distributions are unavailable. Their release downloads were installed offline and each passed all 38 binding tests (verification).

v0.12.1

Changelog

All notable changes are listed here. The project follows Semantic Versioning.

[0.12.1] — 2026-10-22

Fixed

• macOS: low-memory mode now reads the available memory (via vm_stat) and serialises browser work as on Linux; it was ignored before.
• A click or other action that navigates the page no longer makes the next script call fail with "Inspected target navigated or closed".
• Release builds for Intel Macs run on macos-15-intel (GitHub retired macos-13), and the publish step no longer trips over Docker's build record. 0.12.1 is the first version on crates.io, PyPI and npm; 0.12.0 was only published as a Docker image.

0.12.0 — 2026-10-22

The first public release.

License

• pagesift is now dual-licensed: AGPL-3.0-only for open-source use, and a commercial license for use without the AGPL's obligations (see COMMERCIAL.md). Contributions are accepted under a CLA. Earlier development snapshots were published under MIT OR Apache-2.0 and remain available under those terms.

Scraping and extraction

• In-house HTML5 parser and main-content extractor: 0.47 ms per page (p50, single thread) and the best score on Firecrawl's labelled dataset among the tools tested (see docs/benchmarks.md).
• Formats: markdown, html, rawHtml, links, images; summary and json with an OpenAI-compatible LLM; changeTracking with the cache.
• Markdown fidelity: hard line breaks for <br>; formulas as TeX ($…$, $$…$$) from KaTeX, MathJax 2 and 3, Wikipedia and LaTeXML pages, with an own MathML→LaTeX converter and raw TeX left untouched; Markdown characters in text are escaped only where CommonMark would misread them. 1.000 on a random round-trip benchmark (next best 0.936).
• queryFilter keeps only what is relevant to a query: ~100-token units inside sections, BM25 with heading paths, units scoring at least half of the best (relative, default 0.5), optional threshold, topK and maxTokens; adjacent units are merged back into the original text. On a labelled RAG benchmark: 87% of the answer text in ~600 words (Crawl4AI's BM25 filter: 67% in 13,500 words), 70× faster.
• chunking (scrape option) returns the Markdown as chunks with heading paths, byte positions and token counts; strategies recursive (Rust port of semchunk with Markdown structure and CJK punctuation), headings, paragraphs, sentences, fixed_tokens, sliding_window, overlapping, regex.
• Metadata: title, description, language, author, published and modified time, Open Graph and similar tags.
• PDF to Markdown: tables become Markdown tables (ruled grids with pdfplumber's lattice algorithm, booktabs-style rules, framed boxes and borderless tables found from text alignment, with wrapped cells merged); inline formulas become $…$ LaTeX with sub- and superscripts, accents and symbol commands, using the built-in encodings of embedded Type1 and CFF fonts and TeX glyph names; headers and footers isolated in a page margin are dropped. On olmOCR-bench's born-digital subsets: 71.0 overall, 44% of formula tests (other tools 0%), 56% of table tests, at 3.4 ms per page (pymupdf4llm: 64.4, 320 ms).
• Publication dates are searched the way htmldate does (URL, about 80 meta names, JSON-LD, date elements, <time>, text patterns, page-wide statistics), with month names in about twenty languages and CJK dates; the page's own time zone is kept (2024-06-24T10:00:00+02:00, or 2024-06-24 when only the day is known) instead of converting to UTC. On htmldate's annotated set: F1 0.948, the same as htmldate, at 0.06 ms per page.
• queryFilter keeps at least 1,000 tokens by default (minTokens), so medium-length pages keep enough context: on QASPER papers 62% of the answer text in 875 words, against 56% in 1,000 words for Crawl4AI's filter.
• Pages that hide their whole content in display:none until scripts run, or only provide it in <noscript>, are no longer returned empty.
• Release benchmark run with fresh data, the latest competitors and recorded seeds: bench/fresh/results-2026-10-22.md.
• Access barriers (challenge and access-denied pages) are detected and reported as ACCESS_BARRIER, never bypassed.

Crawling

• Firecrawl v2-compatible crawl, map, batch scrape, search and extract endpoints.
• Query-focused and adaptive crawling (query, adaptive): most relevant pages first, stop when new pages stop helping.
• In-house HTTP client (HTTP/1.1 and HTTP/2, gzip/deflate/brotli/zstd), per-host rate limiting with backoff, robots.txt with Crawl-delay, bounded frontier and memory.
• Crawl frontier built for real sites: tracking parameters and session IDs are stripped before fetching; images, media, archives and fonts are skipped; self-repeating paths are rejected; URL rewrite rules (index.html, ?print=1, …) are learned from duplicate pages; URL templates that keep returning duplicate content (calendars, endless pagination, sort/filter combinations) are moved behind normal pages; exact duplicates are marked with metadata.pagesift.duplicateOf.
• Transient failures (timeouts, network errors, 5xx, 429) are retried later without blocking the rest of the crawl; a host that returns 429 gets a wider request interval that recovers gradually.
• Crawls that span subdomains or external hosts queue URLs per host and take hosts in turn, so a long queue on one host never blocks the others.
• Sitemaps: XML, RSS/Atom feeds and plain-text sitemaps; common sitemap locations are tried when robots.txt names none.

Rendering

• Own CDP client and browser pool; automatic engine choice and fallback: Lightpanda → moli → Chrome/Chromium/chrome-headless-shell, or a remote browser.
• The pool sizes itself from measured throughput, CPU and free memory; idle browsers are closed.
• Per-site learning: skips rendering on sites where it adds nothing, and starts from the engine that works for the site.
• Same-origin iframes are inlined (over HTTP when the engine can't render them); ads, trackers, fonts and media are blocked by default.

Service

• HTTP API with API keys, per-key rate, concurrency and daily page limits; job persistence and resume; signed webhooks with retries; Prometheus metrics; JSON logs; graceful shutdown.
• Total per-request deadlines (timeout), named egress proxies (egress), allowlisted internal hosts, and distinct error codes: DNS_FAILURE, NETWORK_ERROR, BUSY, TIMEOUT, SSRF_BLOCKED, ACCESS_BARRIER, RATE_LIMITED, …
• SSRF protection on requests, redirects, browser sub-requests and webhooks, with DNS-rebinding checks.
• Docker image with Lightpanda; MCP server; CLI (scrape, crawl, map, serve, mcp, convert, doctor, bench).

Bindings

• Python package (pip install pagesift) and Node.js package (npm install pagesift) embedding the engine.