v0.4.0

SaaS API

• API Key management: create/validate/revoke with SHA-256 hashing (SQLite)
• Tier-based rate limiting: per-second + daily quota (free=100, pro=5k, biz=50k)
• Bearer token auth on all REST endpoints (opt-in via --with-auth)
• CLI: ue key create/list/revoke + ue serve --with-auth
• Auto-create default free key on first auth-enabled server start

Multi-Channel Captcha Voting

• 3-channel weighted voting: Grayscale (1.0) + Canny edge (2.0) + ddddocr ONNX (0.3)
• _detect_ddddocr() with silent import/model-failure degradation
• Expected single-shot recognition: 70% → ~90%

Scrapling Public API Migration

• Migrated from private StealthySession to public StealthyFetcher.fetch()
• Browser DOM, Canvas, CDP, screenshots all run inside page_action callback
• Removed BrowserPool (StealthyFetcher manages its own lifecycle)
• Net: -676 lines of code

Real-World Validation (19/20 URLs pass)

• Fallback chain fix: >= → > prevents failed stages from blocking
• Canvas Hook: injection moved to page_setup, uses page.add_init_script()
• Canvas Hook: JS syntax fix, main-thread evaluate, pre-collected data check
• Pipeline: don't break chain when pre-collected canvas/cdp data exists
• StageContext.caps: unified capability dict normalization for 8 stages
• Browser DOM: longest selector match + body fallback

README Repositioning

• Hero tagline: Built for the pages other scrapers can't touch
• First screen highlights 3 exclusive capabilities (Canvas, Font, Captcha)

Tests

• 275 → 363 tests, no regressions

v0.3.0

Async Support

• extract_async() on ExtractionStage — default wraps sync in executor, zero breaking changes
• Pipeline.run_async() fully async: parallel search, verify, AND extraction phases
• Crawler.crawl_stream() truly parallel via asyncio.Semaphore
• CrawlConfig.max_concurrent for configurable concurrency

Browser Pool

• BrowserPool class: pre-warm and reuse StealthySession instances
• Thread-safe sync + async APIs, idle timeout eviction, stats monitoring
• BrowserPoolExhausted for clear error handling at capacity

Concurrency Config

• PipelineConfig.enable_concurrent_extraction — parallelize across candidates
• PipelineConfig.max_concurrent_candidates

Integration Tests

• tests/conftest.py with pytest markers (integration, network, slow) + flag gating
• tests/integration/ — real-URL end-to-end pipeline tests
• 275 → 284+ tests

Fixed

• CurlCffiStage.can_handle() string-type crash
• crawl_stream(): time.sleep() → asyncio.sleep()

v0.2.0 — Full-Platform Extraction

v0.2.0 Highlights

• 9-layer adaptive fallback chain — Jina → curl_cffi → Browser DOM → Canvas Hook → CDP → OCR → Vision LLM → Captcha Slider → Font Decrypt
• Docker image — docker compose up -d one-click deploy
• File extraction — PDF, Word, Excel, PPT, EPUB → Markdown
• Deep crawling — BFS/DFS/BestFirst + link scoring + SQLite persistence
• REST API Server — FastAPI + Swagger docs
• MCP Server — Claude/Cursor integration
• Captcha solving — Slider (OpenCV) + Font decryption (fontTools) — no API key needed
• Mobile extraction — ADB screenshot + OCR + auto-scroll
• WeChat Mini Programs — MITM proxy + wxapkg unpacking
• Structured extraction — Schema/Regex/LLM strategies + table extraction
• Plugin system — Third-party extraction stages via entry points
• Adaptive learning — SQLite strategy cache per domain
• 275 tests + ruff 0 + mypy 0 + CI all green
• ADR × 3 — Architecture Decision Records

Full changelog: CHANGELOG.md

v0.1.0

Initial release of Universal Extractor.

Core

• 9-layer fallback extraction: Jina → HTTP → Browser DOM → Canvas Hook → CDP → OCR → Vision LLM → Captcha Slider → Font Decrypt
• Pipeline orchestration engine with search → verify → extract → validate workflow
• WebLens: search + filter + extract integration engine
• Cross-validator for multi-source chapter alignment and merging
• CLI with ue and universal-extractor entry points
• MCP server for AI agent integration

Captcha & Anti-Bot

• Captcha solving: slider, font decryption, and API-based solvers
• Canvas Hook: intercept fillText/strokeText on Canvas elements
• Font decryption: fontTools cmap parsing + ddddocr glyph OCR

Infrastructure

• Strategy cache for per-domain capability profiling (automatic best-layer selection)
• Session manager with persistent browser profiles
• Proxy manager with auto-rotation and health checking
• Rate limiter with adaptive throttling
• Screenshot utilities: viewport capture, dedup, and stitching

Extra

• JD (Job Description) engine with LLM structuring
• Crawler with resume support for multi-page sites