scrAPE v0.30.0: Distributed Task Leasing, Cloud CAS Sync, and Multimodal VLM DOM Healing

Key Highlights (v0.30.0)

1. Distributed Task Leasing & Autonomous Worker Daemons (Component 1)

• Atomic Idempotency Locking (AC1.1): Upgraded RedisStreamTaskBroker with distributed mutual exclusion locks (SET scrape:completed:{task_id} NX EX 86400). Guaranteed single execution and single disk output across worker restarts, re-claims, and network partitions (verified via unmocked process termination tests).
• Poison-Pill Routing & Dead Letter Stream (AC1.4): Malformed or unprocessable crawl tasks are detected after max retries and automatically shunted to the Dead Letter Stream (scrape:dead_letter), preventing PEL head-of-line blocking.
• Autonomous Worker Daemons (DistributedWorkerNode) (AC1.5 & AC1.6): New daemon (src/core/distributed_worker.py) and standalone CLI entrypoint (src/cli/worker.py) implementing strict write-before-ack ordering, background heartbeats (scrape:workers:heartbeats), active node tracking, and automatic stale consumer garbage collection (AC1.7).
• Adversarial Schema Validation (AC1.3): 84 adversarial fuzz test cases against Pydantic task schemas (src/core/task_schema.py) validating strict rejection of path traversal, SSRF payloads, and invalid execution boundaries.

2. Cloud Content-Addressable Storage (CAS) S3/R2 Synchronization (Component 2)

• Asynchronous Cloud Replication (CASCloudSyncer) (AC2.5 & AC2.6): Bounded spooling queue (maxsize=1000) providing asynchronous block replication to Amazon S3, Cloudflare R2, and MinIO with immediate backpressure (CASQueueFullError) under network saturation.
• Canonical Key Validation & Traversal Immunity (AC2.3): Strict 64-character lowercase hex digest validation (validate_cas_key) mathematically eliminates directory traversal, null-byte injection, and non-hex object key fabrication across local and cloud CAS storage.
• Unconditional SSRF Endpoint Defense (AC2.2): Canonical validate_s3_endpoint_url blocks AWS EC2 metadata (169.254.169.254), GCP metadata (metadata.google.internal), Azure metadata (metadata.azure.com), link-local IPs, private network CIDRs, and default loopbacks.
• Source-Level Credential & Header Redaction (AC2.1 & AC2.4): redact_s3_error strips AWS access keys (AKIA...), secret access keys, presigned signatures (X-Amz-Signature), and basic auth credentials from all logs and error traces. Ephemeral presigned URLs are clamped to ≤900s TTL and kept purely in memory, never written to disk or run_summary.json.
• Zero-Dependency Architecture: boto3 decoupled into optional [cloud] extra; pure local CAS operations run with zero cloud SDKs installed. Verified on clean environments via dedicated CI runner test-base-minimal.

3. Multimodal Vision-Language (VLM) DOM Healing (Component 3)

• Tier 4 Multimodal Self-Healing (VisionDOMHealer): Seamlessly integrated into SelfHealingDOMParser (src/core/self_healing_parser.py) as a fail-safe fallback when rule-based (T1), heuristic (T2), and text LLM (T3) strategies fail.
• Prompt Injection Immunity (AC3.1): System prompts isolate untrusted page content and attribute strings within strict XML boundary tags (<untrusted_scraped_data>). Validated against a 75-vector parameterized fuzzing corpus (PROMPT_INJECTION_ADVERSARIAL_CORPUS) with 100% rejection of system overrides, delimiter breakouts, script/iframe smuggling, and SQL/shell injection payloads.
• Output Parsing & Pseudo-Class Filtering: Strict structured regex validation requiring standard media element prefixes (img, video, source, picture, [data-src]) and blocking dangerous pseudo-classes (:is, :has, :where, :scope, :root).
• Structural Default-Deny Allowlist (AC3.4): is_safe_vlm_interaction_target() enforces a default-deny allowlist accepting only verified media player controls (play, pause, mute, fullscreen) and overlay/cookie dismissals (close, dismiss, accept, reject cookies), rejecting destructive form submissions, checkout buttons, and arbitrary navigational links.
• Live DOM Validation Gate & 7-Day TTL (AC3.5): Repaired selectors must match $\ge 1$ DOM media element before cache persistence; empty or unvalidated selectors are never written to cache. Cached repairs strictly expire after 7 days (MAX_REPAIRED_SELECTOR_AGE_SECONDS = 604800).
• Circuit Breakers & Global Budget Ceiling (AC3.2): DomainVLMTracker trips after 3 consecutive failures for any single domain to fail closed; global session budget ceiling (max_vlm_calls, default 50) halts calls when exhausted.
• Explicit User Consent & Memory Lifecycle (AC3.3 & AC3.6): Hosted third-party providers (Gemini, OpenAI) fail closed unless --vlm-provider-consent is explicitly supplied. ScreenshotContext ensures deterministic memory buffer disposal and aborts under critical host RAM pressure (>90% via psutil). Built exclusively with raw httpx REST calls (0 added base dependencies).

4. QA Validation, Test Matrix & Empirical Proofs

• Full Regression Test Suite:
  • Local Workstation (Windows 11, Python 3.13): 903 passed, 4 deselected, 0 failed in 147.76s (907 collected).
  • GitHub Actions CI Matrix (Workflow Run 35750843737 and 35807593427): All 7 jobs passed (886 passed across all 6 OS/Python runners; 566 passed on dedicated test-base-minimal runner).
• Security Scan (Workflow Run 35807593255): 5/5 security jobs passed (Gitleaks, Bandit, Semgrep, Trivy, OSV-Scanner).
• CodeQL Advanced (Workflow Run 35807593384): Automated gate passed; GitHub Code Scanning REST API verified with [] (0 open alerts).
• Empirical CI Failure Gate Tests:
  • Pull Request #8: Proven automated pipeline block on injected path traversal flaw.
  • Pull Request #9: Proven automated pipeline block on committed cloud access keys.

Full Changelog: v0.29.0...v0.30.0

scrAPE v0.29.0: Master Architecture, Autonomous Scraper & Parquet Pipeline

Full Changelog: v0.28.0...v0.29.0

Full Changelog: v0.28.0...v0.29.0

Full Changelog: v0.28.0...v0.29.0

v0.28.0

Full Changelog: v0.27.0...v0.28.0

v0.27.0

Full Changelog: v0.25.0...v0.27.0

v0.25.0: Responsive Dashboard, Native Low-RAM Profile, and Zero-Mock QA Validation

Release Notes — scrAPE v0.25.0

Release Date: September 20, 2026
Focus: Responsive Dashboard UI, Native Low-RAM Stealth Profile, Dependency & Packaging Reconciliation, Zero-Mock QA Validation.

---

Key Highlights

1. Responsive Brutalist Dashboard & Mobile Touch Targets

• Implemented fluid layout media queries in frontend/templates/index.html:
  • Desktop (≥1440px): Fixed 280px tactical command sidebar with brutalist metrics grid.
  • Tablet (768–1024px): Responsive multi-column layout with flexible wrapping cards.
  • Mobile (≤480px): Collapsible single-column layout with 0 horizontal page overflow.
• All interactive controls, cards, and flags now strictly adhere to WCAG touch-target sizing (min-height: 44px).
• HTMX partial swaps (Seed Studio, Run Inspector, Live Telemetry) verified across all viewports without layout shifts.

2. Native Local Low-RAM Profile (Zero-Docker / Zero-WSL)

• WSL & Docker Overhead Reclaimed: Host memory usage reduced by ~3.7 GB (vmmemWSL excluded).
• ENABLE_FLARESOLVERR_FALLBACK is now explicitly defaulted to False.
• The stealth fallback architecture operates entirely through native local browser and network engines:

  Tier 1: Httpx (Direct / Spoofed Headers)
     ↓
  Tier 2: Curl_cffi (TLS Fingerprint Impersonation)
     ↓
  Tier 3: Crawlee Bridge (Local Node.js 22 + Cheerio / Puppeteer Stealth)
     ↓
  Tier 4: Crawl4AI (Playwright Async Stealth & Heuristic JS Wait)
     ↓
  Tier 5: DrissionPage (CDP-based Chromium Automation)
     ↓
  Tier 6: Helium & Nodriver (Headless / Headful Fallbacks)
     ↓
  Tier 7: Camoufox (Fingerprint-Injected Firefox Engine)
  

3. Dependency & Packaging Hardening

• Crawlee Node.js Bridge: Replaced incompatible stream-json v3.x override with compatible stream-json v1.8.x, ensuring smooth loading and 0 vulnerabilities.
• Python 3.13 Crypto Compatibility: Pinned cryptography (46.0.7) to restore compatibility with pyOpenSSL 25.3.0 and resolve _lib.GEN_EMAIL deprecation errors during heavy SPA JavaScript evaluation.
• Dataset Exporter: Permitted tilde (~) in path sanitization regex, resolving Windows 8.3 short-path exports in temporary directories.
• CLI Launcher: Added -h / --help flag and graceful non-interactive subshell handling to prevent NoConsoleScreenBufferError.

---

QA & Validation Summary

Phase	Description	Result	Details
Phase 1	Responsive Layout Audit	PASS	Desktop (1440px), Tablet (820px), Mobile (375px): 0 overflow, 44px touch targets.
Phase 2	WebUI, CLI & Subsystems	PASS	Seed Studio CRUD, SSRF live matrix (5/5 blocked), CLI wizards, 6/6 dormant ML modules tested.
Phase 3	Domain-Mapped Batch Run	PASS	7/7 seeds completed (apple, hana_bunny, meenfox, eatwaffles, takomayuyi, akariiiii_cos, lionel_messi).
Phase 4	Container Static Audit	PASS	Dockerfile multi-stage, non-root appuser, loopback 127.0.0.1 port bindings.
Phase 5	Security & Native Stability	PASS	0 Bandit High issues (17,865 LOC), 0 bare except:, 0 credential leaks, native tier degradation verified.
Phase 6	Cross-Check Logs	PASS	Zero unhandled tracebacks in logs/ and output/.
Phase 7	Release Readiness	PASS	Version bumped to 0.25.0, changelog synchronized, git tagged.

Full Changelog: v0.22.0...v0.25.0

scrAPE v0.22.0

Release Notes — scrAPE v0.22.0

Release Date: July 29, 2026
Version: v0.22.0
Status: Production Release

---

Highlights of Release v0.22.0

1. 8-Tier WAF Stealth Pipeline & CapSolver Integration

• CapSolver Captcha Token Injection: Automatic detection and solving for Cloudflare Turnstile, reCAPTCHA v2/v3, and hCaptcha sitekeys with direct injection into SessionPool and disk cookie caches (data/sessions/).
• Host Preferred Engine Learning: Dynamic reordering of stealth strategies per host based on historical success rates (_preferred_engine_by_host), avoiding redundant fallback steps.
• Node.js Crawlee Stealth Bridge: Health check integration (CrawleeStrategy.is_available()) auto-spawning the Node.js Express server on port 10002.
• WebUI Stealth Telemetry Card: Real-time #stealth-telemetry-card displaying solve counts, active circuit breaker cooldowns, and engine bindings per domain (/api/telemetry/stealth).

2. AI Dataset Auto-Tagging & Multi-Stage LoRA Quality Export

• Hybrid Vision Auto-Tagger (src/utils/dataset_tagger.py): Opt-in WD14 Booru ViT vision model classification (landscape, portrait, square, highres, lowres) combined with metadata and filename tokenization. Enforces trigger tag placement at index 0 in .txt sidecars.
• Multi-Stage LoRA Quality Gate (src/utils/dataset_exporter.py):
  • Stage 1: Min resolution filter ($\ge 512\text{px}$).
  • Stage 2: Perceptual dHash near-duplicate check ($\text{Hamming distance} \le 4$).
  • Stage 3: Aesthetic score threshold filter ($\ge 5.5 / 10.0$).
• WebUI LoRA Export Modal: Aesthetic Score Threshold selector (export-min-score) in the WebUI export modal and automatic post-scrape sidecar tag generation.

3. Watchdog Multi-Channel Webhook Notifier Architecture

• Pluggable BaseNotifier Interface (src/utils/notification_manager.py): Standard abstract contract with dynamic provider registration (register_provider).
• Supported Providers:
  • TelegramNotifier (wrapping TelegramBotNotifier).
  • DiscordNotifier (rich color-coded JSON embeds).
  • SlackNotifier (Block Kit formatted JSON).
  • CustomWebhookNotifier (generic HTTP POST for Apprise, N8N, Zapier, Matrix, Pushover).
• Parallel Dispatcher: NotificationPipeline thread pool dispatching alerts concurrently.
• WebUI Test Endpoint: POST /api/notifications/test returning per-channel delivery breakdowns.

4. WebUI Live Canvas Physics Visualizer & Code Quality

• Live Crawl Tree Visualizer: HTML5 Canvas force-directed spring graph with search input, depth selectors, and interactive Node Inspector drawer.
• Sonar/IDE Warning Remediation: Cleared all 7 reported IDE warnings in index.html (regex backtracking optimizations, replaceAll() conversion, and rendering loop optimization).

5. Automated Test Suite Metrics

• 310/310 Unit and Integration Tests Passing (100% Success Rate).

---

Migration & API Summary

• New FastAPI Endpoints:
  • GET /api/telemetry/stealth — Stealth pipeline stats, solve counts, engine bindings.
  • POST /api/notifications/test — Multi-channel webhook ping test.
• Modified Query Parameters:
  • GET /api/export/dataset/download/{subject}/{run_id}?repeats=10&min_resolution=512&min_aesthetic_score=5.5
• New Environment Variables:
  • DISCORD_WEBHOOK_URL
  • SLACK_WEBHOOK_URL
  • CUSTOM_WEBHOOK_URL

Full Changelog: v0.20.0...v0.22.0

scrAPE v0.20.0

scrAPE v0.20.0 — SSE Log Streaming, Type Safety, Blacklist Management & Embedded Media Extraction

What's Changed in v0.20.0

• Domain Blacklist System & Dynamic Unblacklisting (src/utils/blacklist.py):
  • Added remove_from_blacklist(domain: str) to cleanly remove domain entries from data/blacklist.json in-memory and on disk.
  • Hardened unit tests (scratch/test_performance_quality_features.py) with isolated test domains (test-cooldown-domain.org) and guaranteed try...finally teardown cleanup.
• Text-Embedded Video & Media Extraction (src/scraper/video_scraper.py):
  • Expanded _extract_videos_from_scripts() to scan raw text content across structural HTML tags (<p>, <div>, <span>, <article>, <section>).
  • Added regex pattern matchers for direct YouTube (youtube.com/watch?v=, youtu.be/), Vimeo (vimeo.com/), HLS (.m3u8), DASH (.mpd), and MP4/WebM video links embedded inside plain text nodes.
• Strict BeautifulSoup Tag & Type Safety Guards (src/core/semantic_selectors.py, src/scraper/video_scraper.py, src/scraper/google_images.py, src/cli/cleanup.py):
  • Added _get_attr_str() helper and explicit isinstance(el, Tag) type guards across extract_semantic_fallback_images(), extract_semantic_fallback_videos(), extract_videos_from_html(), and layout container detectors.
  • Refined format_size(size: float | int) signature in src/cli/cleanup.py to fix in-place float division assignment type mismatches.
• SSE Real-Time WebUI Log Streaming (frontend/app.py, src/utils/logger.py):
  • Integrated LogBroadcaster Pub-Sub SSE endpoint (/api/logs/stream) for streaming server logs directly to HTMX frontend consoles over async generators, while retaining full backward compatibility for /api/logs?offset=N legacy polling.
• Domain Rules & Search Provider Resilience (src/core/managers.py, src/scraper/google_images.py):
  • Consolidated thread-safe DomainRulesManager with automatic mtime disk reload for data/domain_config.json and data/url_normalisation_rules.json.
  • Upgraded DuckDuckGo and Bing search provider redirect decoders (/l/?uddg=, /bing/url?link=) with automatic fallback failover.
• Complete Test Suite Verification (scratch/):
  • Expanded and verified all 7 dedicated scratch test suites, achieving 100% clean test execution across 173 tests.

Full Changelog: v0.19.0...v0.20.0

scrAPE v0.19.0

scrAPE v0.19.0 Release Notes

Release Date: July 23, 2026
Package Version: 0.19.0
Git Tag: v0.19.0
Distribution Build: dist/scrape_dashboard-0.19.0-py3-none-any.whl & dist/scrape_dashboard-0.19.0.tar.gz

---

🌟 High-Impact Upgrades in v0.19.0

1. FlareSolverr 127.0.0.1:8191 & Background Docker Auto-Start

• Configured default FLARESOLVERR_URL to "http://127.0.0.1:8191/v1" with fallback to localhost:8191, resolving Windows IPv6 resolution latency.
• Implemented background Docker container launch (docker start flaresolverr) when port 8191 is unreachable on startup.
• Enriched downstream CDN streaming media requests with harvested domain session cookies (session_id).

2. Dual Token-Bucket Speed & Rate Limiting

• Page Rate Limiting (--rate-limit / RPS): Regulates outgoing page HTTP request rate.
• Download Speed Limiting (--dl-speed-limit / KBPS): Throttles network bandwidth across active media chunk streams.

3. Extended High-Resolution URL Heuristics

• Added path transformation rules for Erome (/t/ / /th/ $\rightarrow$ /v/) and WordPress (-scaled.jpg / -scaled.png stripping).

4. Search Query Pre-Filtering (is_search_page_url)

• Intercepts and skips un-crawlable search query endpoints (/search?q=, ?text=, search_query=) on Google, Vimeo, Flickr, and YouTube before request allocation.

---

📦 Distribution Packages Built

dist/
├── scrape_dashboard-0.19.0-py3-none-any.whl   (9.9 KB Wheel Package)
└── scrape_dashboard-0.19.0.tar.gz             (48.2 KB Source Distribution)

---

🧪 Verification Benchmarks

• pytest Test Suite: 131 passed in 2m 48s (0 failures, 0 errors).
• Multi-Seed Live Scrape: 1,643 images and 202 videos (~9.31 GB) collected across test seed runs.