Skip to content

scrAPE v0.22.0

Choose a tag to compare

View all tags
@rhaffle87 rhaffle87 released this 29 Jul 15:35
· 129 commits to main since this release
v0.22.0
639883b

Release Notes — scrAPE v0.22.0

Release Date: July 29, 2026
Version: v0.22.0
Status: Production Release

Highlights of Release v0.22.0

1. 8-Tier WAF Stealth Pipeline & CapSolver Integration

  • CapSolver Captcha Token Injection: Automatic detection and solving for Cloudflare Turnstile, reCAPTCHA v2/v3, and hCaptcha sitekeys with direct injection into SessionPool and disk cookie caches (data/sessions/).
  • Host Preferred Engine Learning: Dynamic reordering of stealth strategies per host based on historical success rates (_preferred_engine_by_host), avoiding redundant fallback steps.
  • Node.js Crawlee Stealth Bridge: Health check integration (CrawleeStrategy.is_available()) auto-spawning the Node.js Express server on port 10002.
  • WebUI Stealth Telemetry Card: Real-time #stealth-telemetry-card displaying solve counts, active circuit breaker cooldowns, and engine bindings per domain (/api/telemetry/stealth).

2. AI Dataset Auto-Tagging & Multi-Stage LoRA Quality Export

  • Hybrid Vision Auto-Tagger (src/utils/dataset_tagger.py): Opt-in WD14 Booru ViT vision model classification (landscape, portrait, square, highres, lowres) combined with metadata and filename tokenization. Enforces trigger tag placement at index 0 in .txt sidecars.
  • Multi-Stage LoRA Quality Gate (src/utils/dataset_exporter.py):
    • Stage 1: Min resolution filter ($\ge 512\text{px}$).
    • Stage 2: Perceptual dHash near-duplicate check ($\text{Hamming distance} \le 4$).
    • Stage 3: Aesthetic score threshold filter ($\ge 5.5 / 10.0$).
  • WebUI LoRA Export Modal: Aesthetic Score Threshold selector (export-min-score) in the WebUI export modal and automatic post-scrape sidecar tag generation.

3. Watchdog Multi-Channel Webhook Notifier Architecture

  • Pluggable BaseNotifier Interface (src/utils/notification_manager.py): Standard abstract contract with dynamic provider registration (register_provider).
  • Supported Providers:
    • TelegramNotifier (wrapping TelegramBotNotifier).
    • DiscordNotifier (rich color-coded JSON embeds).
    • SlackNotifier (Block Kit formatted JSON).
    • CustomWebhookNotifier (generic HTTP POST for Apprise, N8N, Zapier, Matrix, Pushover).
  • Parallel Dispatcher: NotificationPipeline thread pool dispatching alerts concurrently.
  • WebUI Test Endpoint: POST /api/notifications/test returning per-channel delivery breakdowns.

4. WebUI Live Canvas Physics Visualizer & Code Quality

  • Live Crawl Tree Visualizer: HTML5 Canvas force-directed spring graph with search input, depth selectors, and interactive Node Inspector drawer.
  • Sonar/IDE Warning Remediation: Cleared all 7 reported IDE warnings in index.html (regex backtracking optimizations, replaceAll() conversion, and rendering loop optimization).

5. Automated Test Suite Metrics

  • 310/310 Unit and Integration Tests Passing (100% Success Rate).

Migration & API Summary

  • New FastAPI Endpoints:
    • GET /api/telemetry/stealth — Stealth pipeline stats, solve counts, engine bindings.
    • POST /api/notifications/test — Multi-channel webhook ping test.
  • Modified Query Parameters:
    • GET /api/export/dataset/download/{subject}/{run_id}?repeats=10&min_resolution=512&min_aesthetic_score=5.5
  • New Environment Variables:
    • DISCORD_WEBHOOK_URL
    • SLACK_WEBHOOK_URL
    • CUSTOM_WEBHOOK_URL

Full Changelog: v0.20.0...v0.22.0

Assets 4
Loading