v0.6.11

Added

• Scrapy stealth stats
  Request, response, success, failure, status, ban, recycle, proxy-use, and DNS-use
  counters appear in crawler.stats, globally and by driver where useful. Current
  driver, profile, redacted proxy, ban streak, and active DNS host count are also
  exposed. Collection reuses existing response / ban checks: no extra body parsing,
  network requests, or per-domain high-cardinality stats.

• Middleware closes engines on spider_closed
  Chrome, the DNS CONNECT relay, and the browser asyncio loop are torn down when
  the spider finishes instead of lingering until process exit.

• Full spider example
  examples/full_spider.py demonstrates settings,
  per-request drivers, snapshots, ban detection, and stealth stats. README links
  to it instead of embedding a long copy.

Changed

• Faster browser shutdown
  BrowserEngine.close() / recycle stop Chrome before draining asyncio tasks, use
  shorter teardown timeouts, and delete nodriver temp data (profiles, caches,
  cookies, GPU/shader data, and logs) on a background thread via
  _cleanup_browser_temp_data().

v0.6.11a1

Added

• Basic / turbo — session recycle after consecutive bans
  STEALTH_RECYCLE_AFTER_BANS (and STEALTH_RECYCLE_COOLDOWN_S) apply to basic and
  turbo: after N consecutive banned responses, cached HTTP clients/sessions are cleared and
  the engine default fingerprint profile and proxy (from STEALTH_PROXIES) are rotated.
  Same ban heuristics as the browser engine (is_browser_session_ban). Explicit meta
  profile / proxy still win. BanStreakTracker lives in utils/session.py.

• STEALTH_PROXIES on config
  Proxy pool is loaded from Scrapy settings into config.STEALTH_PROXIES and seeded as the
  engine default; rotated on ban-streak recycle.

Changed

• Renamed recycle settings
  BROWSER_RESTART_AFTER_BANS → STEALTH_RECYCLE_AFTER_BANS,
  BROWSER_RESTART_COOLDOWN_S → STEALTH_RECYCLE_COOLDOWN_S (apply to all drivers).

• Removed rotate_profile / rotate_proxy meta flags
  Profile and proxy now change automatically on ban-streak session recycle only.
  Set STEALTH_PROXIES in settings; use explicit meta["stealth"]["profile"] /
  ["proxy"] when you need a fixed identity.

Fixed

• Meta-only proxy cleared to None on recycle
  When STEALTH_PROXIES is empty, recycle keeps the request's meta["stealth"]["proxy"]
  instead of wiping the engine default.

• Concurrent recycle storm / stuck-together console lines
  BanStreakTracker claims only once per ban wave so parallel Scrapy threads do not all
  recycle and print at once. Console output is lock-protected with flush=True.

v0.6.10

[0.6.10] - 2026-07-23

Added

• Custom DNS overrides (STEALTH_DNS_OVERRIDES / meta["stealth"]["dns"])
  Pin hostnames to fixed origin IPs so basic / turbo connect via that address while keeping the hostname for TLS SNI, Host header, and certificate verification. Configure globally via Scrapy settings / config.STEALTH_DNS_OVERRIDES, or per-request with a bare IP ("dns": "203.0.113.10") or a {host: ip} map. The browser driver applies the effective map via a local CONNECT relay that dials the pinned IP (not Chrome --host-resolver-rules). Invalid IPs raise ValueError at startup / resolve time.

• Automatic PyPI update check
  When StealthDownloaderMiddleware is loaded, scrapy-stealth checks PyPI once per process in a background thread. If a newer version is published, an info message is printed with pip install -U scrapy-stealth and a link to that release on PyPI (e.g. https://pypi.org/project/scrapy-stealth/0.7.0/). Network errors are silent and never interrupt crawling.

• Local CI helper (scripts/check.py, CHECK.md)
  Run the same ruff, format, mypy, and pytest checks as GitHub Actions locally before pushing.

• is_browser_session_ban() — stricter ban detection for browser restarts
  New helper in utils/antibot.py. HTTP block codes (403, 429, 503) always count; keyword and JS-challenge heuristics apply only to short pages (< 2500 bytes). Large HTTP 200 documents that embed anti-bot scripts (e.g. DataDome on RS Online) are no longer treated as bans.

Changed

• **Dependency: Required for the DnsOptions API used by custom DNS overrides (ResolverOptions).

• Basic engine DNS — apply dns_options on Client(...)
  wreq ignores per-request dns_options= on get()/post(); clients are now cached per (http2, dns map) like turbo sessions.

• Browser engine DNS — local CONNECT relay
  Replaced unreliable --host-resolver-rules with a DNS-aware local proxy relay (same mechanism as proxy auth injection). Pinned hosts are dialed by IP while Chrome keeps the original hostname for TLS.

• Browser relay — silence shutdown races / dial IPs without getaddrinfo
  Pinned-IP connects use sock_connect so Windows Proactor no longer hits RuntimeError: cannot schedule new futures after shutdown when Chrome still CONNECT-retries during browser restart. Loop/executor teardown errors in the relay callback are swallowed.

• Browser engine — blank tab / wait / relay consistency
  Disabled enable_begin_frame_control on tab create. Browser always uses the local CONNECT relay (even with no DNS/proxy). Replaced nodriver page.wait() with _wait_for_document. _wait_for_status returns ~0.75s after document complete when Navigation Timing never fills. _smart_wait long-poll only for short challenge/script-only shells. Challenge HTML heuristics no longer match bare akamai / captcha / please wait. / enable javascript / datadome / kasada.

• Browser engine — close tab as soon as _smart_wait passes
  _smart_wait returns immediately when body content is ready (settle is a max wait for growth, not a sleep after ready). HTML is captured, the fetch tab is closed via CDP, then the response is returned. Chrome is stopped when idle so the window is not left on about:blank.

• Browser splash — show docs/static/logo.png
_splash_url() loads the package logo via file:// when the file exists (falls back to about:blank).

• .gitignore — ignore stealth_snapshots/, the default output directory for browser snapshots saved via the @snapshot decorator.

• BROWSER_RESTART_COOLDOWN_S — default reduced from 60 to 15 seconds and reworked. Cooldown now spaces ban-triggered restarts when every concurrent request keeps returning 403, without blocking the first restart or all subsequent restarts for a full minute. Configurable via config.BROWSER_RESTART_COOLDOWN_S.

• BanStreakTracker — restart is signalled once per ban wave (_restart_due flag); bans during an active restart (_restarting=True) are ignored; streak resets when restart begins (acknowledge_restart() before _reset_browser()).

Fixed

• Browser engine — false “5 consecutive bans” restarts on HTTP 200
  _maybe_restart now uses is_browser_session_ban() instead of generic is_blocked() + is_js_challenge(), which matched anti-bot script fragments in otherwise valid product pages.

• Browser engine — only one restart after 5 bans, then never again
  Removed the broken 60s cooldown that treated _last_restart = 0 as “just restarted” and blocked the first restart; later removed the all-or-nothing cooldown that prevented any second restart within 60s.

• Browser engine — restart storm every 1–2 seconds under concurrent 403s
  Fixed duplicate restart signals when streak exceeded the threshold, bans piling up during Chrome reset, and concurrent threads all triggering _reset_browser() in the same ban wave.

• Browser engine — CancelledError and empty “request failed” logs during restart
  Fetches cancelled mid-restart now wait for Chrome to come back (_wait_for_browser_ready()) and retry up to 5 times. Fixed a deadlock from calling _wait_for_browser_ready() while already holding the engine lock.

• Browser engine — intermittent empty HTML on JS-heavy pages (HTTP 200)
  _smart_wait no longer skips the settle delay when the body text is already long.

v0.6.10a1

Added

• Browser restart cooldown (BROWSER_RESTART_COOLDOWN_S)
  Minimum seconds between browser restarts (default 60). Prevents restart storms when many concurrent tabs all receive 403s from the same blocked session.

Fixed

• Browser engine — restart not firing after 5 consecutive bans
  BanStreakTracker.record() no longer resets the streak or starts the cooldown until the restart actually completes (acknowledge_restart()). Previously, a restart signal could be consumed while another restart was already in progress (_restarting=True), leaving Chrome running with a false cooldown active.

• Browser engine — Browser engine timed out after 30s under load
  The browser fetch deadline now includes settle time and headroom for status polling and tab-queue wait (stealth_timeout + settle + 12).

• Windows — ValueError: I/O operation on closed pipe on browser restart
  Suppressed benign asyncio subprocess teardown noise via sys.unraisablehook; added a short post-join pause in _stop_loop on Windows.

• Engine errors — backend library tracebacks hidden
  and other backend failures are now re-raised as StealthTimeoutError / StealthConnectionError via raise_stealth() (from None), so Scrapy logs show only scrapy-stealth exception frames.

v0.6.9

[0.6.9] - 2026-06-29

Added

• Proxy bypass list (BROWSER_PROXY_BYPASS_LIST)
  Route chosen domains around the proxy in the browser engine. The user-supplied list is passed to Chrome's --proxy-bypass-list
  launch flag, so requests to those domains connect to the origin directly instead of through the proxy relay. Supports the full Chrome
  bypass syntax — bare hostnames, wildcards (*.example.com), IP/CIDR ranges, ports, and the <local> token. Configured globally via
  config/settings; only takes effect when a proxy is in use.

Fixed

• Browser engine — pending tasks destroyed on ban-triggered restart
  When one Scrapy thread triggered a browser restart after consecutive bans,
  other concurrent _run_fetch coroutines and their _smart_wait sleep()
  children were left running on the old event loop and destroyed during
  teardown. Restarts now block new fetches behind a restart barrier, drain
  all pending loop tasks before stopping Chrome, and retry transient
  connection errors once on the fresh browser.

• Browser engine — wrong-tab / cannot call get() concurrently
  Replaced browser.get(url, new_tab=True) with direct cdp.target.create_target(url) to
  guarantee a 1:1 mapping between the created CDP target and the Tab object, eliminating the
  wrong-tab race and the duplicate _listener_task that caused the concurrency assertion.

• Browser engine — _do_fetch tasks leaked on timeout
  Tasks continued running after future.result(timeout=...) raised TimeoutError, holding
  _tab_sem slots and producing "Task was destroyed but it is pending!" on teardown.
  The task is now cancelled directly via loop.call_soon_threadsafe(task.cancel) on timeout.

• Browser engine — "Event loop is closed" log noise
  _chain_future callbacks and call_soon_threadsafe handles firing against a closed loop
  after _reset_browser are now suppressed by a teardown filter on the asyncio and
  concurrent.futures loggers.

• Browser engine — AttributeError: 'NoneType' object has no attribute 'get'
  Snapshotting browser = self._browser at _do_fetch entry prevents _reset_browser
  nulling self._browser mid-execution from reaching browser.get().

• Browser engine — Akamai 403 consuming full 30 s timeout
  _wait_for_status now fast-exits on error page titles (Access Denied, Forbidden, etc.)
  returning 403 immediately. _smart_wait exits early when body length stops growing for 3 s.

• Browser engine — logo.png splash causing wrong-tab on startup
  _splash_url() now returns "about:blank" instead of a file:// URI.

• Proxy relay — orphaned handle() tasks on restart / shutdown
  ProxyRelay.await_closed() now cancels and awaits all live handle() tasks before
  closing the server, replacing the bare server.close() that left tasks running.

• Windows Proactor — InvalidStateError crashing the browser loop thread
  _run_loop wraps loop.run_forever() in try/except asyncio.InvalidStateError;
  the loop exception handler suppresses it as well.

• Windows browser-restart log noise (WinError 995)
  Suppressed benign Windows Proactor teardown errors logged when the event loop and proxy relay are torn down during a browser restart.
  The loop exception handler now ignores WinError 995 (ERROR_OPERATION_ABORTED) and WinError 64 (ERROR_NETNAME_DELETED)
  alongside the existing 10054 (WSAECONNRESET); genuine errors are still surfaced. The restart itself was always succeeding — only
  the spurious ERROR tracebacks are gone.

• Temp profiles — uc_* dirs accumulating in %TEMP%
_cleanup_browser_profiles() removes stale nodriver temp dirs on every restart and shutdown.

Changed

• Console — timestamp now styled Fore.YELLOW to match Scrapy's log format.

---

v0.6.9b1

Fixed

• Browser engine — wrong-tab / cannot call get() concurrently
  Replaced browser.get(url, new_tab=True) with direct cdp.target.create_target(url) to
  guarantee a 1:1 mapping between the created CDP target and the Tab object, eliminating the
  wrong-tab race and the duplicate _listener_task that caused the concurrency assertion.

• Browser engine — _do_fetch tasks leaked on timeout
  Tasks continued running after future.result(timeout=...) raised TimeoutError, holding
  _tab_sem slots and producing "Task was destroyed but it is pending!" on teardown.
  The task is now cancelled directly via loop.call_soon_threadsafe(task.cancel) on timeout.

• Browser engine — "Event loop is closed" log noise
  _chain_future callbacks and call_soon_threadsafe handles firing against a closed loop
  after _reset_browser are now suppressed by a _ClosedLoopFilter on the asyncio and
  concurrent.futures loggers.

• Browser engine — AttributeError: 'NoneType' object has no attribute 'get'
  Snapshotting browser = self._browser at _do_fetch entry prevents _reset_browser
  nulling self._browser mid-execution from reaching browser.get().

• Browser engine — Akamai 403 consuming full 30 s timeout
  _wait_for_status now fast-exits on error page titles (Access Denied, Forbidden, etc.)
  returning 403 immediately. _smart_wait exits early when body length stops growing for 3 s.

• Browser engine — logo.png splash causing wrong-tab on startup
  _splash_url() now returns "about:blank" instead of a file:// URI.

• Proxy relay — orphaned handle() tasks on restart / shutdown
  ProxyRelay.await_closed() now cancels and awaits all live handle() tasks before
  closing the server, replacing the bare server.close() that left tasks running.

• Windows Proactor — InvalidStateError crashing the browser loop thread
  _run_loop wraps loop.run_forever() in try/except asyncio.InvalidStateError;
  the loop exception handler suppresses it as well.

• Temp profiles — uc_* dirs accumulating in %TEMP%
_cleanup_browser_profiles() removes stale nodriver temp dirs on every restart and shutdown.

Changed

• Console — timestamp now styled Fore.YELLOW to match Scrapy's log format.

v0.6.9a2

Fixed

• Windows browser-restart log noise (WinError 995)
  Suppressed benign Windows Proactor teardown errors logged when the event loop and proxy relay are torn down during a browser restart.
  The loop exception handler now ignores WinError 995 (ERROR_OPERATION_ABORTED) and WinError 64 (ERROR_NETNAME_DELETED)
  alongside the existing 10054 (WSAECONNRESET); genuine errors are still surfaced. The restart itself was always succeeding — only
  the spurious ERROR tracebacks are gone.

v0.6.9a1

Added

• Proxy bypass list (BROWSER_PROXY_BYPASS_LIST)
  Route chosen domains around the proxy in the browser engine. The user-supplied list is passed to Chrome's --proxy-bypass-list
  launch flag, so requests to those domains connect to the origin directly instead of through the proxy relay. Supports the full Chrome
  bypass syntax — bare hostnames, wildcards (*.example.com), IP/CIDR ranges, ports, and the <local> token. Configured globally via
  config/settings; only takes effect when a proxy is in use.

v0.6.8

Added

• Intelligent content wait (_smart_wait)
  Automatically detects JavaScript challenges, CAPTCHAs, and anti-bot interstitial pages and waits for meaningful page content before returning a response, improving success rates on protected websites.
• Advanced challenge detection
  Added comprehensive detection for Cloudflare, DataDome, Akamai, Kasada, and other common anti-bot challenge pages.
• Randomized browser fingerprinting
  Browser sessions now launch with realistic randomized window sizes and language configurations to reduce fingerprint consistency across sessions.
• Intelligent browser restart (BROWSER_RESTART_AFTER_BANS)
  Browser instances are now restarted only after a configurable number of consecutive bans or challenge responses, replacing the previous fixed-request restart strategy.
• Static asset blocking (BROWSER_STATIC_ASSETS_BLOCK)
  Optional blocking of images, fonts, stylesheets, and other non-essential assets via Chrome DevTools Protocol, reducing bandwidth usage and improving page load performance.
• StealthDependencyError
  New typed exception for optional dependency loading failures, providing platform-specific guidance for resolving missing native libraries and runtime dependencies.

Fixed

• Windows browser restart race condition
  Resolved event-loop teardown and restart timing issues that could produce InvalidStateError exceptions during browser restarts.
• Windows dependency loading failures
  Improved handling of wreq and curl_cffi DLL loading errors with actionable error messages instead of opaque import tracebacks.
• Deferred dependency loading
  Optional browser-profile dependencies are now loaded lazily, preventing unrelated engines from failing when specific native dependencies are unavailable.
• Browser response rendering
  Improved response handling to ensure successful pages are fully rendered before being returned to Scrapy.

Changed

• Browser restart strategy
  Replaced the request-count-based restart mechanism with ban-aware restart logic, reducing unnecessary browser restarts during healthy crawls.
• Test suite refactoring
  Simplified browser-related test cases and reduced mock complexity for improved maintainability.

Performance

• Reduced bandwidth consumption
  Static asset blocking can significantly decrease network usage and page load times when visual assets are not required.
• Improved browser stability
  Smarter restart behavior reduces browser churn while maintaining long-running crawl reliability.

v0.6.8a2

Added

• StealthDependencyError — typed exception for compiled-dependency failures
  New exception class in exceptions.py that inherits from both StealthException and
  ImportError, fitting naturally into both the package exception hierarchy and standard
  except ImportError handlers.
  Raised whenever a compiled optional dependency (wreq, curl_cffi) fails to load —
  typically because a required native DLL or shared library could not be found.

  The exception provides a platform-aware, actionable message at raise time:

  • Windows — instructs the user to install both x64 and x86 Visual C++ Redistributables
    (2015–2022) with direct download links.
  • Linux — suggests the appropriate apt-get / yum packages for missing system
    libraries (libssl, libcurl).

  StealthDependencyError is exported from the top-level package and added to __all__,
  making it catchable in user code alongside the other stealth exceptions.

Fixed

• engines/basic.py — ImportError: DLL load failed while importing wreq on fresh Windows
  The bare from wreq.blocking import Client and from wreq.proxy import Proxy module-level
  imports crashed immediately on machines without the Visual C++ Redistributable installed,
  surfacing as an opaque DLL load failed traceback deep inside Scrapy's middleware loader.
  Both imports are now wrapped in try/except ImportError and delegate to
  StealthDependencyError.check("wreq", exc) for a clear, actionable error message.

• engines/turbo.py — same DLL failure for curl_cffi on fresh Windows
  from curl_cffi import CurlHttpVersion and from curl_cffi.requests import Session suffer
  the same failure path as wreq when the VCRT is absent.
  Both imports are now guarded with StealthDependencyError.check("curl_cffi", exc).

• utils/profiles.py — wreq.emulation crash at import time propagated silently
  from wreq.emulation import Emulation, Profile was a module-level import, meaning the
  entire profiles module — and by extension every engine that imports it — failed to load
  on VCRT-missing machines, producing the same deep DLL load failed traceback.
  The import is now guarded with a _WREQ_AVAILABLE flag; Emulation and Profile fall
  back to None so the module loads cleanly. The private _require_wreq() helper raises
  StealthDependencyError at the point of actual use (inside _resolve_basic), not at
  import time, keeping the turbo and browser drivers unaffected on machines where
  wreq is broken but curl_cffi loads fine.