Added
-
Scrapy stealth stats
Request, response, success, failure, status, ban, recycle, proxy-use, and DNS-use
counters appear incrawler.stats, globally and by driver where useful. Current
driver, profile, redacted proxy, ban streak, and active DNS host count are also
exposed. Collection reuses existing response / ban checks: no extra body parsing,
network requests, or per-domain high-cardinality stats. -
Middleware closes engines on
spider_closed
Chrome, the DNS CONNECT relay, and the browser asyncio loop are torn down when
the spider finishes instead of lingering until process exit. -
Full spider example
examples/full_spider.pydemonstrates settings,
per-request drivers, snapshots, ban detection, and stealth stats. README links
to it instead of embedding a long copy.
Changed
- Faster browser shutdown
BrowserEngine.close()/ recycle stop Chrome before draining asyncio tasks, use
shorter teardown timeouts, and delete nodriver temp data (profiles, caches,
cookies, GPU/shader data, and logs) on a background thread via
_cleanup_browser_temp_data().