Skip to content

v0.6.11

Latest

Choose a tag to compare

@fawadss1 fawadss1 released this 04 Aug 13:40
· 18 commits to master since this release
04bd2af

Added

  • Scrapy stealth stats
    Request, response, success, failure, status, ban, recycle, proxy-use, and DNS-use
    counters appear in crawler.stats, globally and by driver where useful. Current
    driver, profile, redacted proxy, ban streak, and active DNS host count are also
    exposed. Collection reuses existing response / ban checks: no extra body parsing,
    network requests, or per-domain high-cardinality stats.

  • Middleware closes engines on spider_closed
    Chrome, the DNS CONNECT relay, and the browser asyncio loop are torn down when
    the spider finishes instead of lingering until process exit.

  • Full spider example
    examples/full_spider.py demonstrates settings,
    per-request drivers, snapshots, ban detection, and stealth stats. README links
    to it instead of embedding a long copy.

Changed

  • Faster browser shutdown
    BrowserEngine.close() / recycle stop Chrome before draining asyncio tasks, use
    shorter teardown timeouts, and delete nodriver temp data (profiles, caches,
    cookies, GPU/shader data, and logs) on a background thread via
    _cleanup_browser_temp_data().