Skip to content

v0.2.6

Choose a tag to compare

@github-actions github-actions released this 29 Jul 08:49
· 53 commits to main since this release

Added

  • TorPoolProxyUrl: support for tor-pool,
    which fronts many Tor instances with one sticky SOCKS port. The SOCKS5
    username is a session key, so a scrape keeps the same exit IP until it
    rotates; rotation goes through the pool's API and skips Tor's ~10s NEWNYM
    cooldown by reassigning to an already-built instance.

    Set token to a proxy-scoped token from the pool — it is required by
    tor-pool 0.2 and later, and is sent both as the SOCKS5 password and as a
    bearer token on the pool's API. Without it the pool answers 401, and because
    those calls are best-effort the failure would otherwise pass as a warning while
    the pool quietly stopped hearing about soft blocks; that specific case is
    logged at error instead.

  • ProxyManager.report_failure(): reports 403s, challenges, rate limits and
    transport errors to the pool. This is the only signal that catches a soft block
    — a proxy relaying bytes cannot see a 403 or a captcha inside an HTTPS tunnel —
    and it is what lets the pool quarantine a burnt exit. Sent automatically by the
    engine; call it directly when your own code detects a block.

    The pool weighs a report by what it says went wrong, so each one carries a
    kind as well as free text. A 429 that no challenge handler claimed is sent as
    rate_limited rather than as a generic failure: a throttle says the exit works
    and is being asked for too much, and reported as a block it would retire a
    working exit while the next one is throttled just the same.

  • scraper.engine.proxy_manager.FAILURE_KINDS, the mapping from a failure reason
    to the kind sent alongside it. Reasons the engine raises itself are all
    covered, as is the pool's own
    vocabulary for callers passing it straight through; anything else is still
    reported and the pool counts it as unclassified. Sent explicitly rather than
    left to the pool to read out of the free text — its aliases exist for callers
    written before kinds did, so leaning on them means a vocabulary drift on either
    side quietly downgrades every report to unremarkable.

  • examples/13_tor_pool.py.

Fixed

  • Rotating a proxy now drops pooled connections. A live keep-alive stays bound
    to its original exit, so without this the exit IP appeared not to change until
    the socket happened to be evicted.
  • A pool that no longer knows a session is no longer a warning. Acting on a
    report, the pool takes the instance out of rotation and unpins its sessions, so
    the next report about that session answers 404 — routine, and the next
    request re-pins to a healthy instance, but it logged a warning per report for
    exactly the exit that was failing most.