v0.2.6
Added
-
TorPoolProxyUrl: support for tor-pool,
which fronts many Tor instances with one sticky SOCKS port. The SOCKS5
username is a session key, so a scrape keeps the same exit IP until it
rotates; rotation goes through the pool's API and skips Tor's ~10s NEWNYM
cooldown by reassigning to an already-built instance.Set
tokento aproxy-scoped token from the pool — it is required by
tor-pool 0.2 and later, and is sent both as the SOCKS5 password and as a
bearer token on the pool's API. Without it the pool answers401, and because
those calls are best-effort the failure would otherwise pass as a warning while
the pool quietly stopped hearing about soft blocks; that specific case is
logged aterrorinstead. -
ProxyManager.report_failure(): reports 403s, challenges, rate limits and
transport errors to the pool. This is the only signal that catches a soft block
— a proxy relaying bytes cannot see a 403 or a captcha inside an HTTPS tunnel —
and it is what lets the pool quarantine a burnt exit. Sent automatically by the
engine; call it directly when your own code detects a block.The pool weighs a report by what it says went wrong, so each one carries a
kind as well as free text. A 429 that no challenge handler claimed is sent as
rate_limitedrather than as a generic failure: a throttle says the exit works
and is being asked for too much, and reported as a block it would retire a
working exit while the next one is throttled just the same. -
scraper.engine.proxy_manager.FAILURE_KINDS, the mapping from a failure reason
to the kind sent alongside it. Reasons the engine raises itself are all
covered, as is the pool's own
vocabulary for callers passing it straight through; anything else is still
reported and the pool counts it as unclassified. Sent explicitly rather than
left to the pool to read out of the free text — its aliases exist for callers
written before kinds did, so leaning on them means a vocabulary drift on either
side quietly downgrades every report to unremarkable. -
examples/13_tor_pool.py.
Fixed
- Rotating a proxy now drops pooled connections. A live keep-alive stays bound
to its original exit, so without this the exit IP appeared not to change until
the socket happened to be evicted. - A pool that no longer knows a session is no longer a warning. Acting on a
report, the pool takes the instance out of rotation and unpins its sessions, so
the next report about that session answers404— routine, and the next
request re-pins to a healthy instance, but it logged a warning per report for
exactly the exit that was failing most.