feat(camoufox): rotate egress proxies across /v1/fetch sessions - #9
Merged
Conversation
Adds operator-side proxy rotation so sustained fetch traffic isn't clamped by a single egress IP's PerimeterX rate limit. With `PX_PROXIES="http://a,http://b,socks5://c"` set, the session pool binds each newly-spawned `PersistentSession` to the next proxy in round-robin order. Empty / unset → direct connection, unchanged. What lands - New `infrastructure/proxy_pool.rs` — `ProxyPool::from_env` parses the CSV (whitespace-trim, drop blanks), `.next()` returns `Option<String>` round-robin via `AtomicUsize` cursor. Three unit tests cover empty-pool, rotation order, and trimming semantics. - `PersistentSession::spawn` now takes `proxy: Option<&str>` and threads it into `build_capabilities`, which already accepts a proxy URL and converts it to the geckodriver `proxy` capability. - `SessionPool::new` takes `Arc<ProxyPool>`; `acquire`'s lazy-spawn branch pulls the next proxy before constructing the session. Sessions inherit their proxy for their entire 5-min TTL — same cookies, same egress, same fingerprint. - `CamoufoxPool::new` reads `PX_PROXIES` and logs the rotation size at startup so the operator can see at a glance whether direct or proxied mode is active. What's not in this PR (intentional) - No per-proxy health tracking. A dead proxy in the rotation surfaces as the matching session's first fetch failing; the session then ages out via the TTL. A follow-up can add per-proxy success/fail counters and skip-broken-proxy logic (ADR-territory: error model, eviction). - Solve path (`/v1/solve`) still uses caller-supplied `HarvestRequest.proxy` only; auto-rotation there is a separate consideration since solves are infrequent and operator-driven. Deployment doc updated with `PX_PROXIES` syntax and a Tor smoke-test recipe (with the caveat that many sites block Tor exit IPs). Build + clippy strict + 3 new unit tests all green on the workspace.
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds operator-side proxy rotation for the persistent Camoufox session pool. With
PX_PROXIESset, each newly-spawned session binds to the next proxy in round-robin order. Empty/unset → direct connection (no behavioural change).This is the structural answer to "sustained scraping above what a single IP can carry": pedidosya's PerimeterX rate-limits the egress at roughly 10-30 req/min before serving block pages, so the only way to feed a downstream like the pedidosya scraper its required ≥40 req/min is to fan out across multiple egress IPs.
What's in
infrastructure/proxy_pool.rs—ProxyPool::from_env("PX_PROXIES")parses a CSV (whitespace-trimmed, blanks dropped);.next()returnsOption<String>round-robin via anAtomicUsizecursor.None= "no proxy, go direct". 3 unit tests cover empty pool / rotation order / trimming.PersistentSession::spawnnow acceptsproxy: Option<&str>and threads it intobuild_capabilities, which translates to the standard webdriverproxycapability (handled by geckodriver as Firefox network prefs).SessionPooltakesArc<ProxyPool>in its constructor; the lazy-spawn branch ofacquirepullsproxies.next()before constructing the session. Sessions stick to their proxy for their entire 5-min TTL so cookies + JA3 + egress stay coherent.CamoufoxPool::newreadsPX_PROXIESat startup and logs the rotation size, so the operator can confirm direct vs proxied mode from a single startup line.What's not in (intentional)
/v1/solvestill consumes the caller-suppliedHarvestRequest.proxy. Solves are infrequent, operator-driven, and already have a hook; auto-rotation there is its own decision.Why
Live testing against pedidosya in the previous session triggered the WAF's IP-level rate limit ("tráfico inusual" Spanish block page) after about 50 ad-hoc calls. No code change to the session pool, fetcher, or scraper helps a flagged IP; the only real defenses are (a) wait, (b) rotate IPs. This PR is (b).
Deployment
PX_PROXIES="http://user:pass@proxy1.example:8080,socks5://user:pass@proxy2.example:1080" \ PX_FETCH_MAX_PER_DOMAIN=2 \ ./target/release/px-serverWith
PX_FETCH_MAX_PER_DOMAIN=Nandlen(proxies)=M, the pool spawns up toNbrowsers per domain, each on the next proxy in the rotation. Effective concurrent egress paths =N × min(M, N)before round-robin reuse begins.[Tor smoke-test recipe in deployment.md.]
Test plan
cargo fmt --all -- --checkcargo clippy --workspace --all-targets --all-featuresagainst the lefthook rule setcargo test -p pxsolver-camoufox --lib— 5 tests passing (2 pre-existing config tests + 3 newproxy_pool::tests)session_pool.rs99,proxy_pool.rs108)🤖 Generated with Claude Code