Skip to content

feat(camoufox): rotate egress proxies across /v1/fetch sessions - #9

Merged
KeyCode17 merged 1 commit into
mainfrom
feat/proxy-rotation
May 19, 2026
Merged

feat(camoufox): rotate egress proxies across /v1/fetch sessions#9
KeyCode17 merged 1 commit into
mainfrom
feat/proxy-rotation

Conversation

@KeyCode17

Copy link
Copy Markdown
Owner

Summary

Adds operator-side proxy rotation for the persistent Camoufox session pool. With PX_PROXIES set, each newly-spawned session binds to the next proxy in round-robin order. Empty/unset → direct connection (no behavioural change).

This is the structural answer to "sustained scraping above what a single IP can carry": pedidosya's PerimeterX rate-limits the egress at roughly 10-30 req/min before serving block pages, so the only way to feed a downstream like the pedidosya scraper its required ≥40 req/min is to fan out across multiple egress IPs.

What's in

  • infrastructure/proxy_pool.rsProxyPool::from_env("PX_PROXIES") parses a CSV (whitespace-trimmed, blanks dropped); .next() returns Option<String> round-robin via an AtomicUsize cursor. None = "no proxy, go direct". 3 unit tests cover empty pool / rotation order / trimming.
  • PersistentSession::spawn now accepts proxy: Option<&str> and threads it into build_capabilities, which translates to the standard webdriver proxy capability (handled by geckodriver as Firefox network prefs).
  • SessionPool takes Arc<ProxyPool> in its constructor; the lazy-spawn branch of acquire pulls proxies.next() before constructing the session. Sessions stick to their proxy for their entire 5-min TTL so cookies + JA3 + egress stay coherent.
  • CamoufoxPool::new reads PX_PROXIES at startup and logs the rotation size, so the operator can confirm direct vs proxied mode from a single startup line.

What's not in (intentional)

  • Per-proxy health tracking. A dead proxy in the rotation surfaces as the matching session's first fetch failing; the session ages out via the TTL. A follow-up can add success/fail counters and skip-broken-proxy logic — that's an ADR-level conversation about the error model and eviction policy.
  • Solve-path rotation. /v1/solve still consumes the caller-supplied HarvestRequest.proxy. Solves are infrequent, operator-driven, and already have a hook; auto-rotation there is its own decision.
  • No per-session sticky-proxy retry. Combined with PR perf(camoufox): tune fetcher for 30+ req/min target #8's backoff retry, a session that's blocked won't be re-acquired with a different proxy until it ages out. A targeted retry-with-new-proxy is a follow-up.

Why

Live testing against pedidosya in the previous session triggered the WAF's IP-level rate limit ("tráfico inusual" Spanish block page) after about 50 ad-hoc calls. No code change to the session pool, fetcher, or scraper helps a flagged IP; the only real defenses are (a) wait, (b) rotate IPs. This PR is (b).

Deployment

PX_PROXIES="http://user:pass@proxy1.example:8080,socks5://user:pass@proxy2.example:1080" \
PX_FETCH_MAX_PER_DOMAIN=2 \
./target/release/px-server

With PX_FETCH_MAX_PER_DOMAIN=N and len(proxies)=M, the pool spawns up to N browsers per domain, each on the next proxy in the rotation. Effective concurrent egress paths = N × min(M, N) before round-robin reuse begins.

[Tor smoke-test recipe in deployment.md.]

Test plan

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets --all-features against the lefthook rule set
  • cargo test -p pxsolver-camoufox --lib — 5 tests passing (2 pre-existing config tests + 3 new proxy_pool::tests)
  • Lefthook pre-commit hooks pass locally (fmt, clippy, loc≤200, forbidden-patterns)
  • Each file under the 200-LOC axum-best-practice rule (session_pool.rs 99, proxy_pool.rs 108)
  • Live: end-to-end smoke against pedidosya with at least 2 proxies — operator-side, the test box currently has none and the home IP is still PX-rate-limited

🤖 Generated with Claude Code

Adds operator-side proxy rotation so sustained fetch traffic isn't
clamped by a single egress IP's PerimeterX rate limit. With
`PX_PROXIES="http://a,http://b,socks5://c"` set, the session pool
binds each newly-spawned `PersistentSession` to the next proxy in
round-robin order. Empty / unset → direct connection, unchanged.

What lands
- New `infrastructure/proxy_pool.rs` — `ProxyPool::from_env` parses
  the CSV (whitespace-trim, drop blanks), `.next()` returns
  `Option<String>` round-robin via `AtomicUsize` cursor. Three unit
  tests cover empty-pool, rotation order, and trimming semantics.
- `PersistentSession::spawn` now takes `proxy: Option<&str>` and
  threads it into `build_capabilities`, which already accepts a
  proxy URL and converts it to the geckodriver `proxy` capability.
- `SessionPool::new` takes `Arc<ProxyPool>`; `acquire`'s lazy-spawn
  branch pulls the next proxy before constructing the session.
  Sessions inherit their proxy for their entire 5-min TTL — same
  cookies, same egress, same fingerprint.
- `CamoufoxPool::new` reads `PX_PROXIES` and logs the rotation size
  at startup so the operator can see at a glance whether direct or
  proxied mode is active.

What's not in this PR (intentional)
- No per-proxy health tracking. A dead proxy in the rotation
  surfaces as the matching session's first fetch failing; the
  session then ages out via the TTL. A follow-up can add
  per-proxy success/fail counters and skip-broken-proxy logic
  (ADR-territory: error model, eviction).
- Solve path (`/v1/solve`) still uses caller-supplied
  `HarvestRequest.proxy` only; auto-rotation there is a separate
  consideration since solves are infrequent and operator-driven.

Deployment doc updated with `PX_PROXIES` syntax and a Tor smoke-test
recipe (with the caveat that many sites block Tor exit IPs).

Build + clippy strict + 3 new unit tests all green on the workspace.
@KeyCode17
KeyCode17 merged commit 66842e3 into main May 19, 2026
6 checks passed
@KeyCode17
KeyCode17 deleted the feat/proxy-rotation branch May 19, 2026 07:09
@KeyCode17
KeyCode17 restored the feat/proxy-rotation branch May 19, 2026 07:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant