v1.4.0
Added
-
CdpSolver— drive Chrome without nodriver, on every Python this package supports. nodriver cannot be imported below 3.10 or from 3.14, so on both ends of the range there was no solver at all and a challenged origin failed honestly instead of being solved. This speaks CDP over a WebSocket directly: the seven calls a solve actually needs, against forty thousand lines of generated bindings for the rest of the protocol. Needs the newcdpextra, which is deliberately unmarked — a marker would reintroduce the gap it exists to close.It never enables a CDP domain, which is the reason to own the wire rather than wrap a driver. Eagerly-enabled domains are a known tell and a general-purpose driver has to enable them, because it cannot know what its caller will ask for next;
Runtime.evaluateandPage.navigateare commands, not subscriptions.Split into a transport, a backend and a solver because the transport is not Chrome-specific: WebDriver BiDi is the same WebSocket JSON-RPC shape, so a Firefox backend reuses it and only has to speak a second vocabulary.
-
BrowserSolver.impersonation— a solver says what its clearance binds to.ScraperConfig.profile()forced chrome on every request to every origin the moment any solver was configured, which is right only for solvers that drive Chrome. A Firefox-based one was overridden into contradicting itself: clearance earned under a Firefox User-Agent, replayed over a Chrome handshake. Defaults tochrome, so nothing bundled changes behaviour. -
interactive_solve_timeout— wait for a person when the window is visible. The solve loop detects success by polling and does not care who cleared the page, so a human needs no protocol of their own, only enough time. A solver settinginteractivegets this budget (300s) instead ofsolve_timeout(90s), and the origin is named in a log line — a browser window appearing with nothing said about it reads as the app misbehaving. Separate fromsolve_timeoutrather than a larger value of it, because raising the one number would make every unattended failure four times slower. Both bundled solvers set it from whether they run headed.
Fixed
-
A proxy URL is now translated into what Chrome accepts, or refused.
--proxy-serverrejects the whole flag — any scheme — when the URL carries credentials, and rejectssocks5has an unknown scheme. Both failed asERR_NO_SUPPORTED_PROXIESon every navigation, so a solve through a tor-pool or an authenticated proxy could never have worked, and reported as "the solver finished without a clearance cookie".socks5his now translated tosocks5, which resolves at the proxy anyway. Credentials raiseTierUnavailablenaming the reason: dropping them and launching would be worse, since for a pool the username is the session key and the browser would earn its clearance on a different exit than the requests that replay it. -
--disable-blink-features=AutomationControlled, on every solver. Blink otherwise setsnavigator.webdriverto true — one boolean saying "automated" that every detector reads. Found by measuring rather than by reasoning: the new solver cleared none of six challenged hosts and spent the full 60s on each, and cleared all six in under ten seconds with the flag. A driver library may set this for you; relying on that is how a solve fails slowly for a reason nothing reports. -
A headless browser could not clear a challenge, and the cause was one substring. Headless Chrome writes
HeadlessChromeinto its User-Agent, and that was the whole of the penalty.NoDriverSolvernow reads its own User-Agent once per process and launches headless under the corrected one, as a launch flag — notNetwork.setUserAgentOverride, which looks equivalent but suppresses theSec-CH-UAheader, trading a browser that admits to being headless for one that claims to be Chrome and sends no brands.Measured over 46 challenged sites, 166 solves: headless cleared nothing before the fix and clears 27 of the 27 hosts a headed browser clears after it, at the same speed, plus one that headed does not.
-
The advice to run a virtual display on a server was wrong, and so was its reasoning. Headless was said to give itself away through a software WebGL renderer; forcing that renderer on a machine with a GPU changed nothing and every host still cleared. In a container nothing cleared at all — not headless, not headless with the User-Agent corrected, not headed under Xvfb — because Debian's
chromiumomits theGoogle Chromebrand fromSec-CH-UA, which every request carries. That is the browser build showing through, and no display setting hides it. Install the browser a real visitor runs.