Skip to content

v1.4.0

Choose a tag to compare

@github-actions github-actions released this 31 Jul 21:38
· 18 commits to main since this release

Added

  • CdpSolver — drive Chrome without nodriver, on every Python this package supports. nodriver cannot be imported below 3.10 or from 3.14, so on both ends of the range there was no solver at all and a challenged origin failed honestly instead of being solved. This speaks CDP over a WebSocket directly: the seven calls a solve actually needs, against forty thousand lines of generated bindings for the rest of the protocol. Needs the new cdp extra, which is deliberately unmarked — a marker would reintroduce the gap it exists to close.

    It never enables a CDP domain, which is the reason to own the wire rather than wrap a driver. Eagerly-enabled domains are a known tell and a general-purpose driver has to enable them, because it cannot know what its caller will ask for next; Runtime.evaluate and Page.navigate are commands, not subscriptions.

    Split into a transport, a backend and a solver because the transport is not Chrome-specific: WebDriver BiDi is the same WebSocket JSON-RPC shape, so a Firefox backend reuses it and only has to speak a second vocabulary.

  • BrowserSolver.impersonation — a solver says what its clearance binds to. ScraperConfig.profile() forced chrome on every request to every origin the moment any solver was configured, which is right only for solvers that drive Chrome. A Firefox-based one was overridden into contradicting itself: clearance earned under a Firefox User-Agent, replayed over a Chrome handshake. Defaults to chrome, so nothing bundled changes behaviour.

  • interactive_solve_timeout — wait for a person when the window is visible. The solve loop detects success by polling and does not care who cleared the page, so a human needs no protocol of their own, only enough time. A solver setting interactive gets this budget (300s) instead of solve_timeout (90s), and the origin is named in a log line — a browser window appearing with nothing said about it reads as the app misbehaving. Separate from solve_timeout rather than a larger value of it, because raising the one number would make every unattended failure four times slower. Both bundled solvers set it from whether they run headed.

Fixed

  • A proxy URL is now translated into what Chrome accepts, or refused. --proxy-server rejects the whole flag — any scheme — when the URL carries credentials, and rejects socks5h as an unknown scheme. Both failed as ERR_NO_SUPPORTED_PROXIES on every navigation, so a solve through a tor-pool or an authenticated proxy could never have worked, and reported as "the solver finished without a clearance cookie". socks5h is now translated to socks5, which resolves at the proxy anyway. Credentials raise TierUnavailable naming the reason: dropping them and launching would be worse, since for a pool the username is the session key and the browser would earn its clearance on a different exit than the requests that replay it.

  • --disable-blink-features=AutomationControlled, on every solver. Blink otherwise sets navigator.webdriver to true — one boolean saying "automated" that every detector reads. Found by measuring rather than by reasoning: the new solver cleared none of six challenged hosts and spent the full 60s on each, and cleared all six in under ten seconds with the flag. A driver library may set this for you; relying on that is how a solve fails slowly for a reason nothing reports.

  • A headless browser could not clear a challenge, and the cause was one substring. Headless Chrome writes HeadlessChrome into its User-Agent, and that was the whole of the penalty. NoDriverSolver now reads its own User-Agent once per process and launches headless under the corrected one, as a launch flag — not Network.setUserAgentOverride, which looks equivalent but suppresses the Sec-CH-UA header, trading a browser that admits to being headless for one that claims to be Chrome and sends no brands.

    Measured over 46 challenged sites, 166 solves: headless cleared nothing before the fix and clears 27 of the 27 hosts a headed browser clears after it, at the same speed, plus one that headed does not.

  • The advice to run a virtual display on a server was wrong, and so was its reasoning. Headless was said to give itself away through a software WebGL renderer; forcing that renderer on a machine with a GPU changed nothing and every host still cleared. In a container nothing cleared at all — not headless, not headless with the User-Agent corrected, not headed under Xvfb — because Debian's chromium omits the Google Chrome brand from Sec-CH-UA, which every request carries. That is the browser build showing through, and no display setting hides it. Install the browser a real visitor runs.