Repository navigation
v1.5.0
Added
-
BidiSolver— Firefox, over WebDriver BiDi. A clearance binds to the fingerprint that earned it, so a Chrome-only solver forced chrome impersonation on every request to every origin — and firefox is the profile that reaches the most hosts. This declaresimpersonation = "firefox"and the trade disappears. Clearing itself is a tie: 29 of 46 challenged hosts against Chrome's 28, disagreeing on seven, four to Firefox and three to Chrome. The transport moved toscraper.wireand both backends share it unchanged — a second vocabulary, not a second client.One cost, stated rather than buried. A WebDriver session sets
navigator.webdrivertrue by spec and no preference overrides it, so this deletes the property with a preload script. That is patching a surface value rather than not emitting one, and a site reading the prototype descriptor would see through it. Worth nineteen of the 46 hosts; ten challenge without ever reading it. -
find_chromium,find_firefox,pick_chromium,pick_firefox— finding an installed browser is the library's job now. Both solvers looked for one with aPATHscan, which answers for Linux and only Linux: on macOS with Chrome and Firefox installed,CdpSolver()raised "no browser executable was found", and the same on Windows. It went unnoticed because the one consumer always passedexecutable=. This looks inside macOS application bundles, under the Windows program directories, and across distribution and flatpak paths — six Chromium brands, since Brave and Edge answer CDP exactly as Chrome does, plus Firefox with ESR and LibreWolf.
Removed
NoDriverSolverand the nodriver dependency are gone.CdpSolverdoes the same job on every Python this package supports and was not worse head to head — 12 hosts cleared to 11, at the same median — against fifty thousand lines of dependency and a marker excluding two interpreters. This is a breaking change: replacescraper.NoDriverSolverwithscraper.CdpSolver. The[browser]extra stays as an alias for[cdp].
Changed
- Headless is the default. It does not cost a solve, measured over 46 hosts, and most places this runs have no display to put a window on. Pass
headless=Falsefor one, which also buysinteractive_solve_timeout. - A solve timeout now bounds the whole solve, not just the part after the navigation, which could double the budget the caller asked for.
Fixed
- In a container, set
TZbefore blaming anything else. A container runs on UTC, and a browser whose timezone contradicts where its address geolocates reads as automation: same six hosts, same binary, same egress address — 1 of 6 under UTC, 6 of 6 withTZmatching the exit. Debian'schromium, whichbrowser.pyanddocs/tiers.mdpreviously named as the cause, is real but second at 1 of 6 with the clock right. validate_url's docstring described it backwards. A scheme-less string does not pass, and that strictness is the point — the other two helpers repair what a person typed, this one checks it.