Skip to content

v1.5.0

Choose a tag to compare

@github-actions github-actions released this 01 Aug 05:36
· 12 commits to main since this release

Added

  • BidiSolver — Firefox, over WebDriver BiDi. A clearance binds to the fingerprint that earned it, so a Chrome-only solver forced chrome impersonation on every request to every origin — and firefox is the profile that reaches the most hosts. This declares impersonation = "firefox" and the trade disappears. Clearing itself is a tie: 29 of 46 challenged hosts against Chrome's 28, disagreeing on seven, four to Firefox and three to Chrome. The transport moved to scraper.wire and both backends share it unchanged — a second vocabulary, not a second client.

    One cost, stated rather than buried. A WebDriver session sets navigator.webdriver true by spec and no preference overrides it, so this deletes the property with a preload script. That is patching a surface value rather than not emitting one, and a site reading the prototype descriptor would see through it. Worth nineteen of the 46 hosts; ten challenge without ever reading it.

  • find_chromium, find_firefox, pick_chromium, pick_firefox — finding an installed browser is the library's job now. Both solvers looked for one with a PATH scan, which answers for Linux and only Linux: on macOS with Chrome and Firefox installed, CdpSolver() raised "no browser executable was found", and the same on Windows. It went unnoticed because the one consumer always passed executable=. This looks inside macOS application bundles, under the Windows program directories, and across distribution and flatpak paths — six Chromium brands, since Brave and Edge answer CDP exactly as Chrome does, plus Firefox with ESR and LibreWolf.

Removed

  • NoDriverSolver and the nodriver dependency are gone. CdpSolver does the same job on every Python this package supports and was not worse head to head — 12 hosts cleared to 11, at the same median — against fifty thousand lines of dependency and a marker excluding two interpreters. This is a breaking change: replace scraper.NoDriverSolver with scraper.CdpSolver. The [browser] extra stays as an alias for [cdp].

Changed

  • Headless is the default. It does not cost a solve, measured over 46 hosts, and most places this runs have no display to put a window on. Pass headless=False for one, which also buys interactive_solve_timeout.
  • A solve timeout now bounds the whole solve, not just the part after the navigation, which could double the budget the caller asked for.

Fixed

  • In a container, set TZ before blaming anything else. A container runs on UTC, and a browser whose timezone contradicts where its address geolocates reads as automation: same six hosts, same binary, same egress address — 1 of 6 under UTC, 6 of 6 with TZ matching the exit. Debian's chromium, which browser.py and docs/tiers.md previously named as the cause, is real but second at 1 of 6 with the clock right.
  • validate_url's docstring described it backwards. A scheme-less string does not pass, and that strictness is the point — the other two helpers repair what a person typed, this one checks it.