v0.4.1
Behaviour change to read before upgrading. A detected captcha on a page whose products are already rendered is no longer solved. This site carries a reCAPTCHA in its sign-up modal that guards nothing you want, and solving it spent a paid task on nothing. Pass
--solve-captcha alwaysto restore the previous behaviour.
Four fixes from a code audit, each reproduced against the old code before being changed.
A captcha could crash a run that had data
With no API key, solve_recaptcha raised RuntimeError out of the handler and out of scrape() — a traceback in place of products that were already on the page. A missing key, or any solver error, is now a warning and the run continues. If the challenge really was blocking, that surfaces as exit 3, which the project already had a distinct code for.
--solve-captcha decides whether to pay at all:
| Value | Behaviour |
|---|---|
when-blocked (new default) |
Solve only when the catalogue is not already readable. Product links are counted on the spot — no waiting — so the check is free. |
always |
Solve on any detection: for whoever would rather spend a solve than risk missing content that appears only afterwards. |
Deliberately not implemented as "run the readiness wait first": on a page the captcha genuinely gates, that would burn 20 seconds before solving, and solving first is what makes the products appear.
Proxy credentials could reach the browser's command line
The pyppeteer and Selenium engines appended the whole --proxy value to Chromium's --proxy-server — which becomes part of the browser process's argv, readable by anything that can run ps — and logged the URL verbatim. Nothing ever leaked from this repository; this was about a user's own credentials at runtime. Playwright was already correct.
Now only scheme://host:port reaches the command line and logs are masked. pyppeteer additionally sends the credentials over CDP via page.authenticate, which is supported and actually authenticates. Selenium warns that it dropped them: Chromium's flag cannot authenticate at all, and letting anyone believe a user:pass URL is doing something is worse than saying it is not.
Three legal JSON-LD shapes were mishandled
| Input | Old behaviour |
|---|---|
"offers": null |
AttributeError — run dead. A default applies only to an absent key; explicit nulls occur in the wild. |
image as an ImageObject |
KeyError — run dead over a decorative field. Now reads url/contentUrl from any shape schema.org allows. |
products inside @graph |
Silently zero products — an "empty category" reported for what is really an unread format. |
Prices: space grouping and prefixed dollars
1 234 €parsed as 234 — an order of magnitude off, silently. Plain space, NBSP (U+00A0) and narrow NBSP (U+202F) are all handled now. Space grouping requires full three-digit groups, so a size list beside a price (5 yrs, 6 yrs 200 €) cannot merge into one number.HK$1,234reported 1234 USD — the wrong currency, against this project's own cross-country comparison use.HK$,A$,C$,S$,NZ$,NT$,R$,AU$,CA$andUS$now map properly; a bare$still reads as USD, which is what it means on the US site.
Documented rather than "fixed"
- The geo-redirect claim is narrowed to what was measured. The README asserted currency follows exit IP, full stop; Farfetch also documents a customer-set shopping location with currency following the shipping destination. A fresh, cookie-less visit is redirected on exit IP — which is this scraper's case, since every rotation starts a new browser — and readers are now pointed at verifying the market in the output rather than assuming.
- The price overlay's load-bearing assumption is stated and pinned by a test: every price in a tile is taken to belong to one discount chain, so an installment price inside a tile would be read as the product price. A live 106-tile check found none — the page's Klarna/
Ratentext sits in the footer, outside any tile — so this is recorded as a known limitation rather than guarded with locale-chasing word lists or a ratio threshold that would reject this site's real 60%+ discounts.
Verified
226 offline checks, zero skips. Live runs after each change: 96 products, all priced, all with an image. A live 106-tile capture re-parsed with zero price differences against the reference run.
See CHANGELOG.md.