v0.4.2 — the fingerprint's device scale factor is applied

If you use --fingerprint, this one changes what the browser reports.

playwright_context_kwargs mapped the user agent, the locale, the timezone
and the screen onto the browser context and ignored deviceScaleFactor,
which the fingerprint API returns beside them.

Measured 2026-09-11 against the live API and a live browser:

	fingerprint says	browser reported, before
user agent	Chrome/146 on Windows	✅ same
locale	nl-NL	✅ same
timezone	Europe/Amsterdam	✅ same
screen width	1536	✅ same
devicePixelRatio	1.25	1 ❌

That is the mismatch the flag exists to prevent, on an axis any
fingerprinter reads for free and for nothing: the paid identity said one
thing and the browser said another, on every run, silently.

Playwright takes it as its own context option, so the fix is to pass it.
Verified in a live browser both ways and pinned in the offline suite.

Found while auditing a new sibling repo against the family notes — all
five repos in this family had it.

🤖 Generated with Claude Code

v0.4.1

Behaviour change to read before upgrading. A detected captcha on a page whose products are already rendered is no longer solved. This site carries a reCAPTCHA in its sign-up modal that guards nothing you want, and solving it spent a paid task on nothing. Pass --solve-captcha always to restore the previous behaviour.

Four fixes from a code audit, each reproduced against the old code before being changed.

A captcha could crash a run that had data

With no API key, solve_recaptcha raised RuntimeError out of the handler and out of scrape() — a traceback in place of products that were already on the page. A missing key, or any solver error, is now a warning and the run continues. If the challenge really was blocking, that surfaces as exit 3, which the project already had a distinct code for.

--solve-captcha decides whether to pay at all:

Value	Behaviour
when-blocked (new default)	Solve only when the catalogue is not already readable. Product links are counted on the spot — no waiting — so the check is free.
always	Solve on any detection: for whoever would rather spend a solve than risk missing content that appears only afterwards.

Deliberately not implemented as "run the readiness wait first": on a page the captcha genuinely gates, that would burn 20 seconds before solving, and solving first is what makes the products appear.

Proxy credentials could reach the browser's command line

The pyppeteer and Selenium engines appended the whole --proxy value to Chromium's --proxy-server — which becomes part of the browser process's argv, readable by anything that can run ps — and logged the URL verbatim. Nothing ever leaked from this repository; this was about a user's own credentials at runtime. Playwright was already correct.

Now only scheme://host:port reaches the command line and logs are masked. pyppeteer additionally sends the credentials over CDP via page.authenticate, which is supported and actually authenticates. Selenium warns that it dropped them: Chromium's flag cannot authenticate at all, and letting anyone believe a user:pass URL is doing something is worse than saying it is not.

Three legal JSON-LD shapes were mishandled

Input	Old behaviour
"offers": null	AttributeError — run dead. A default applies only to an absent key; explicit nulls occur in the wild.
image as an ImageObject	KeyError — run dead over a decorative field. Now reads url/contentUrl from any shape schema.org allows.
products inside @graph	Silently zero products — an "empty category" reported for what is really an unread format.

Prices: space grouping and prefixed dollars

• 1 234 € parsed as 234 — an order of magnitude off, silently. Plain space, NBSP (U+00A0) and narrow NBSP (U+202F) are all handled now. Space grouping requires full three-digit groups, so a size list beside a price (5 yrs, 6 yrs 200 €) cannot merge into one number.
• HK$1,234 reported 1234 USD — the wrong currency, against this project's own cross-country comparison use. HK$, A$, C$, S$, NZ$, NT$, R$, AU$, CA$ and US$ now map properly; a bare $ still reads as USD, which is what it means on the US site.

Documented rather than "fixed"

• The geo-redirect claim is narrowed to what was measured. The README asserted currency follows exit IP, full stop; Farfetch also documents a customer-set shopping location with currency following the shipping destination. A fresh, cookie-less visit is redirected on exit IP — which is this scraper's case, since every rotation starts a new browser — and readers are now pointed at verifying the market in the output rather than assuming.
• The price overlay's load-bearing assumption is stated and pinned by a test: every price in a tile is taken to belong to one discount chain, so an installment price inside a tile would be read as the product price. A live 106-tile check found none — the page's Klarna/Raten text sits in the footer, outside any tile — so this is recorded as a known limitation rather than guarded with locale-chasing word lists or a ratio threshold that would reject this site's real 60%+ discounts.

Verified

226 offline checks, zero skips. Live runs after each change: 96 products, all priced, all with an image. A live 106-tile capture re-parsed with zero price differences against the reference run.

See CHANGELOG.md.

v0.4.0

Concurrency. --concurrency N (Playwright) fetches pages through N parallel workers. It defaults to 1, so the default run is exactly the sequential one.

python3 playwright_scraper.py --url "$URL" --pages 20 \
  --concurrency 4 --proxy-file exits.txt

Measured on a live 4-page run

--concurrency 3 :  57s
sequential      :  98s

rows: 333 both ways
SKU ORDER identical: True     <- not merely the same set
field diffs: 0

Matching order rather than just the set is the payoff from the refactor below: the order pages happen to arrive in no longer reaches the output at all.

Design

A worker owns its browser. Playwright's sync API ties a browser to the thread that created it, so a shared browser is not an option even in principle — each worker opens its own.

A worker owns one exit for its lifetime. Not an exit per page: the invariant from 0.3.0 is that a session must not change address mid-flight, and a worker is one session. Workers start on different exits and can walk the rest of the pool if one gets blocked. Each holds its own pool object, so no thread needs a lock — safe by construction rather than by discipline.

Guardrails rather than silent behaviour. Raising concurrency without --proxy-file warns that N workers send N times the traffic from one address. It is refused with --cdp-endpoint, where the Scraping Browser API allows one live connection per profile (profile_locked). Above 8 workers it warns about memory.

Page 1 is always fetched alone, because its content decides whether pages 2..N can be addressed independently at all. A listing paginated with a cursor rather than ?page=N falls back to one page at a time and says so.

Dispatch stops at the end of the listing — a page returning no products sets a shared event, so asking for 50 pages of a 5-page category costs at most concurrency - 1 extra fetches rather than 45.

Also in this release

The groundwork that made the above possible, with no behaviour change of its own:

• Page URLs are planned up front from page 1 rather than chained off each previous page's next-link — but only when the site's own link agrees with the ?page=N convention, which is verified rather than assumed.
• Results are merged in page order, not arrival order. Dedupe that mutated a running set inside the loop made the output depend on the order pages arrived in — harmless while that order was fixed, wrong the moment fetches overlap.
• pages_failed added to the run-metadata sidecar. pages_completed described a run only while pages were strictly ordered: "3 of 10" could only mean 1-2-3.

Verified

200 offline checks, zero skips. Live: --concurrency 1 byte-identical to 0.3.0 (174 rows, identical SKU order, zero diffs); --concurrency 3 byte-identical to sequential across 4 pages.

See CHANGELOG.md and the Concurrency section.

v0.3.0

Proxy rotation. --proxy was a single static string applied once at browser launch and never changed — the shape of a demo, not of the thing proxies are bought for. This release adds the pattern that makes a pool worth holding.

cat > exits.txt <<'LIST'
# one proxy URL per line; blanks and # comments ignored
http://ACCOUNT:PASSWORD@HOST:9999
http://ACCOUNT:PASSWORD@HOST:10000
LIST

python3 playwright_scraper.py --url "$URL" --pages 20 \
  --proxy-file exits.txt --proxy-rotate per-page --proxy-shuffle

Flag	Default	Description
--proxy-file	–	One proxy URL per line. Wins over --proxy, and logs that it did.
--proxy-rotate	per-run	per-run: one exit per run. per-page: a new exit every page.
--proxy-shuffle	off	Don't have concurrent runs all start on the first line.
--proxy-block-retries	2	Retry a challenged page from this many other exits before giving up.

Three decisions where the cheap version would have been wrong

A rotation relaunches the browser. Swapping the proxy under a live session would be cheaper and wrong: cookies a bot manager issued against exit A, replayed from exit B, are a stronger signal than either address alone. Each exit gets a genuinely fresh cookie jar and storage. per-page therefore costs a browser start per page, and per-run stays the default because a session that changes address mid-flight is itself suspicious.

An unusable exit rotates rather than burning retries. Found live against an unreachable proxy: Chromium reports this as a PWError (net::ERR_PROXY_CONNECTION_FAILED), not a PWTimeout — so before this it escaped as an unhandled traceback, which is the likeliest failure the first time anyone points --proxy-file at a real list. A timeout deserves another try at the same exit; a dead proxy deserves a different one.

Credentials never reach server. Playwright passes that string to Chromium as a command-line switch, so a user:pass left in it would sit in the browser's argv for anything that can run ps. They go in the username/password fields instead. Logs mask credentials but keep host and port — which exit a run used is the point of the log, and is not the secret.

A malformed proxy list is rejected at load with the offending line named and exit 2, rather than surfacing as a connection failure on page 1 with nothing pointing at the cause.

Scope

Rotation is Playwright-only; --proxy still works on every engine. Both are ignored with --cdp-endpoint, where the remote browser brings its own exit.

Verified

183 offline checks (17 new, covering validation, credential masking, argv safety, wrap-around rotation, and proxy-error-vs-timeout classification). Live: the no-proxy path unchanged at 174 products over 2 pages; a deliberately dead 2-exit pool produced the full intended sequence — unusable exit → rotate → unusable exit → rotate → clean exit 4, no traceback, credentials masked throughout. A successful run through a working proxy was not tested — no live proxy credentials were available.

See CHANGELOG.md and the Proxies section.

v0.2.1

Patch release: four fixes, one of which was a live bug rather than a hypothetical.

Multi-page runs were silently returning page 1

Measured live on 2026-09-07: farfetch.com serves no anchor matching any of the three NEXT_PAGE_SELECTOR entries this project shipped with. So --pages 3 fetched one page and exited 0 — a complete-looking, successful run holding a third of the data.

Three layers now, weakest signal last:

• <link rel="next"> leads the selector list — the site does serve it (in <head>), and a W3C/SEO convention outlives a build-generated data-testid.
• product_parser.page_url() reconstructs ?page=N when no selector matches at all, preserving existing filters and replacing rather than duplicating an existing page param.
• The loop terminates on a page contributing no new sku — a property of the data — instead of on a missing link, a property of a selector.

Verified: 263 unique products across 3 pages from a residential IP, 256 from a datacenter one, status: complete both times.

Page-load retries

A navigation timeout ended the whole run on the first failure — one network flap on page 12 of 50 discarded the rest. --retries (default 3) with a doubling --retry-delay, on every engine. scraper_api_client.py had retries all along; the asymmetry inside one repo was the bug.

Empty CSV now carries its header

write_csv([]) wrote zero bytes, so a consumer of --allow-empty output failed on read instead of reading a valid table with no rows.

A CI job that skipped its own checks

engine-smoke installed playwright and pyppeteer but not selenium, so every selenium-guarded check group skipped in CI — hiding a crash in a mock that had gone stale against the UA change in v0.1.1. The job now installs all three engines and fails if any group reports skipped.

Canary

Requests 3 pages, not 1 — with one page, pagination is never exercised, which is exactly how the bug above stayed invisible. It also now asserts pages_completed, status == "complete" and DOM price-confirmation coverage rather than just a product count.

First scheduled-workflow run (dispatched manually, from a GitHub Actions datacenter IP, no proxy and no API key): passed, 256 products across 3 pages.

See CHANGELOG.md for details.

v0.2.0

First release verified against the live site. A real run of playwright_scraper.py on the README's own known-good category URL, from a residential IP with no proxy, no API key and no paid product, returned 96 products and cleared Akamai silently — confirming the README's central claim, which every earlier release had only fixture-level evidence for. The documented geo-redirect was confirmed at the same time (.com/shopping → .com/de/shopping, EUR, localised titles), as were the compounded-discount figures the tests were built against.

Changed

• Output is now sixteen columns, not fifteen: Product gained price_source. Anything parsing the CSV header or asserting a column count needs updating.
• A JSON-LD offer with no priceCurrency now yields currency: null instead of a guessed "USD", and the DOM discount-overlay no longer overwrites a currency the structured data stated explicitly.

Added

• price_source — says how much to trust price: jsonld+dom (the rendered tile was found and reconciled — trustworthy), jsonld (structured data only; the tile was missing or disagreed, so on a discounted item this may be the pre-promo price), or dom (CSS fallback, no cross-check). The same column previously held all three with no way to tell them apart.
• Run metadata sidecar — every run that writes output also writes <out>.meta.json with status (complete/partial/failed), stop_reason, pages requested vs. completed, and the start/final URL. A failed run writes no sidecar, so it can't contradict the previous run's still-intact output.
• Exit code 6 for a partial run — a timeout or challenge partway through pagination still saves what it gathered but no longer looks identical to a complete run. The site's own pagination running out still exits 0: there was nothing more to fetch.
• 3-letter ISO currency codes in the fallback parser (AED 100, 100 CHF), matched against an allowlist of real ISO 4217 codes so a size chart (XXL 100) can't become a phantom price. Such tiles previously matched nothing and were dropped as "not a product", losing every product on those locales.
• diff_runs.py refuses an assortment diff when either side's sidecar says the run was partial (products on never-fetched pages would otherwise read as delisted); --force opts out. It also reports a source_changed bucket for a price that differs alongside price_source — two snapshots that rendered differently, not a site-side change — which --fail-on-change ignores.
• The parser logs DOM price-confirmation coverage per run, warning below 90%.

Fixed

• canary.yml never recorded the scraper's exit code. GitHub Actions runs run: steps under bash -e, so a non-zero exit aborted the script before the line that captured it — leaving the entire exit-3-vs-4-vs-124 interpretation dead in exactly the cases it existed for. The canary now also sanity-checks the data it fetched (≥10 products, ≥90% with a non-null price) rather than treating "exit 0" as success.

See CHANGELOG.md for the full list and README.md for usage.

v0.1.1

Six P0 correctness fixes (#4), verified against actual code before fixing, each pinned by a smoke_test.py regression:

• solve_recaptcha() was crashing — callers passed api_version=/min_score= it no longer accepted, and its body called a function (_solve_with_2captcha) that didn't exist. Restored dispatch to _solve_with_2captcha_v1/_v2.
• Thousands-separator bug: $1,234 parsed as 1.234. Fixed in product_parser._prices_in — a single separator with exactly 3 trailing digits is now read as a thousands grouping (none of the four supported currencies use 3-digit decimal subunits).
• Currency corruption: the DOM discount-overlay was overwriting a correct JSON-LD priceCurrency with a $→USD guess, silently turning AUD/CAD/SGD/HKD/NZD rows into USD.
• Exit code 3 (blocked) was unreachable from any browser engine — a challenge page parsed to 0 products and exited 4, indistinguishable from a genuinely empty category. Shared detect_bot_challenge across all four engines.
• Hardcoded Chrome/124.0.0.0 UA in all three browser engines, now built from each browser's own real launched version at runtime.
• Playwright pagination bug: raw get_attribute("href") concatenated by hand instead of urljoin, breaking on absolute-path hrefs.

Also: a new engine-smoke CI job installs playwright+pyppeteer so the previously-always-skipped engine-specific checks actually run in CI.

See README.md for usage, and CONTRIBUTING.md for the properties tests pin.

v0.1.0

First tagged release.

Engines: Playwright (recommended), Selenium, pyppeteer, and a browserless HTTP client via the 2Captcha Scraper API — same CLI, same parsing core, same JSON/CSV output.

Parsing: JSON-LD primary, CSS + URL-pattern fallback. Two-stage discount correction (tile prices overlaid on JSON-LD) recovers original_price/discount_pct on discounted listings.

Optional 2Captcha integrations: captcha solving (reCAPTCHA v3/v2, auto-reconciled against the site's own widget), the Scraping Browser API over --cdp-endpoint, proxies, and fingerprints — all optional, none required to run against a listing page.

Since the initial commit:

• Cross-page dedup by sku (dedupe_by_sku in output_writer.py), wired into all three browser engines' pagination loop.
• diff_runs.py — diffs two JSON outputs by sku (added/removed/changed), for scheduled price/assortment monitoring.
• .github/workflows/canary.yml — a daily live run against farfetch.com, since the main tests workflow is deliberately offline-only.
• pyproject.toml (installable via pip install .[playwright] etc.), tests/test_smoke.py (pytest entry point), Dockerfile/.dockerignore.

See README.md for full usage, flags, and exit codes.