v0.4.0

Concurrency. --concurrency N (Playwright) fetches pages through N parallel workers. It defaults to 1, so the default run is exactly the sequential one.

python3 playwright_scraper.py --url "$URL" --pages 20 \
  --concurrency 4 --proxy-file exits.txt

Measured on a live 4-page run

--concurrency 3 :  57s
sequential      :  98s

rows: 333 both ways
SKU ORDER identical: True     <- not merely the same set
field diffs: 0

Matching order rather than just the set is the payoff from the refactor below: the order pages happen to arrive in no longer reaches the output at all.

Design

A worker owns its browser. Playwright's sync API ties a browser to the thread that created it, so a shared browser is not an option even in principle — each worker opens its own.

A worker owns one exit for its lifetime. Not an exit per page: the invariant from 0.3.0 is that a session must not change address mid-flight, and a worker is one session. Workers start on different exits and can walk the rest of the pool if one gets blocked. Each holds its own pool object, so no thread needs a lock — safe by construction rather than by discipline.

Guardrails rather than silent behaviour. Raising concurrency without --proxy-file warns that N workers send N times the traffic from one address. It is refused with --cdp-endpoint, where the Scraping Browser API allows one live connection per profile (profile_locked). Above 8 workers it warns about memory.

Page 1 is always fetched alone, because its content decides whether pages 2..N can be addressed independently at all. A listing paginated with a cursor rather than ?page=N falls back to one page at a time and says so.

Dispatch stops at the end of the listing — a page returning no products sets a shared event, so asking for 50 pages of a 5-page category costs at most concurrency - 1 extra fetches rather than 45.

Also in this release

The groundwork that made the above possible, with no behaviour change of its own:

• Page URLs are planned up front from page 1 rather than chained off each previous page's next-link — but only when the site's own link agrees with the ?page=N convention, which is verified rather than assumed.
• Results are merged in page order, not arrival order. Dedupe that mutated a running set inside the loop made the output depend on the order pages arrived in — harmless while that order was fixed, wrong the moment fetches overlap.
• pages_failed added to the run-metadata sidecar. pages_completed described a run only while pages were strictly ordered: "3 of 10" could only mean 1-2-3.

Verified

200 offline checks, zero skips. Live: --concurrency 1 byte-identical to 0.3.0 (174 rows, identical SKU order, zero diffs); --concurrency 3 byte-identical to sequential across 4 pages.

See CHANGELOG.md and the Concurrency section.