v0.1.3 — fingerprint_client reads .env

A one-defect patch, and the defect is that a documented mechanism was not
applied on one path.

fingerprint_client.py's --key defaulted to
os.environ.get("TWOCAPTCHA_KEY") and only that. So a key put in .env —
exactly as §3, the README and .env.example instruct — worked for every
engine and failed here with "No API key".

It now reads through env_config.env_value, calling load_env() itself
because this is a standalone entry point that no engine has necessarily run
first.

Why the loader and not os.environ

Measured both ways with TWOCAPTCHA_KEY=your_2captcha_api_key_here exported:

	What the user saw
before	Fingerprint API rejected the key (401) — note this is a separate subscription from captcha solving
after	TWOCAPTCHA_KEY is still set to the placeholder from .env.example — treating it as unset

The old message sent the reader off to check a subscription they never
needed.

Provenance

Found on a sibling repo's first live --fingerprint run, then checked
across the family before patching: five repos had it, one had already fixed
it. This release makes all seven identical.

Pinned by a check verified to fail on the old code — that it loads .env
itself, reads through the loader, does not read os.environ directly,
that a placeholder still reads as unset, and that the help string does not
interpolate its default, which is one substring away from printing a live
credential to anyone who types --help.

Nothing else changed. No flag, no default, no output column.

v0.1.2 — the fingerprint's device scale factor is applied

If you use --fingerprint, this one changes what the browser reports.

playwright_context_kwargs mapped the user agent, the locale, the timezone
and the screen onto the browser context and ignored deviceScaleFactor,
which the fingerprint API returns beside them.

Measured 2026-09-11 against the live API and a live browser:

	fingerprint says	browser reported, before
user agent	Chrome/146 on Windows	✅ same
locale	nl-NL	✅ same
timezone	Europe/Amsterdam	✅ same
screen width	1536	✅ same
devicePixelRatio	1.25	1 ❌

That is the mismatch the flag exists to prevent, on an axis any
fingerprinter reads for free and for nothing: the paid identity said one
thing and the browser said another, on every run, silently.

Playwright takes it as its own context option, so the fix is to pass it.
Verified in a live browser both ways and pinned in the offline suite.

Found while auditing a new sibling repo against the family notes — all
five repos in this family had it.

🤖 Generated with Claude Code

v0.1.1 — fixture hygiene

Nothing changes for a caller. No flag, column, exit code or output shape
moved from v0.1.0. If you are scripting against this, v0.1.1 is a drop-in.

Use this release rather than v0.1.0 — its source archive is the scrubbed
one.

Fixed

• The fixtures carried session material and one real person's data. A real
  page capture brings the session that fetched it — three anonymous, expired
  sessionId values and their CSRF tokens — and, on a review, a real
  customer's display name, profile permalink, review id, uploaded-photo ids
  and words. All of it is now an obvious placeholder. Everything Amazon
  generates around it (the accessibility filler that is part of the body
  node's text, the date wording, the badges, the markup) is untouched, because
  that is what the checks actually read: the structure of a review, not the
  person.
• A guard so it does not come back with the next capture. Patterns rather
  than literals, so a new capture's values are caught too, and verified to
  fire rather than only to stay quiet.
• Both Claude workflows failed instead of skipping without their token,
  putting a permanently red check on every pull request. A check that is
  always red teaches everyone to ignore checks.

244 offline checks (was 239).

Known, and not reachable from here

The pre-scrub values remain in this repository's history — in the v0.1.0 tag
and in PR #1's own refs, which stay attached to that pull request. What that
history holds is three expired anonymous session ids, their dead CSRF tokens,
and one Amazon review that is public on amazon.com today. Removing it entirely
would need a fresh repository, not a commit on top.

v0.1.0 — rebuilt on the 2scraper template

This replaces the previous scraper entirely. If you were using the old
requests + lxml code (main.py, scraper.py, parser.py,
marketplace.py, output.py), nothing carries over: the CLI, the module
layout and the output format are all different. There is no upgrade path and
no compatibility shim — pin the previous commit if you depend on it.

Two changes to be aware of before you script against this:
--2captcha-key is now --twocaptcha-key, and a key never belongs on a
command line anyway (put it in .env); and a run that finds nothing now
writes nothing and exits 4, instead of overwriting your last good output
with an empty file. Pass --allow-empty if an empty result is the answer you
want recorded.

First release on the architecture shared across this family of scrapers.

What it does

Mode	Reads
--mode listing (default)	Search results (/s?k=) and best-seller grids (/zgbs/), paginated
--mode product	One /dp/{ASIN} page — brand, seller, bullets, images, availability
--mode reviews	The reviews Amazon renders to a visitor with no account (~13 per product)

All 21 marketplaces, chosen by the hostname of --url — there is no
--marketplace flag, so a flag and a URL cannot disagree about which site a
run is reading.

Four engines: Playwright (primary), Selenium, pyppeteer, and a
browserless client for the 2Captcha Scraper API. They share the page-state
policy (page_flow.py) and the status/exit-code mapping
(output_writer.finish_run), so they cannot drift on what a page means or
what a run reports.

It works with no key, no proxy and no account

Measured 2026-09-08 from a residential European exit, plain local Chromium:

$ python3 playwright_scraper.py \
    --url "https://www.amazon.com/s?k=bluetooth+headphones&i=electronics" --pages 3

Parsed 30 rows from page 1.   Price coverage on page 1: 24/30 (80%).
Parsed 30 rows from page 2.   Price coverage on page 2: 22/30 (73%).
Parsed 30 rows from page 3.   Price coverage on page 3: 25/30 (83%).
Saved 77 products -> amazon_products.json     # 90 rows, 13 duplicates dropped
Wrote run metadata -> amazon_products.meta.json (status=complete)

Amazon's front door met that run twice and neither time cost anything: an AWS
WAF JavaScript challenge (a browser clears it in 3.9 s by itself) and a 503
throttle (a retry cleared it). The README lists what the paid products actually
buy on top of that.

Amazon answers a request five ways, and four want different responses

Signal	Response
HTTP 202 + awswaf.com/challenge.js	Wait. Not a block, not solvable, unbilled
HTTP 503 "Sorry" / "Tut uns Leid"	Retry, then a different exit
/errors/validateCaptcha	The only path a 2Captcha key is for
/ap/signin in the final URL	Stop — no proxy or solve changes it
anything else	Parse it

Collapsing those into one "blocked" is how a run either gives up on a page it
would have got, or pays for a challenge no solver can answer.

Notable in the parser

• No JSON-LD path, because Amazon publishes none — 0
  application/ld+json blocks on search pages, best-seller grids and product
  pages. Rows come from data-asin, with a /dp/{ASIN} URL-pattern fallback.
• The url column is rebuilt as https://www.{host}/dp/{ASIN}: a
  sponsored tile's link is a click tracker with the product path
  percent-encoded inside it, so reading hrefs both misses sponsored products
  and stores trackers.
• Currency follows the exit IP, not the domain. From a European exit,
  amazon.co.uk and amazon.co.jp both quote EUR — and the converted price drifts
  with the exchange rate, which is what diff_runs.py --price-tolerance-pct
  exists for (default 0, so nothing is swallowed silently).

Honest gaps

• The image-captcha solve path is unproven against the live site. 47
  navigations across five marketplaces produced WAF challenges and 503s and
  never an image captcha, so it is exercised only by an offline fixture written
  from Amazon's documented markup.
• scraper_api_client.py is untested against amazon.com — no Scraper API
  subscription was available.
• brand is null on every listing row (Amazon's search markup has no
  brand field), and there is no prime column (no Prime marker appeared on
  any tile captured from a cross-border exit, so it would have been null on
  every row).
• --mode reviews has no --pages: /product-reviews/{ASIN} redirects to
  /ap/signin on page 1, so the paginated history is not available to an
  anonymous visitor at all.

Testing

239 offline checks (python3 smoke_test.py) with fixtures cut from real
captures and verified to parse identically to the untrimmed originals; a daily
canary against a real 3-page listing with floors set from the measurements
above; tests.yml on Python 3.9 and 3.12, plus engine-smoke as one venv per
engine so each is tested at the version users actually install.

Full detail, including the nine bugs that only a live run found, is in
CHANGELOG.md
and PR #1.