Rebuild on the 2scraper template: three engines, three modes, 230 offline checks

[jehrr]

Replaces the previous requests + lxml scraper with the architecture used
across this family. Nothing from the old code is source- or output-compatible;
it is kept outside the repo as a reference, not deleted from history.

48 files, +11,225 / −2,697. 239 offline checks green, and every mode
verified against the live site.

What was ported verbatim

output_writer.py, proxy_pool.py, env_config.py, diff_runs.py,
captcha_solver.py, fingerprint_client.py, scraper_api_client.py, both CI
workflows, issue templates, CONTRIBUTING/SECURITY/CODE_OF_CONDUCT, Dockerfile,
pyproject.toml.

What is new or Amazon-specific

File	Role
product_parser.py	The site. No JSON-LD path — Amazon publishes none (0 application/ld+json blocks on search pages, best-seller grids and product pages, measured 2026-09-08). Rows come from data-asin, with a /dp/{ASIN} URL-pattern fallback.
page_flow.py	The page-state policy, shared by all three engines so they cannot drift on what a page means.
three modes	--mode listing (search + best-seller grids, paginated), --mode product, --mode reviews.
21 marketplaces	Derived from --url's hostname; no --marketplace flag, so a flag and a URL cannot disagree.

Deliberate deviations from the family document

Four, each with its reason, because an audit reading the family document will
flag all of them:

1. No JSON-LD reader, and the tile-price overlay is NOT ported. There is
   no second source of truth to reconcile against, so porting the overlay
   would be dead code that looks load-bearing. price_source instead records
   which DOM node was read (offscreen / split / detail).
2. Two row schemas (family §9 has one): Product for listing and product
   modes, Review for reviews. <out>.meta.json records mode and source,
   and diff_runs.py refuses a reviews run and a mode mismatch rather than
   pretending it can compare them.
3. page_flow.py is shared rather than duplicated per engine — the same
   argument the family already makes for finish_run(), applied one step
   earlier, because Amazon's triage is far more involved than Farfetch's.
   Deliberately no JavaScript crosses that boundary: Selenium's
   execute_script needs return expr while the other two take () => expr.
4. A page column, and no prime column. data-index restarts on every
   page, so position alone is ambiguous across a paginated run. No Prime
   marker appeared on any tile captured from a cross-border exit (0 of 22 on
   .com, 0 of 16 on .de, 0 of 60 on .co.jp), so that column would have been
   null on every row of every run.

Amazon's five answers, and the four different right responses

This is the part worth reviewing most carefully.

Signal	What it is	Response
HTTP 202 + awswaf.com/challenge.js	AWS WAF JavaScript challenge	Wait. A browser takes the token and reloads itself — measured 3.9 s. Not a block, not solvable, costs nothing.
HTTP 503 "Sorry" / "Tut uns Leid"	throttle	Reload up to --retries, then a different exit.
/errors/validateCaptcha	Amazon's own image captcha	The only path a 2Captcha key is for.
/ap/signin in the final URL	needs an account	Stop. No proxy or solve changes it.
anything else	content	Parse it.

Sending the WAF challenge to the solver, or reporting it as a block, were the
first two mistakes made here. Signin is decided from the final URL and never
from the HTML: the real signin page contains /ap/signin zero times,
while every ordinary Amazon page links to it from its header.

Nine bugs found by running it live, which reading the code did not find

1. review_count was garbage on every row of every mode. An aria-label
   reading "4.4 out of 5 stars, 279,961 ratings" was read by stripping all
   digits: 445279961. Coverage looked like a healthy 100%. The suite now
   asserts values on four real fixtures, not presence.
2. Every best-seller grid reported "needs sign-in" — the header's
   /ap/signin link matched an HTML-based detector, so that whole page kind
   was unscrapeable.
3. A best-seller page returned 30 of 50 cards and reported complete. Fixed
   by scrolling to document.body.scrollHeight (not a fixed wheel distance)
   and requiring count and height to hold still for three rounds. Ranks are
   cross-checked after merging, because a per-page check cannot see a gap
   between pages.
4. --mode reviews was a coin flip. Amazon serves two page variants and
   the variant is sticky to the session — six reloads in a review-less
   context stayed empty. Only a fresh browser re-rolls it, at a measured ~55%
   hit rate, so the engines spend up to five sessions on it. Two wrong
   conclusions were drawn before that (scroll harder; reload), both killed by
   measurement.
5. The detail price was the struck-through list price.
   #corePriceDisplay_desktop_feature_div is present and empty while the
   next container holds the number, and the list price lives in the same
   container. Fixed with :not(.a-text-price) and "try selectors until one
   parses", not "until one exists".
6. The detail review_count came from a "customers also viewed" carousel
   item (13,930 instead of 87,349). Now scoped to #averageCustomerReviews.
7. Ratings were null on all 60 amazon.co.jp rows — Japanese states the
   scale first (5つ星のうち4.2).
8. 199 kr parsed as no price at all: the symbol was in the
   marketplace-resolution table but not in the matching alternation, so a
   whole locale's prices were silently invisible.
9. pyppeteer cannot launch from a worker thread (it installs signal
   handlers), and its bundled x86_64 Chromium 117 will not start on an arm64
   Mac at all. Hence handleSIGINT=False and --chromium-path.

Measured, and used as the canary's floors

From a residential European exit, 2026-09-08, with no key, no proxy and no
account:

• /s?k=bluetooth+headphones&i=electronics, 3 pages → 90 rows, 76–77 after
  dedupe, status=complete, exit 0.
• 16–30 organic tiles per search page on .com, 16 on .de, 60 on .co.jp.
• Best-seller grids: 50 cards/page, ranks 1–100 contiguous across two pages.
• 13 reviews per product page.
• Price present on 71–87% of search rows — Amazon withholds it on real
  products — and 100% of grid rows.
• Cross-border prices are converted and drift with the exchange rate: two
  runs ten minutes apart differed on 10 of 43 shared ASINs by 0.039–0.058%.
  That is what diff_runs.py --price-tolerance-pct exists for, and it
  defaults to 0 so nothing is swallowed silently.

What is NOT verified

• The image-captcha solve path. 47 navigations across five marketplaces
  produced WAF challenges and 503s and never an image captcha, so it is
  exercised only by an offline fixture written from Amazon's documented
  markup. README, TROUBLESHOOTING.md and smoke_test.py each say so.
• scraper_api_client.py against amazon.com. No Scraper API subscription
  was available; its docstring says so and points at --cdp-url.

Testing

python3 smoke_test.py — 239 checks, no network, no browser, no key. Fixtures
are cut from real captures with <script>/<style> stripped, verified to
parse identically to the untrimmed originals before being committed; exactly
one fixture is synthetic and the file names it.

Note smoke_test.py needs its # -*- coding: utf-8 -*- line: CPython 3.9's
tokenizer, with no declared encoding, fails on a multi-byte character far
inside a very long source line (the amazon.co.jp fixture is one
21,000-character line whose first Japanese character is at column 3,654) while
compile() on the same bytes succeeds.

CI: tests.yml offline only on 3.9 + 3.12, plus engine-smoke which installs
all three engines and fails on any reported skip. canary.yml runs one real
3-page listing daily with the floors above.

Fixed in the second commit, found by reading CI's log rather than its badge

The first run went green while engine-smoke verified almost nothing about
one of the three engines.

• puppeteer_scraper imported its driver inside _Session.open(), so the
  module imported cleanly with no pyppeteer installed at all. The suite guards
  that import and reports a skip when it fails, and CI fails on any reported
  skip — the whole mechanism was vacuous for this engine, because the import
  could not fail. Now at module level, like its two siblings and the rest of
  the family.
• Because nothing imported pyppeteer, CI did not notice it was testing a
  stub. pip install playwright pyppeteer selenium, unpinned into one
  environment, resolved the three engines' mutually unsatisfiable pins by
  reaching for pyppeteer 0.0.25 instead of the >=1.0.2 the requirements
  file asks for. engine-smoke is now a matrix of one venv per engine,
  installing from that engine's own requirements file — what the README tells
  users to do, and the only way to get the versions they get. pip check runs
  per venv as well.

CI now reports playwright 1.62.0 / pyee 13, pyppeteer 2.0.0 / pyee 11 /
urllib3 1.26, selenium 4.48.0 — and a new check asserts all three engines keep
importing their driver at module level, so this cannot drift back quietly.

Not changed

• The landing page (index.html) and the downloaded proxy list stay out of
  the repo — .gitignore covers both.
• No logged-in scraping: the full review history needs an account, which risks
  the user's account and could not be covered by a canary.
• The org profile README row waits until this repo is public — a link to a
  private repo is a 404 for every visitor.

🤖 Generated with Claude Code