v0.2.0 — the browser is optional on this site

Behaviour change: --retries 0 is now a usage error (exit 2) in every
entry point. It used to be accepted and made the load loop body
unreachable — nothing was fetched, and the engine then reported the
browser's own blank page as a block: exit 3, with a 39-byte "capture", in
about 40 milliseconds. If a script of yours passes 0, pass 1: the flag
counts ATTEMPTS, so 1 means "try once and do not retry".

Also, --retries on scraper_api_client.py used to mean extra attempts
while meaning attempts everywhere else. It now means attempts there too,
and its default moves 1 → 2 so the number of requests actually made is
unchanged.

Worked through a third-party audit dated 2026-10-13.

http_scraper.py — no browser at all

Sleep Number server-renders its whole catalogue into the page, so there is
nothing to wait for and nothing to scroll. The README has said so since
v0.1.0; the recommended path launched Chromium anyway. Measured 2026-10-13
through one US residential exit, three runs each:

	run 1	run 2	run 3	median
http_scraper.py	1.4s	1.3s	2.1s	1.4s
playwright_scraper.py	4.2s	2.9s	64.2s	4.2s

52 rows each, identical on every field. It keeps the whole output
contract and deliberately takes no --headless, --cdp-endpoint,
--fingerprint or captcha flags — there is no browser to configure, and no
challenge has ever been observed on this site.

It still needs an exit the site serves. Removing the browser does not
change the address, and the address is the whole difficulty here.

A run is now checked against the site's own count

The payload states total_results, and on 2026-10-13 it equalled the
products in the payload exactly across four categories (7, 10, 20, 9). A run
that parses fewer distinct products than the site counted now reports
partial (exit 6) with stop_reason: parser_incomplete and
products_missing in the sidecar, instead of exit 0 and a quietly short
file.

That replaces a coverage threshold: a threshold is a guess about how much
loss is acceptable, and total_results is the response proving the loss.

Fixed

• --retries 0 fetched nothing and called it a block. The audit ran
  exactly that and reported the site unreachable; its 39-byte evidence was
  about:blank. The site does refuse datacentre addresses — --retries 1
  gets the real 923-byte CloudFront 403 — so the conclusion was right and
  the evidence was about our bug.
• The canary asserted a sale that has since ended. It required the three
  canary categories to disagree about discounting by 20 points, from figures
  measured during a September sale. On 13 October they read 0, 11 of 194 and
  0, with the parser working perfectly — the check would have been red every
  day for a catalogue that simply is not discounted. Removed; the discount
  column is still guarded by assertions that hold either way.
• A test fixture claimed what no response says, inheriting
  total_results from the full capture while its products were trimmed.
• The canary now runs both transports daily and asserts they agree field for
  field.

Documentation

The README promised a sidecar "per run" and meant "per run that writes
rows" — a blocked run deliberately writes neither file, and now says so.
Figures re-measured rather than inherited: 91 products across 52
categories today against 88 and 51 in September.

---

69 offline checks, passing in each of three engine virtualenvs; ten new
controls, all valid.

v0.1.0 — Sleep Number scraper

First release. Reads sleepnumber.com's catalogue — one row per size
variant, with its own sku, price, sale price, discount, rating, review
count and availability — through four entry points that share one parser.

Read this first: there is no captcha here, and no solver can help

Measured 2026-09-17. From a datacentre address every URL on the site —
/robots.txt included — answers an identical 919-byte CloudFront
"Request blocked" 403. It carries no x-amzn-waf-action header, no
challenge iframe and no widget of any kind, so there is nothing on that page
for any solver, at any price, to work on: the request never reaches Sleep
Number at all.

From a residential exit the same requests answer 200 with the full
catalogue — verified on a GB exit and a US exit, so the block is datacentre
reputation, not geography.

What the paid products buy here is one thing: an exit the site will
serve. The catalogue itself needs no key, no captcha solving and no
fingerprint. That is a statement about this site's refusal and never a
statement about what 2Captcha can solve.

What it does

• Two modes. --mode listing reads a /categories/… or /collections/…
  page; --mode product reads a /products/… detail page. Both produce the
  same rows from the same parser, because the site's own loader data gives
  them the same object shape.
• Four entry points — Playwright (recommended), pyppeteer, Selenium, and
  a browserless client for the 2Captcha Scraper API. All share
  finish_run() and STATE_POLICY, so they cannot drift on exit codes.
• Reads the React Router hydration payload. A category page carries zero
  product links and zero "sku" strings as markup — the grid is painted
  client-side. The catalogue is inlined as a turbo-stream document, a flat
  array in which every value is an index into that same array. This repo
  ships the decoder, so the primary path never touches a CSS class and never
  waits for a grid to paint.
• A row is a size variant. QCM10 and KCM10 are one model in two sizes
  at genuinely different prices — ComfortMode spans $989.10 to $2,069.10
  across seven sizes.
• JSON + CSV, a 24-column schema, and a .meta.json sidecar recording
  status, failed page numbers and the site's own result arithmetic.

Measured

listing	products	rows	priced	discounted
/categories/mattresses	7	52	52	52
/categories/sheets-pillowcases	10	211	211	13
/categories/furniture	20	169	169	157

Price coverage is 100% on every listing measured. The sellable catalogue is
88 products across 51 categories, from the site's own sitemap.

Traps the parser handles

• The site's own JSON-LD points at its origin host — every URL reads
  http://sndotcom.fly.dev/…, the Fly application hostname. Read verbatim,
  every row would link to a dev host.
• ?page=N is ignored. ?page=2 and ?page=99 both return page one,
  HTTP 200, with the payload's own page still reading 1. --concurrency is
  refused with that reason rather than silently ignored.
• regular equals sale on everything not discounted — 198 of 211 sheet
  variants. Copied across it would assert a was-price on most of the
  catalogue.
• reCAPTCHA is configured on this site and never rendered. recaptcha
  and captcha each occur once on every page, so either used as a block
  marker would report the whole catalogue as challenged.

Known limitation

Selenium cannot work here from a datacentre. chromedriver cannot
authenticate a proxy, and the proxy is the whole point on this site. It
strips the credentials, warns, and reports exit 3 with the reason rather
than a misleading "0 products". Use Playwright or pyppeteer, or run Selenium
from a residential connection.

Full detail, every figure dated and reproducible, in the
README.