Repository navigation
v0.1.0 — Sleep Number scraper
First release. Reads sleepnumber.com's catalogue — one row per size
variant, with its own sku, price, sale price, discount, rating, review
count and availability — through four entry points that share one parser.
Read this first: there is no captcha here, and no solver can help
Measured 2026-09-17. From a datacentre address every URL on the site —
/robots.txt included — answers an identical 919-byte CloudFront
"Request blocked" 403. It carries no x-amzn-waf-action header, no
challenge iframe and no widget of any kind, so there is nothing on that page
for any solver, at any price, to work on: the request never reaches Sleep
Number at all.
From a residential exit the same requests answer 200 with the full
catalogue — verified on a GB exit and a US exit, so the block is datacentre
reputation, not geography.
What the paid products buy here is one thing: an exit the site will
serve. The catalogue itself needs no key, no captcha solving and no
fingerprint. That is a statement about this site's refusal and never a
statement about what 2Captcha can solve.
What it does
- Two modes.
--mode listingreads a/categories/…or/collections/…
page;--mode productreads a/products/…detail page. Both produce the
same rows from the same parser, because the site's own loader data gives
them the same object shape. - Four entry points — Playwright (recommended), pyppeteer, Selenium, and
a browserless client for the 2Captcha Scraper API. All share
finish_run()andSTATE_POLICY, so they cannot drift on exit codes. - Reads the React Router hydration payload. A category page carries zero
product links and zero"sku"strings as markup — the grid is painted
client-side. The catalogue is inlined as a turbo-stream document, a flat
array in which every value is an index into that same array. This repo
ships the decoder, so the primary path never touches a CSS class and never
waits for a grid to paint. - A row is a size variant.
QCM10andKCM10are one model in two sizes
at genuinely different prices — ComfortMode spans $989.10 to $2,069.10
across seven sizes. - JSON + CSV, a 24-column schema, and a
.meta.jsonsidecar recording
status, failed page numbers and the site's own result arithmetic.
Measured
| listing | products | rows | priced | discounted |
|---|---|---|---|---|
/categories/mattresses |
7 | 52 | 52 | 52 |
/categories/sheets-pillowcases |
10 | 211 | 211 | 13 |
/categories/furniture |
20 | 169 | 169 | 157 |
Price coverage is 100% on every listing measured. The sellable catalogue is
88 products across 51 categories, from the site's own sitemap.
Traps the parser handles
- The site's own JSON-LD points at its origin host — every URL reads
http://sndotcom.fly.dev/…, the Fly application hostname. Read verbatim,
every row would link to a dev host. ?page=Nis ignored.?page=2and?page=99both return page one,
HTTP 200, with the payload's ownpagestill reading 1.--concurrencyis
refused with that reason rather than silently ignored.regularequalssaleon everything not discounted — 198 of 211 sheet
variants. Copied across it would assert a was-price on most of the
catalogue.- reCAPTCHA is configured on this site and never rendered.
recaptcha
andcaptchaeach occur once on every page, so either used as a block
marker would report the whole catalogue as challenged.
Known limitation
Selenium cannot work here from a datacentre. chromedriver cannot
authenticate a proxy, and the proxy is the whole point on this site. It
strips the credentials, warns, and reports exit 3 with the reason rather
than a misleading "0 products". Use Playwright or pyppeteer, or run Selenium
from a residential connection.
Full detail, every figure dated and reproducible, in the
README.