Skip to content

v0.1.0 — Sleep Number scraper

Choose a tag to compare

@jehrr jehrr released this 17 Sep 15:29
· 11 commits to main since this release

First release. Reads sleepnumber.com's catalogue — one row per size
variant
, with its own sku, price, sale price, discount, rating, review
count and availability — through four entry points that share one parser.

Read this first: there is no captcha here, and no solver can help

Measured 2026-09-17. From a datacentre address every URL on the site —
/robots.txt included — answers an identical 919-byte CloudFront
"Request blocked" 403. It carries no x-amzn-waf-action header, no
challenge iframe and no widget of any kind, so there is nothing on that page
for any solver, at any price, to work on: the request never reaches Sleep
Number at all.

From a residential exit the same requests answer 200 with the full
catalogue — verified on a GB exit and a US exit, so the block is datacentre
reputation, not geography.

What the paid products buy here is one thing: an exit the site will
serve.
The catalogue itself needs no key, no captcha solving and no
fingerprint. That is a statement about this site's refusal and never a
statement about what 2Captcha can solve.

What it does

  • Two modes. --mode listing reads a /categories/… or /collections/…
    page; --mode product reads a /products/… detail page. Both produce the
    same rows from the same parser, because the site's own loader data gives
    them the same object shape.
  • Four entry points — Playwright (recommended), pyppeteer, Selenium, and
    a browserless client for the 2Captcha Scraper API. All share
    finish_run() and STATE_POLICY, so they cannot drift on exit codes.
  • Reads the React Router hydration payload. A category page carries zero
    product links and zero "sku" strings as markup — the grid is painted
    client-side. The catalogue is inlined as a turbo-stream document, a flat
    array in which every value is an index into that same array. This repo
    ships the decoder, so the primary path never touches a CSS class and never
    waits for a grid to paint.
  • A row is a size variant. QCM10 and KCM10 are one model in two sizes
    at genuinely different prices — ComfortMode spans $989.10 to $2,069.10
    across seven sizes.
  • JSON + CSV, a 24-column schema, and a .meta.json sidecar recording
    status, failed page numbers and the site's own result arithmetic.

Measured

listing products rows priced discounted
/categories/mattresses 7 52 52 52
/categories/sheets-pillowcases 10 211 211 13
/categories/furniture 20 169 169 157

Price coverage is 100% on every listing measured. The sellable catalogue is
88 products across 51 categories, from the site's own sitemap.

Traps the parser handles

  • The site's own JSON-LD points at its origin host — every URL reads
    http://sndotcom.fly.dev/…, the Fly application hostname. Read verbatim,
    every row would link to a dev host.
  • ?page=N is ignored. ?page=2 and ?page=99 both return page one,
    HTTP 200, with the payload's own page still reading 1. --concurrency is
    refused with that reason rather than silently ignored.
  • regular equals sale on everything not discounted — 198 of 211 sheet
    variants. Copied across it would assert a was-price on most of the
    catalogue.
  • reCAPTCHA is configured on this site and never rendered. recaptcha
    and captcha each occur once on every page, so either used as a block
    marker would report the whole catalogue as challenged.

Known limitation

Selenium cannot work here from a datacentre. chromedriver cannot
authenticate a proxy, and the proxy is the whole point on this site. It
strips the credentials, warns, and reports exit 3 with the reason rather
than a misleading "0 products". Use Playwright or pyppeteer, or run Selenium
from a residential connection.

Full detail, every figure dated and reproducible, in the
README.