Repository navigation
Behaviour change:
--retries 0is now a usage error (exit 2) in every
entry point. It used to be accepted and made the load loop body
unreachable — nothing was fetched, and the engine then reported the
browser's own blank page as a block: exit 3, with a 39-byte "capture", in
about 40 milliseconds. If a script of yours passes 0, pass 1: the flag
counts ATTEMPTS, so 1 means "try once and do not retry".Also,
--retriesonscraper_api_client.pyused to mean extra attempts
while meaning attempts everywhere else. It now means attempts there too,
and its default moves 1 → 2 so the number of requests actually made is
unchanged.
Worked through a third-party audit dated 2026-10-13.
http_scraper.py — no browser at all
Sleep Number server-renders its whole catalogue into the page, so there is
nothing to wait for and nothing to scroll. The README has said so since
v0.1.0; the recommended path launched Chromium anyway. Measured 2026-10-13
through one US residential exit, three runs each:
| run 1 | run 2 | run 3 | median | |
|---|---|---|---|---|
http_scraper.py |
1.4s | 1.3s | 2.1s | 1.4s |
playwright_scraper.py |
4.2s | 2.9s | 64.2s | 4.2s |
52 rows each, identical on every field. It keeps the whole output
contract and deliberately takes no --headless, --cdp-endpoint,
--fingerprint or captcha flags — there is no browser to configure, and no
challenge has ever been observed on this site.
It still needs an exit the site serves. Removing the browser does not
change the address, and the address is the whole difficulty here.
A run is now checked against the site's own count
The payload states total_results, and on 2026-10-13 it equalled the
products in the payload exactly across four categories (7, 10, 20, 9). A run
that parses fewer distinct products than the site counted now reports
partial (exit 6) with stop_reason: parser_incomplete and
products_missing in the sidecar, instead of exit 0 and a quietly short
file.
That replaces a coverage threshold: a threshold is a guess about how much
loss is acceptable, and total_results is the response proving the loss.
Fixed
--retries 0fetched nothing and called it a block. The audit ran
exactly that and reported the site unreachable; its 39-byte evidence was
about:blank. The site does refuse datacentre addresses —--retries 1
gets the real 923-byte CloudFront 403 — so the conclusion was right and
the evidence was about our bug.- The canary asserted a sale that has since ended. It required the three
canary categories to disagree about discounting by 20 points, from figures
measured during a September sale. On 13 October they read 0, 11 of 194 and
0, with the parser working perfectly — the check would have been red every
day for a catalogue that simply is not discounted. Removed; the discount
column is still guarded by assertions that hold either way. - A test fixture claimed what no response says, inheriting
total_resultsfrom the full capture while its products were trimmed. - The canary now runs both transports daily and asserts they agree field for
field.
Documentation
The README promised a sidecar "per run" and meant "per run that writes
rows" — a blocked run deliberately writes neither file, and now says so.
Figures re-measured rather than inherited: 91 products across 52
categories today against 88 and 51 in September.
69 offline checks, passing in each of three engine virtualenvs; ten new
controls, all valid.