Repository navigation
Behaviour changes.
- A listing where SHEIN states it holds products but none could be parsed
now exits 5 withstop_reason: parser_found_nothing. It used to
exit 4 ("empty"), which claimed the catalogue was empty when the page
said otherwise.selenium_scraper.pyandpuppeteer_scraper.pynow reject negative
values for--retries,--retry-delay,--scroll-delay,
--delay-jitter,--rate-limit-cooldown,--block-retries,
--risk-challenge-rounds,--max-solvesand--proxy-block-retries,
and a--scraper-api-timeoutbelow 1, with exit 2, as Playwright
already did.--retries -1used to navigate zero times and exit 4.
--proxy-block-retries -1is now refused in all three engines.- A
SHEIN_URLin.envor the environment no longer overrides a typed
--queryor--category(it is ignored with a warning). It still
fills--urlwhen nothing was typed.- The daily canary now goes red when SHEIN serves no rows. Its live job
needs theSHEIN_CDP_ENDPOINTrepository secret and fails without it.
Fixed
- Output files are written atomically: to a temporary file in the
same directory, flushed and synced, then renamed over the target. A run
that fails part-way through a write leaves the previous output (and its
.meta.json) intact instead of a truncated file. New files get the
usual umask-derived mode; an existing file keeps its mode. - CSV cells that a spreadsheet would run as a formula (starting with
=,+,-,@, tab, CR or LF) get a leading apostrophe in CSV
output only. Numbers are left alone and JSON keeps the site's text. The
number of escaped cells is recorded ascsv_cells_escapedin the
.meta.json. - The canary tested nothing for two weeks. From 2026-10-01 to
2026-10-15 every scheduled run was refused by SHEIN's
/risk/action/limitgateway (exit 3, zero rows) and reported green. The
GitHub-runner job is now labelled as expecting that refusal and fails on
anything else (empty, never obtained, too few rows); a new daily
canary-livejob runs through a Scraping Browser profile and must
return at least 10 rows, 90% of them priced and titled. - A rate-limited run written with
--allow-emptynow records
stop_reason: rate_limitedrather thanblocked. diff_runs.pycompares a JSON run and a CSV run of the same rows as
equal; before, every CSV price read as text and showed as "changed".--fp-tagshelp in the Playwright engine now gives a single OS tag as
its example, which is what the Fingerprint API accepts..gitignorecovers capture and dump directories by kind (captures/,
dumps/,out/,runs/,tmp/, ...) and HTML saved at the repo root.tests.ymlruns the credential scan through.github/ci_checks.py,
the same code the offline suite runs, and the scan refuses to report
clean after reading zero files.
Changed
- The README no longer says Scraping Browser credentials last about a
day; that figure was never measured for this repo. An HTTP 401 is the
signal to fetch a fresh endpoint. - Source comments, workflow comments and this changelog no longer refer
to the maintainers' private notes; the rules are stated in place. A new
offline check keeps it that way.