v0.2.0

Behaviour changes.

• A listing where SHEIN states it holds products but none could be parsed
  now exits 5 with stop_reason: parser_found_nothing. It used to
  exit 4 ("empty"), which claimed the catalogue was empty when the page
  said otherwise.
• selenium_scraper.py and puppeteer_scraper.py now reject negative
  values for --retries, --retry-delay, --scroll-delay,
  --delay-jitter, --rate-limit-cooldown, --block-retries,
  --risk-challenge-rounds, --max-solves and --proxy-block-retries,
  and a --scraper-api-timeout below 1, with exit 2, as Playwright
  already did. --retries -1 used to navigate zero times and exit 4.
  --proxy-block-retries -1 is now refused in all three engines.
• A SHEIN_URL in .env or the environment no longer overrides a typed
  --query or --category (it is ignored with a warning). It still
  fills --url when nothing was typed.
• The daily canary now goes red when SHEIN serves no rows. Its live job
  needs the SHEIN_CDP_ENDPOINT repository secret and fails without it.

Fixed

• Output files are written atomically: to a temporary file in the
  same directory, flushed and synced, then renamed over the target. A run
  that fails part-way through a write leaves the previous output (and its
  .meta.json) intact instead of a truncated file. New files get the
  usual umask-derived mode; an existing file keeps its mode.
• CSV cells that a spreadsheet would run as a formula (starting with
  =, +, -, @, tab, CR or LF) get a leading apostrophe in CSV
  output only. Numbers are left alone and JSON keeps the site's text. The
  number of escaped cells is recorded as csv_cells_escaped in the
  .meta.json.
• The canary tested nothing for two weeks. From 2026-10-01 to
  2026-10-15 every scheduled run was refused by SHEIN's
  /risk/action/limit gateway (exit 3, zero rows) and reported green. The
  GitHub-runner job is now labelled as expecting that refusal and fails on
  anything else (empty, never obtained, too few rows); a new daily
  canary-live job runs through a Scraping Browser profile and must
  return at least 10 rows, 90% of them priced and titled.
• A rate-limited run written with --allow-empty now records
  stop_reason: rate_limited rather than blocked.
• diff_runs.py compares a JSON run and a CSV run of the same rows as
  equal; before, every CSV price read as text and showed as "changed".
• --fp-tags help in the Playwright engine now gives a single OS tag as
  its example, which is what the Fingerprint API accepts.
• .gitignore covers capture and dump directories by kind (captures/,
  dumps/, out/, runs/, tmp/, ...) and HTML saved at the repo root.
• tests.yml runs the credential scan through .github/ci_checks.py,
  the same code the offline suite runs, and the scan refuses to report
  clean after reading zero files.

Changed

• The README no longer says Scraping Browser credentials last about a
  day; that figure was never measured for this repo. An HTTP 401 is the
  signal to fetch a fresh endpoint.
• Source comments, workflow comments and this changelog no longer refer
  to the maintainers' private notes; the rules are stated in place. A new
  offline check keeps it that way.

v0.1.0

Read before relying on it:

• A run that has rows but did not finish cleanly exits 6 (partial), with the cause in the
  sidecar's stop_reason (blocked, rate_limited, remote_api_error, failed_pages,
  rejected_rows). Exits 3/4/5 mean no rows and no file.
• SHEIN's own /risk/challenge is solved automatically, but not every time: on
  2026-09-30, 3 of 5 fresh US Scraping Browser profiles passed (each on round 3). The other two
  got code=9001 on every answer, including correct ones: a risk-score rejection nothing on the
  page can fix. The run then exits 3; switch profiles.
• Paid 2Captcha solves are capped per run by --max-solves (default 8, 0 = never pay).

SHEIN (us.shein.com) search, category and product-page scraper. It has three engines
(Playwright, Selenium, pyppeteer) sharing one fetch loop, outputs JSON/CSV, and uses the 2scraper
family's output contract.

What it does

• Reads the page's own window.gbRawData (search/category) or JSON-LD (product pages), merged
  across scroll rounds. A malformed record is skipped and counted, never the whole batch.
• Passes SHEIN's /risk/challenge: clicks the "I am human" checkbox for free, then solves the 3x3
  image grid (2Captcha GridTask) or the icon-sequence widget (CoordinatesTask).
• --cdp-endpoint (Scraping Browser API): one connection per run, retried through a locked
  profile, and a clear message when an endpoint has expired (HTTP 401).
• diff_runs.py refuses to compare different selections (URL, --sort), reports currency changes
  separately, and reads a capped top-N's missing SKUs as left_selection, not delisted.

Measured live, 2026-09-30 (US Scraping Browser API profiles)

• Playwright: challenge passed, then 10 of 10 products with prices, status=complete.
• Puppeteer: reconnected after an HTTP 500 (profile still locked), status=complete, 10 products.
• Selenium: cannot use the Scraping Browser API (chromedriver takes no credentials). From a local
  Chrome, SHEIN rejected correct challenge answers or rate-limited it (exit 3).
• After about ten runs in a row, a profile hit /risk/action/limit, and the run exited 3 with the
  rate-limit message. --rate-limit-cooldown retries once.

Checks: 97 offline checks, including an end-to-end run of the shared loop on a real gbRawData
capture. CI: offline suite on Python 3.9 and 3.12, the built wheel imported outside the checkout,
the Docker image (Chromium launches inside), and each engine in its own venv.

Full history: CHANGELOG.md.