Repository navigation
v0.1.0
Read before relying on it:
- A run that has rows but did not finish cleanly exits 6 (partial), with the cause in the
sidecar'sstop_reason(blocked,rate_limited,remote_api_error,failed_pages,
rejected_rows). Exits 3/4/5 mean no rows and no file.- SHEIN's own
/risk/challengeis solved automatically, but not every time: on
2026-09-30, 3 of 5 fresh US Scraping Browser profiles passed (each on round 3). The other two
gotcode=9001on every answer, including correct ones: a risk-score rejection nothing on the
page can fix. The run then exits 3; switch profiles.- Paid 2Captcha solves are capped per run by
--max-solves(default 8,0= never pay).
SHEIN (us.shein.com) search, category and product-page scraper. It has three engines
(Playwright, Selenium, pyppeteer) sharing one fetch loop, outputs JSON/CSV, and uses the 2scraper
family's output contract.
What it does
- Reads the page's own
window.gbRawData(search/category) or JSON-LD (product pages), merged
across scroll rounds. A malformed record is skipped and counted, never the whole batch. - Passes SHEIN's
/risk/challenge: clicks the "I am human" checkbox for free, then solves the 3x3
image grid (2CaptchaGridTask) or the icon-sequence widget (CoordinatesTask). --cdp-endpoint(Scraping Browser API): one connection per run, retried through a locked
profile, and a clear message when an endpoint has expired (HTTP 401).diff_runs.pyrefuses to compare different selections (URL,--sort), reports currency changes
separately, and reads a capped top-N's missing SKUs asleft_selection, not delisted.
Measured live, 2026-09-30 (US Scraping Browser API profiles)
- Playwright: challenge passed, then 10 of 10 products with prices,
status=complete. - Puppeteer: reconnected after an HTTP 500 (profile still locked),
status=complete, 10 products. - Selenium: cannot use the Scraping Browser API (chromedriver takes no credentials). From a local
Chrome, SHEIN rejected correct challenge answers or rate-limited it (exit 3). - After about ten runs in a row, a profile hit
/risk/action/limit, and the run exited 3 with the
rate-limit message.--rate-limit-cooldownretries once.
Checks: 97 offline checks, including an end-to-end run of the shared loop on a real gbRawData
capture. CI: offline suite on Python 3.9 and 3.12, the built wheel imported outside the checkout,
the Docker image (Chromium launches inside), and each engine in its own venv.
Full history: CHANGELOG.md.