Skip to content

v0.1.0

Choose a tag to compare

@chakh1ik chakh1ik released this 30 Sep 13:34
· 7 commits to main since this release

Read before relying on it:

  • A run that has rows but did not finish cleanly exits 6 (partial), with the cause in the
    sidecar's stop_reason (blocked, rate_limited, remote_api_error, failed_pages,
    rejected_rows). Exits 3/4/5 mean no rows and no file.
  • SHEIN's own /risk/challenge is solved automatically, but not every time: on
    2026-09-30, 3 of 5 fresh US Scraping Browser profiles passed (each on round 3). The other two
    got code=9001 on every answer, including correct ones: a risk-score rejection nothing on the
    page can fix. The run then exits 3; switch profiles.
  • Paid 2Captcha solves are capped per run by --max-solves (default 8, 0 = never pay).

SHEIN (us.shein.com) search, category and product-page scraper. It has three engines
(Playwright, Selenium, pyppeteer) sharing one fetch loop, outputs JSON/CSV, and uses the 2scraper
family's output contract.

What it does

  • Reads the page's own window.gbRawData (search/category) or JSON-LD (product pages), merged
    across scroll rounds. A malformed record is skipped and counted, never the whole batch.
  • Passes SHEIN's /risk/challenge: clicks the "I am human" checkbox for free, then solves the 3x3
    image grid (2Captcha GridTask) or the icon-sequence widget (CoordinatesTask).
  • --cdp-endpoint (Scraping Browser API): one connection per run, retried through a locked
    profile, and a clear message when an endpoint has expired (HTTP 401).
  • diff_runs.py refuses to compare different selections (URL, --sort), reports currency changes
    separately, and reads a capped top-N's missing SKUs as left_selection, not delisted.

Measured live, 2026-09-30 (US Scraping Browser API profiles)

  • Playwright: challenge passed, then 10 of 10 products with prices, status=complete.
  • Puppeteer: reconnected after an HTTP 500 (profile still locked), status=complete, 10 products.
  • Selenium: cannot use the Scraping Browser API (chromedriver takes no credentials). From a local
    Chrome, SHEIN rejected correct challenge answers or rate-limited it (exit 3).
  • After about ten runs in a row, a profile hit /risk/action/limit, and the run exited 3 with the
    rate-limit message. --rate-limit-cooldown retries once.

Checks: 97 offline checks, including an end-to-end run of the shared loop on a real gbRawData
capture. CI: offline suite on Python 3.9 and 3.12, the built wheel imported outside the checkout,
the Docker image (Chromium launches inside), and each engine in its own venv.

Full history: CHANGELOG.md.