Skip to content

v0.2.0

Latest

Choose a tag to compare

@jehrr jehrr released this 16 Oct 06:22
7dd9cb4

Behaviour changes.

  • A listing where SHEIN states it holds products but none could be parsed
    now exits 5 with stop_reason: parser_found_nothing. It used to
    exit 4 ("empty"), which claimed the catalogue was empty when the page
    said otherwise.
  • selenium_scraper.py and puppeteer_scraper.py now reject negative
    values for --retries, --retry-delay, --scroll-delay,
    --delay-jitter, --rate-limit-cooldown, --block-retries,
    --risk-challenge-rounds, --max-solves and --proxy-block-retries,
    and a --scraper-api-timeout below 1, with exit 2, as Playwright
    already did. --retries -1 used to navigate zero times and exit 4.
    --proxy-block-retries -1 is now refused in all three engines.
  • A SHEIN_URL in .env or the environment no longer overrides a typed
    --query or --category (it is ignored with a warning). It still
    fills --url when nothing was typed.
  • The daily canary now goes red when SHEIN serves no rows. Its live job
    needs the SHEIN_CDP_ENDPOINT repository secret and fails without it.

Fixed

  • Output files are written atomically: to a temporary file in the
    same directory, flushed and synced, then renamed over the target. A run
    that fails part-way through a write leaves the previous output (and its
    .meta.json) intact instead of a truncated file. New files get the
    usual umask-derived mode; an existing file keeps its mode.
  • CSV cells that a spreadsheet would run as a formula (starting with
    =, +, -, @, tab, CR or LF) get a leading apostrophe in CSV
    output only. Numbers are left alone and JSON keeps the site's text. The
    number of escaped cells is recorded as csv_cells_escaped in the
    .meta.json.
  • The canary tested nothing for two weeks. From 2026-10-01 to
    2026-10-15 every scheduled run was refused by SHEIN's
    /risk/action/limit gateway (exit 3, zero rows) and reported green. The
    GitHub-runner job is now labelled as expecting that refusal and fails on
    anything else (empty, never obtained, too few rows); a new daily
    canary-live job runs through a Scraping Browser profile and must
    return at least 10 rows, 90% of them priced and titled.
  • A rate-limited run written with --allow-empty now records
    stop_reason: rate_limited rather than blocked.
  • diff_runs.py compares a JSON run and a CSV run of the same rows as
    equal; before, every CSV price read as text and showed as "changed".
  • --fp-tags help in the Playwright engine now gives a single OS tag as
    its example, which is what the Fingerprint API accepts.
  • .gitignore covers capture and dump directories by kind (captures/,
    dumps/, out/, runs/, tmp/, ...) and HTML saved at the repo root.
  • tests.yml runs the credential scan through .github/ci_checks.py,
    the same code the offline suite runs, and the scan refuses to report
    clean after reading zero files.

Changed

  • The README no longer says Scraping Browser credentials last about a
    day; that figure was never measured for this repo. An HTTP 401 is the
    signal to fetch a fresh endpoint.
  • Source comments, workflow comments and this changelog no longer refer
    to the maintainers' private notes; the rules are stated in place. A new
    offline check keeps it that way.