Skip to content

0.2.0

Choose a tag to compare

@suvorovmika25-gif suvorovmika25-gif released this 28 Sep 14:03
· 7 commits to main since this release
69d4e14

A plain-HTTP engine, a pinned supply chain, and follow-ups to 0.1.1.
Playwright stays the default engine. Nothing is removed or renamed.

Behaviour changes for an existing user:

  • --fingerprint against a Fingerprint API that refuses the key now
    exits 5 (remote API error) with the reason. It used to crash with
    a traceback and exit 1.
  • The Scraper API client's .meta.json now reports the last page it
    fetched as final_url, not the start URL.
  • CI, the canary and the Docker image install from hashed lock files. A
    plain pip install -r requirements.txt is unchanged.

Added

  • http_scraper.py, a plain-HTTP engine. It needs no browser and no key.
    The table is in the served HTML, so a GET is enough.
    • Measured on 2026-09-28, page 1: 1.2 s and 47 MB RSS, against 2.9 s and
      150 MB for Playwright (plus Chromium's own processes). The 25 rows are
      identical to Playwright's.
    • How it fetches:
      • one session, with separate connect and read timeouts;
      • bounded retries of a connection error, a 429 or a 5xx, with a
        jittered exponential backoff and Retry-After honoured (capped);
      • a refusal is never retried from the same address;
      • an honest User-Agent;
      • --proxy / --proxy-file;
      • sequential only.
    • It is optional: Playwright stays the default.
  • listing_run.py: the page loop the two browserless engines share. The
    Scraper API client now runs on it too, so the HTTP engine is not a fourth
    copy of that loop. A transport is fetch(args, url) -> (html, status, final_url) and does its own transient-fault retries. The Scraper API
    client now reports the last page it fetched as final_url, not the start
    URL.

Changed

  • GitHub Actions moved to checkout v7.0.1, setup-python v7.0.0 and
    upload-artifact v7.0.1 (Dependabot, #5). The canary was run on them and
    uploaded its artefact.
  • Every download CI and the image make is pinned.
    • Actions are pinned by commit SHA, with the release in a comment;
      Dependabot proposes updates.
    • CI, the canary and the Docker image install only the hashed
      requirements*.lock files, with --require-hashes. The .txt files stay
      as the loose spec.
    • audit.yml runs pip-audit over every lock on each change and weekly.
      The pyppeteer lock's five urllib3 1.x advisories (pyppeteer needs
      urllib3<2) are ignored by ID and explained in the README.
    • Workflows run with a read-only token.
    • Ported from lg-scraper 0.2.1; the locks resolve to the same pins.

Fixed

  • A refused Fingerprint API crashed the run with a traceback and exit 1.
    It now exits 5, the remote-API code, with the reason and no traceback.
    The case is real: on 2026-09-28 a key with a working captcha-solving
    balance got 403 from /fingerprint/random, because fingerprints are a
    separate subscription.
  • The real-page captcha check added in 0.1.1 reported a script error as a
    skip ("no browser"). An error from the script itself is now a failure.