Repository navigation
0.2.0
A plain-HTTP engine, a pinned supply chain, and follow-ups to 0.1.1.
Playwright stays the default engine. Nothing is removed or renamed.
Behaviour changes for an existing user:
--fingerprintagainst a Fingerprint API that refuses the key now
exits 5 (remote API error) with the reason. It used to crash with
a traceback and exit 1.- The Scraper API client's
.meta.jsonnow reports the last page it
fetched asfinal_url, not the start URL.- CI, the canary and the Docker image install from hashed lock files. A
plainpip install -r requirements.txtis unchanged.
Added
http_scraper.py, a plain-HTTP engine. It needs no browser and no key.
The table is in the served HTML, so a GET is enough.- Measured on 2026-09-28, page 1: 1.2 s and 47 MB RSS, against 2.9 s and
150 MB for Playwright (plus Chromium's own processes). The 25 rows are
identical to Playwright's. - How it fetches:
- one session, with separate connect and read timeouts;
- bounded retries of a connection error, a 429 or a 5xx, with a
jittered exponential backoff andRetry-Afterhonoured (capped); - a refusal is never retried from the same address;
- an honest User-Agent;
--proxy/--proxy-file;- sequential only.
- It is optional: Playwright stays the default.
- Measured on 2026-09-28, page 1: 1.2 s and 47 MB RSS, against 2.9 s and
listing_run.py: the page loop the two browserless engines share. The
Scraper API client now runs on it too, so the HTTP engine is not a fourth
copy of that loop. A transport isfetch(args, url) -> (html, status, final_url)and does its own transient-fault retries. The Scraper API
client now reports the last page it fetched asfinal_url, not the start
URL.
Changed
- GitHub Actions moved to checkout v7.0.1, setup-python v7.0.0 and
upload-artifact v7.0.1 (Dependabot, #5). The canary was run on them and
uploaded its artefact. - Every download CI and the image make is pinned.
- Actions are pinned by commit SHA, with the release in a comment;
Dependabot proposes updates. - CI, the canary and the Docker image install only the hashed
requirements*.lockfiles, with--require-hashes. The.txtfiles stay
as the loose spec. audit.ymlrunspip-auditover every lock on each change and weekly.
The pyppeteer lock's five urllib3 1.x advisories (pyppeteer needs
urllib3<2) are ignored by ID and explained in the README.- Workflows run with a read-only token.
- Ported from lg-scraper 0.2.1; the locks resolve to the same pins.
- Actions are pinned by commit SHA, with the release in a comment;
Fixed
- A refused Fingerprint API crashed the run with a traceback and exit 1.
It now exits 5, the remote-API code, with the reason and no traceback.
The case is real: on 2026-09-28 a key with a working captcha-solving
balance got 403 from/fingerprint/random, because fingerprints are a
separate subscription. - The real-page captcha check added in 0.1.1 reported a script error as a
skip ("no browser"). An error from the script itself is now a failure.