No change to what the scraper does. What changes is how it is installed and
built: CI, the canary and the Docker image now install exact, hash-checked
versions, and every GitHub Action is pinned by commit. If you install with
pip install -r requirements.txt, nothing changes for you; for the versions
CI tested, install the matching requirements*.lock (see the README).
pyppeteer users: that engine's environment carries urllib3 1.26.20,
which has five advisories fixed only in urllib3 2.x — pyppeteer forbids
it. One lets a cross-origin redirect forwardProxy-Authorization. Prefer
Playwright when running with proxy credentials.
Security
- Every GitHub Action is pinned to a commit SHA (
checkoutv4.4.0,
setup-pythonv5.6.0,upload-artifactv4.6.2), with the release as a
comment. A tag can be moved to another commit — the March 2025
tj-actions/changed-filescompromise did exactly that — and the canary
runs with theLG_PROXYsecret in its environment. Dependabot now proposes
updates to the pins. - Hashed lock files for the core and each engine (
requirements*.lock,
plus.github/requirements-ci.lockfor pytest and pip-audit), resolved for
Python 3.9+. CI, the canary and the Docker image install only these, with
--require-hashes; the unpinnedpip install --upgrade pipand
pip install pyteststeps are gone. audit.yml: pip-audit over every lock, on each change and weekly. It
found five urllib3 1.26.20 advisories in the pyppeteer lock, fixed only in
urllib3 2.x, which pyppeteer forbids; they are ignored by ID and documented
in the README, so any new advisory still fails.smoke_test.pyasserts all of the above, so a later edit cannot quietly
unpin an action or add an unlocked install.
Not changed, and why
- Docker image digest and a non-root user — deferred until the image runs
as a service; a non-root user breaks the documented
-v "$PWD/out:/out"mount on Linux. - The
.txtfiles keep their>=floors — they are the specification a
person edits and whatpyproject.tomlmirrors; the locks are generated
from them.