v0.2.0 — a sample is not a full listing
Behaviour change for anyone reading the sidecar. A run that fetched
every page--pagesallowed but stopped before the category ended is now
status: "limited"(still exit 0), not"complete", anddiff_runs.py
refuses it. A scheduled snapshot that relied on--pages 3of a longer
category must ask for more pages than the category has —--pages 10
costs nothing extra, the engine stops at the API's own last page.
Every item below was reproduced before it was fixed, by a third-party audit
on 2026-09-25 and again offline, and each has a regression check that fails
against v0.1.0.
Fixed
- A sample was reported as the whole catalogue. The engines start from
stop_reason = "completed"and kept it when the page loop simply ran out
of--pages;finish_runcounted that as complete. Live,--pages 2of
the 48-product RU television category wrotestatus: completewith 24
products andtotal_results: 48, anddiff_runs.pythen reported 24
removed.finish_runnow calls such a run complete only on the site's
own evidence — every page of itspageCountfetched, or the products
reaching itstotalCount— andlimitedotherwise. Fixed once, in the
sharedfinish_run, so all five engines get it. diff_runs.pytrusted what it could not see. A missing sidecar was
treated as "nothing to check", and two complete runs of different
categories were diffed row by row. Both are now refused without--force.- The lg.com check accepted any host ending in "lg.com" —
notlg.com,
evillg.com— plushttp://,file://, userinfo and non-standard ports.
Now HTTPS only, the hostlg.comor a subdomain of it, default port, no
userinfo; a URL with no scheme says so instead of calling the host a
locale. - The form's action was POSTed to wherever it pointed. It comes from
served HTML; an absolute action (http://169.254.169.254/…) was followed
as-is. It must now be same-origin and under the page's own
/{locale}/mkt/ajax/, every redirect hop of the category GET is checked
against the same rule, and the API POST no longer follows redirects. An
off-origindata-price-sync-urlis dropped. - Output writes were not atomic. A missing
--outdirectory raised
FileNotFoundErrorafter every page had been fetched, and a run killed
mid-write left a torn file where the last good one had been. Files are now
written to a temporary file,fsynced and renamed; the directory is
created; the old sidecar is removed first and the new one written last. - Numeric flags were not validated.
--pages 0finished as an empty
run, a negative--delaycrashed insleep()after page 1,--retries 0
never made a request. All five CLIs now range-check these before any
network call (exit 2). python3 env_config.pyprinted credentials without an@— a CDP
endpoint with?token=or a proxy with a key in its query string. Every
key butLG_URLis now shown as a length only.- The canary checked a sample. It ran
--pages 3of a 4-5 page
category and requiredcomplete; it now reads the whole category and
assertslisting_complete,pages_completed == page_countand at least
95% oftotal_resultswritten. Its--dump-htmlno longer shares a name
with the output file.
Added
- Sidecar fields
schema_version(2),listing_complete,scope
(full_listing/limited_pages),page_countandcompleteness_ratio.
Not changed, and why
- The flat module layout and the single-file
smoke_test.py— the
audit'ssrc/package, shared orchestrator and pytest-module migration
are a family-wide decision (the template keeps them this way on purpose),
not a fix; the status/exit mapping the engines must agree on is already
one sharedfinish_run. - pyppeteer stays, as the original brief requires.
- Supply-chain hardening (Actions pinned by SHA, image digest, non-root
container, lock files,pip-audit) is worth doing and is a separate batch:
a non-root user breaks the documented-v "$PWD/out:/out"mount on Linux
and needs its own answer.