v0.2.0 — a third-party audit, worked through

A third-party audit, worked through end to end. Its headline conclusion was wrong and three of its recommendations were right — and the pass that followed found four more defects it had not reached, two of them in this repo's own guards.

--retries 0 is a usage error now. It used to attempt no request at all and then judge the blank page. A script passing it received exit 4 and no data; it now receives exit 2 with the reason.

The audit's main finding was its own flag

It ran --retries 0, got a 39-byte about:blank and exit 4, and concluded in writing that the browser path was broken and should be replaced. The load loop is range(1, retries + 1) — zero ran it never, so goto was not called and the engine judged a page it had not fetched. The same command with --retries 1, same address and same minute, returned 199,558 bytes and 24 products.

But the recommendation underneath it was right

http_scraper.py — a fourth engine over requests. Same parser, same rows, same sidecar, same exit codes, and the whole install is pip install -r requirements.txt.

	3 pages, same URL, no delay
http_scraper.py	4.8–6.0 s
playwright_scraper.py	9.2–10.6 s

The site serves a project User-Agent, a requests UA, a Chrome UA and no User-Agent header at all byte-identical pages, so a browser here buys nothing. Flags it cannot honour (--cdp-endpoint, --solve-captcha, --fingerprint) are refused rather than accepted and ignored.

The new engine immediately earned its place

It disagreed with the browser engines on 10 of 72 rows, every one a title. The site ships data-module="TruncateText" data-options='{"length": 36}' on each tile and its own JavaScript rewrites the name after load — so for as long as this repo had only browser engines it returned truncated product titles, and nothing could see it because there was nothing to disagree with. The full name is recovered from the thumbnail's alt, present on 171 of 171 measured tiles.

The canary now runs both engines over the same three pages daily and fails if any field differs.

Four more the audit did not reach

• Selenium and pyppeteer CRASHED on a blocked page with no --proxy — AttributeError, exit 1, for a page that was merely blocked. Playwright had the guard all along.
• A bare recaptcha marker fired on a GOOD page fetched over --cdp-endpoint. 2Captcha's auto-solve extension injects sixteen hunter scripts into every page it loads. A listing with no tiles would have been reported as a challenge — exit 3 for a correct answer. A CDP-fetched page is now one of the suite's fixtures, because every capture this repo had until today was fetched locally.
• Every paid solve now goes through one budget — measured with the solver stubbed: 3 invocations against a limit of 1, now 1.
• KOSE_URL in .env made --tags and --category unusable in all four CLIs: the conflict check read an environment variable as a flag the user had typed.

Plus atomic output writes (umask honoured, an existing file's mode kept), CSV formula neutralisation, and <out>.last_attempt.json so a failed run leaves something machine-readable beside a sidecar that still describes good data — the half of the audit that needed no argument.

Declined, with reasons

• Dropping Selenium/pyppeteer. The three engines are what made two of the bugs above visible at all.
• Making playwright_scraper.py quietly not use Playwright. It is a separate CLI instead; the README recommends it and the canary runs both.

576 offline checks, every new one controlled — fix reverted, the named check confirmed red, fix restored. Two of those controls caught checks of mine that were passing for the wrong reason. All four engines verified live, 48 of 48 identical rows; --cdp-endpoint verified too, which it was not a month ago.