v0.3.0

Newly refused flag values. --pages 0, --concurrency 0 and any
negative --pages, --concurrency, --delay, --retry-delay or
--proxy-block-retries are now usage errors (exit 2) naming the flag, in
every engine and in scraper_api_client.py. They used to be accepted: a
zero page count fetched nothing and reported an empty catalogue, and a
negative pause crashed the run after page 1.

pyppeteer, a page that never loads, no --proxy: the run now ends as a
load failure (exit 5, or 6 with rows) instead of a traceback and exit 1.
With a pool, a repeated timeout no longer rotates the exit; only an
unusable exit does, as in the other engines.

Fixed

• pyppeteer crashed with AttributeError when a page failed to load and
  no proxy was configured. The load-failure branch called
  pool.advance(...) on a None pool. It now rotates only for an unusable
  exit, and only when the pool has another exit to rotate to.
• The per-page solve allowance was spent by a detection that bought
  nothing. The pre-classification captcha check used up the page's one
  solve even when it found no challenge, so a challenge first seen after
  classification was never offered to the solver and the page was reported
  as blocked. The allowance is now claimed immediately before the paid
  request, in all three browser engines. A purchase that fails still counts,
  so the cap still holds at one per page across retries and rotations.
• The credential scan read nothing from a clone under a directory named
  build or dist. The skip list was tested against the absolute path. It
  now tests the path relative to the repository, a scan that reads zero
  files fails, and .env* backups, .json and .csv files are read too.
• .gitignore now covers every --dump-html NAME --pages N output
  (NAME.pageN), a root-level *.html dump and the usual working
  directories (live/, out/, tmp/, ...).
• .env.example ships KOSE_URL commented out, so a copied .env does not
  decide what a run fetches.
• The canary's Scraping Browser job no longer passes by skipping: with no
  KOSE_CDP_ENDPOINT secret it fails with an error naming the secret, and an
  exit 5 over CDP is a failure rather than a warning.
• Comments that named columns this schema does not have, stated a category
  total as current, carried another vendor's wording or gave the wrong exit
  code for parser_found_nothing (it is 5) were corrected; the CI sample
  summary counts priced rows instead of a column that does not exist.
• Shipped files, including the README, --help output and the 0.1.1 notes
  below, no longer cite private notes by name or section number; a suite
  check keeps it that way.
• smoke_test.py -v no longer crashes on a check whose detail is not a
  string.

v0.2.0 — a third-party audit, worked through

A third-party audit, worked through end to end. Its headline conclusion was wrong and three of its recommendations were right — and the pass that followed found four more defects it had not reached, two of them in this repo's own guards.

--retries 0 is a usage error now. It used to attempt no request at all and then judge the blank page. A script passing it received exit 4 and no data; it now receives exit 2 with the reason.

The audit's main finding was its own flag

It ran --retries 0, got a 39-byte about:blank and exit 4, and concluded in writing that the browser path was broken and should be replaced. The load loop is range(1, retries + 1) — zero ran it never, so goto was not called and the engine judged a page it had not fetched. The same command with --retries 1, same address and same minute, returned 199,558 bytes and 24 products.

But the recommendation underneath it was right

http_scraper.py — a fourth engine over requests. Same parser, same rows, same sidecar, same exit codes, and the whole install is pip install -r requirements.txt.

	3 pages, same URL, no delay
http_scraper.py	4.8–6.0 s
playwright_scraper.py	9.2–10.6 s

The site serves a project User-Agent, a requests UA, a Chrome UA and no User-Agent header at all byte-identical pages, so a browser here buys nothing. Flags it cannot honour (--cdp-endpoint, --solve-captcha, --fingerprint) are refused rather than accepted and ignored.

The new engine immediately earned its place

It disagreed with the browser engines on 10 of 72 rows, every one a title. The site ships data-module="TruncateText" data-options='{"length": 36}' on each tile and its own JavaScript rewrites the name after load — so for as long as this repo had only browser engines it returned truncated product titles, and nothing could see it because there was nothing to disagree with. The full name is recovered from the thumbnail's alt, present on 171 of 171 measured tiles.

The canary now runs both engines over the same three pages daily and fails if any field differs.

Four more the audit did not reach

• Selenium and pyppeteer CRASHED on a blocked page with no --proxy — AttributeError, exit 1, for a page that was merely blocked. Playwright had the guard all along.
• A bare recaptcha marker fired on a GOOD page fetched over --cdp-endpoint. 2Captcha's auto-solve extension injects sixteen hunter scripts into every page it loads. A listing with no tiles would have been reported as a challenge — exit 3 for a correct answer. A CDP-fetched page is now one of the suite's fixtures, because every capture this repo had until today was fetched locally.
• Every paid solve now goes through one budget — measured with the solver stubbed: 3 invocations against a limit of 1, now 1.
• KOSE_URL in .env made --tags and --category unusable in all four CLIs: the conflict check read an environment variable as a flag the user had typed.

Plus atomic output writes (umask honoured, an existing file's mode kept), CSV formula neutralisation, and <out>.last_attempt.json so a failed run leaves something machine-readable beside a sidecar that still describes good data — the half of the audit that needed no argument.

Declined, with reasons

• Dropping Selenium/pyppeteer. The three engines are what made two of the bugs above visible at all.
• Making playwright_scraper.py quietly not use Playwright. It is a separate CLI instead; the README recommends it and the canary runs both.

576 offline checks, every new one controlled — fix reverted, the named check confirmed red, fix restored. Two of those controls caught checks of mine that were passing for the wrong reason. All four engines verified live, 48 of 48 identical rows; --cdp-endpoint verified too, which it was not a month ago.

v0.1.1 — audit pass

A pass back over the project's own rules before calling this finished. Everything below was found by checking a claim, not by reading the code — which is the point of the exercise.

Fixed

An HTTP 404 was retried instead of believed. All three engines discarded what goto() returned, so every page reached the classifier with status=None. This site answers a missing path with a bare 1,040-byte page carrying no site chrome at all, which without a status classifies as "unknown" — and unknown retries, spending the budget on an address that will never exist.

Playwright and pyppeteer now thread the real status. Selenium cannot — driver.get() returns None and WebDriver exposes no HTTP status — so the parser also reads the status the site states in its own <title>404- …. That body reader is what makes three engines agree instead of two being better informed than the third.

The new not_found state is terminal: no retry, and not counted as blocked, because exit 3 would send the reader after a proxy problem when the URL is simply wrong. Verified end to end on an address this scraper accepts: /site/itemtags/list.aspx with no tags= really does answer 404 — one fetch, no retries, exit 4, previous output not overwritten.

A run pointed at page 10 quietly returned page 1. --url .../c15_p10/ --pages 2 normalised back to page 1 and reported success. The start page is now honoured — the live log shows c15_p2 then c15_p3 for a run aimed at p2.

The credential scan failed on a clean clone. It skipped virtualenvs by directory name, so following this repo's own README with the venv called anything but .venv walked into it and failed on the 32-hex string in pip's vendored packaging/_elffile.py. A venv is recognised structurally now. Measured across the family: 21 of 26 sibling repos still carry the name-based version.

Two planning implementations became one, and a fabricated fixture was replaced with a real capture — NOTFOUND_HTML had been written by hand with a 404- title and was wrong about the one thing it existed to show: a bogus goods code answers HTTP 200 with full chrome.

Verified

• --concurrency 3 over five pages: 120 rows, 120 distinct skus, merged in page order although they arrived 4-3-2-5.
• All three engines still agree: 48 of 48 identical rows.
• The published description, topics, homepage and release notes carry no banned wording — the surfaces a file scan cannot see.
• 459 offline checks, and every new one was controlled: fix reverted, expected failures confirmed by name, fix restored. One control caught a check of mine that was passing for the wrong reason.

v0.1.0 — Maison KOSÉ

First release. Reads Maison KOSÉ (maison.kose.co.jp), Kosé's own online store.

The domain this repo is named for does not resolve. kose.com is registered (GoDaddy, 1998, paid to 2029) and delegated to two nameservers that answer REFUSED for the zone, so 8.8.8.8 and 1.1.1.1 both return SERVFAIL and no client anywhere gets an address. That is a broken delegation, not a bot manager. Kosé's storefront is maison.kose.co.jp — where www.kose.co.jp sends a visitor with its own meta refresh — and that is what this scraper reads.

You need no account

Measured 2026-09-18 from a datacentre address, and re-measured the same day from a bare GitHub runner by the canary in this release:

request	result
curl/8.5.0	HTTP 200, 90,310 bytes
python-requests/2.31.0	identical
a Chrome User-Agent	identical
no User-Agent header at all	identical
12 pages back to back, no delay	12 × HTTP 200

Zero captcha markers of any vendor across 13 captures. The solver, proxy and fingerprint paths are all wired and all verified working — they simply buy nothing here today. Nothing in this repo claims a captcha on this site could not be solved; there is none to solve.

What it does

Two modes over three routes — category listings, tag facets (which cross brands) and product pages — through four engines. All four were run live; the three browser engines returned 48 of 48 identical rows on the same two pages.

216 rows over four live runs, with title, url, sku, brand, price, currency, in_stock, image_url and tax_rate on 216 of 216. Prices ¥495 – ¥264,000.

Site traps it handles

• The brand segment in every URL is decorative. /site/xyz/c/c15/ returns byte-identical bytes to /site/cosmedecorte/c/c15/, so brand is read per tile — one tag run mixed 11 brands across 72 rows.
• A category listing walked past its end serves its LAST PAGE AGAIN, byte-identical and HTTP 200, rather than 404ing or emptying. Runs plan against the site's own N／Mページ counter and still stop on "no new sku".
• &pageno= and &page= are silently ignored on a tag listing; only ?p= paginates it.
• 5 of 20 product pages canonicalise to a DIFFERENT sku — /g/gJQBA/ is the 60 g cream and points at the 20 g /g/gJQGA/. A row's URL is built from the page's own sku; the canonical goes in variant_of.
• A product page reuses the listing's tile class on <div>s for staff-review cards, 15 on one measured page. The tile selector is element-qualified and sees zero.
• Japan charges two consumption-tax rates and prices here are tax-inclusive, so tax_rate is a column: 2 of 171 tiles carried the reduced 8%.

Known limitation

--cdp-endpoint (the Scraping Browser API) is not live-verified in this repo — the profile available when this was built had expired, and those credentials live about a day. Every other path was run live.

423 offline checks; the canary runs a real 3-page scrape daily from a bare runner with no secrets, because that is the README's central claim.