Repository navigation
kohls-scraper v1.0.0
First release, rebuilt on the 2scraper family core. It replaces an
unpublished August 2026 prototype, whose README is not carried over because
most of what it described is no longer true of the site or of the code:
If you used the prototype: its flag for a "local auto-solve API" is
gone — it called a placeholder endpoint that never existed. The parser no longer reads
Tailwind CSS classes; it reads the listing's server-rendered data island.
Pagination isWS=, verified per page, where the prototype had none that
worked. Credentials now come from.env, not from the command line.
Added
- Three browser engines behind one bridge.
playwright_scraper.py,
puppeteer_scraper.pyandselenium_scraper.pysharebrowser_bridge.py
for everything they decide — retries, exit rotation, page states,
planning, concurrency, exit codes — and each only drives its browser.
Measured live on 2026-09-25: Playwright and pyppeteer returned the same 334
rows for one listing, 0 differences. --mode listingfor category grids and search results, from the
Astro island'scatalogData(120 products a page), with a URL-pattern
fallback over the rendered grid that emitted 1,216 of the island's 1,223
rows across 11 captures (7 had no rendered tile) and agreed on every one.--mode product: brand, price ranges, per-SKU stock counts.- Pagination by
WS=offset, checked against the page the server says it
served, and planned against the listing's owntotalPages. scraper_api_client.py(the 2Captcha Scraper API): 120 rows with
--use-cdprouting through the Scraping Browser; refused without it.- Columns beyond the family prefix:
page,position,price_max,
original_price_max,price_label,sort,skus_total,
skus_in_stock. The sidecar records the listing's owntotal_results,
pages_availableandper_page. make_fixtures.py, which builds the offline suite's fixtures from real
captures kept outside the repo, scrubs them, and refuses to write any that
do not parse exactly like their originals.
Fixed — in the family core this repo was built from
Found while bootstrapping, and worth checking in the sibling repos (verify
before patching; not every one will have them):
env_config.pyechoed a user's REAL value to the terminal when it was
left inside braces in.env(the placeholder warning quoted whatever it
found in{…}). It now names only the template's own placeholders.scraper_api_client.pylogged the Scraper API's error BODY unredacted
one screen below thex-debugheader it did redact; the body can echo the
cdpurlwith its password. Also: a non-JSON answer crashed with exit 1,
and a blank document was reported as "0 products" (exit 4) instead of
"never obtained" (exit 5).- The pyppeteer engine accepted
--fingerprintand never applied it, and
the Selenium engine read the user agent fromuserAgent.value, a key the
Fingerprint API does not return. All three engines now build their
identity from one command list (fingerprint_client.cdp_identity_commands).
Not run live — see README. - The pyppeteer engine never sent ANY CDP command it thought it was sending:
pyppeteer'sCDPSession.sendreturns a Future, not a coroutine, and the
engine's loop bridge rejected it — soCaptcha.setAutoSolvewas never
enabled (and the log blamed the endpoint) and--fingerprintwould have
crashed. Found by an independent review; re-run live: "enabled". - The pyppeteer and Selenium engines set a Windows user agent on every
platform — a bare UA override, which a sibling repo measured as served on
navigation 1 and denied on 2-4. None of the engines overrides the UA now. - The readiness wait handed the browser an evaluated STRING, which dies
under a Content-Security-Policy withoutunsafe-eval; it polls
querySelectorAllnow, and treats reaching the threshold as success. .github/ci_checks.pytaken from the newest sibling: it sees credentials
inside JSON-escaped fixtures and skips virtualenvs bypyvenv.cfg.
Known limits
- kohls.com refused every local browser and HTTP client tested, from
residential US exits and datacentre addresses alike, and served the
Scraping Browser API. See README "What you need". - Selenium cannot use the Scraping Browser endpoint (it cannot send the
password) nor authenticate a proxy, so it was run live on the refused
paths only. --fingerprintwas not run live: the account used has no Fingerprint API
plan.- This repo does not implement a solver for Akamai's behavioural challenge;
it is detected and reported as exit 3.