v0.1.8 — fingerprint_client reads .env

A one-defect patch, and the defect is that a documented mechanism was not
applied on one path.

fingerprint_client.py's --key defaulted to
os.environ.get("TWOCAPTCHA_KEY") and only that. So a key put in .env —
exactly as §3, the README and .env.example instruct — worked for every
engine and failed here with "No API key".

It now reads through env_config.env_value, calling load_env() itself
because this is a standalone entry point that no engine has necessarily run
first.

Why the loader and not os.environ

Measured both ways with TWOCAPTCHA_KEY=your_2captcha_api_key_here exported:

	What the user saw
before	Fingerprint API rejected the key (401) — note this is a separate subscription from captcha solving
after	TWOCAPTCHA_KEY is still set to the placeholder from .env.example — treating it as unset

The old message sent the reader off to check a subscription they never
needed.

Provenance

Found on a sibling repo's first live --fingerprint run, then checked
across the family before patching: five repos had it, one had already fixed
it. This release makes all seven identical.

Pinned by a check verified to fail on the old code — that it loads .env
itself, reads through the loader, does not read os.environ directly,
that a placeholder still reads as unset, and that the help string does not
interpolate its default, which is one substring away from printing a live
credential to anyone who types --help.

Nothing else changed. No flag, no default, no output column.

v0.1.7 — the fingerprint's device scale factor is applied

If you use --fingerprint, this one changes what the browser reports.

playwright_context_kwargs mapped the user agent, the locale, the timezone
and the screen onto the browser context and ignored deviceScaleFactor,
which the fingerprint API returns beside them.

Measured 2026-09-11 against the live API and a live browser:

	fingerprint says	browser reported, before
user agent	Chrome/146 on Windows	✅ same
locale	nl-NL	✅ same
timezone	Europe/Amsterdam	✅ same
screen width	1536	✅ same
devicePixelRatio	1.25	1 ❌

That is the mismatch the flag exists to prevent, on an axis any
fingerprinter reads for free and for nothing: the paid identity said one
thing and the browser said another, on every run, silently.

Playwright takes it as its own context option, so the fix is to pass it.
Verified in a live browser both ways and pinned in the offline suite.

Found while auditing a new sibling repo against the family notes — all
five repos in this family had it.

🤖 Generated with Claude Code

v0.1.6 — a bad proxy URL is refused, not crashed on, and never logged

If you pass a proxy through --proxy, --proxy-file or *_PROXY, upgrade. A malformed value crashed the run with a traceback that contained the proxy login and password — so a bad entry could put a live credential into any log the run was piped to, including a public CI log. That is what happened here.

Three defects, all in proxy_pool.py, all triggered by one wrong value on the canary's first run with a real proxy secret.

A malformed URL crashed the run instead of being refused

parse_proxy_line validated the scheme and the host but never the port — and urlparse computes a port lazily, raising only when something finally asks. So a bad entry passed validation and blew up several calls later: exit 1 (crash) where it should have been exit 2 (bad usage).

The value that caused it is the mistake a new user makes:

proxy list file:  scheme://host:port:login:password    ← what was pasted
a proxy URL:      http://login:password@host:port      ← what is wanted

The refusal now says exactly that.

mask() raised on the values that most needed masking

It read parsed.port too — so the one function standing between a password and a log blew up on a malformed URL, and the caller printed the raw string instead. That is how the credential reached the log.

A masker must never raise; it is called precisely when the value is already wrong. It is now total, and every ProxyError reports mask(line) rather than the line.

A pool of N identical entries claimed to be N exits

A copied proxy list is often one address repeated. The pool reported exit 2/50 on every rotation while all fifty left from the same place, and the single-exit warning never fired because it counted entries rather than addresses — so a run believed it was spread over fifty exits while burning one. Duplicates are now dropped and the collapse is logged.

Also removed

A test asserting that "http://host:port:login:pass" "is understood". It checked only that the parser did not reject the line, so it passed while the value was never usable. A test that asserts a function did not complain is not a test that its answer was right.

Same fix shipped to farfetch-scraper and amazon-scraper, which share this file.

Full detail in CHANGELOG.md.

v0.1.5 — --fingerprint now applies the fingerprint

If you have been relying on --fingerprint, it was doing much less than it claimed. The user agent was never set at all, the locale contradicted the fingerprint, and the timezone was ignored. Nothing errored — the flag simply left the browser on its own identity while replacing parts of it.

What	Was	Now
User agent	never set — read from userAgent.value, a key the API returns in neither format	userAgent.userAgent / data.ua
Locale	en-DE for a German fingerprint	de-DE, from intl.contentLocale
Timezone	not applied	Europe/Berlin, from intl.timeZone
Window size	guessed as screen height − 120	the response's own outerWidth/outerHeight

The UA one matters most: the run presented a German fingerprint's screen and locale wearing a local Chromium's user agent — exactly the identity mismatch this project warns about when it says never to stack a second identity on a remote browser.

Verified against a live browser

With a German Windows fingerprint applied, the page now reports that UA, de-DE, Europe/Berlin, and a window smaller than its screen — every one matching the fingerprint rather than the host.

Tests

All of the above, plus the key redaction from v0.1.4, are now pinned against a fingerprint fixture cut from a real response. Including a check that every kwarg passed to new_context is one Playwright accepts — an unknown key is a TypeError at launch, on the paid path, at runtime.

smoke_test.py: 330 checks, up from 311.

Full detail in CHANGELOG.md.

v0.1.4 — key leak fixed; the key-gated paths verified

Security: fingerprint_client.py could print your 2Captcha API key. It sends the key as a URL query parameter, and requests puts the full URL into the text of every HTTP and connection error — so any failure from that endpoint echoed a live key to the terminal and to any log the run was piped to. Upgrade, and rotate the key if you have run that module and kept the output. The same hardening is applied to captcha_solver.py's v1 polling call, the other place the key rides in a URL.

The fingerprint and Scraper API paths had never been run — no key was available. Running them found three defects.

Fixed

• The API key reached the terminal. Every error surfaced from fingerprint_client.py and from captcha_solver.py's v1 polling call is now redacted before it is raised or logged. The endpoint and status survive; the key becomes key=***. The other key-carrying calls were checked and are safe — createTask/getTaskResult use the JSON body, the Scraper API a Bearer header.
• scraper_api_client.py reported a block as an empty result. It discarded the upstream HTTP status and looked only for a challenge marker, and MediaMarkt's 403 page carries none — so a refused request exited 4 ("zero products") instead of 3 ("blocked"). A pipeline branching on the exit code read a block as an empty category. It now uses the same detect_page_state the three browser engines do.
• The documented --tags example never worked. "Windows,Chrome,Desktop" returns 400 every time. tags takes one OS-family value: Windows, Microsoft Windows and Android are accepted; Chrome, Desktop, Mobile and Unknown are rejected, and no combination is accepted with any separator.

Measured, having been guessed

The Scraper API against this site:

without --cdp-url   upstream 403,    13,922 bytes,  0 products -> exit 3
with    --cdp-url   upstream 200, 1,693,678 bytes, 12 products -> exit 0
                    12/12 confirmed against a rendered tile, EUR

Its own exit is refused like every other datacentre address, so --cdp-url is not optional here — but at $0.0005 a task it is the cheapest way to read this site once a residential session is in the path.

--fingerprint end to end: fetched, cached and applied, after which MediaMarkt still answered 403 from a datacentre address — which is the point.

Concurrent dispatch, tested at last

No live run can reach it where page 1 is blocked, so it is now driven directly with the browser stubbed out: every queued page fetched exactly once, outcomes restorable to page order, the end-of-listing event stopping dispatch (4 fetches against 49 queued pages), unattempted pages reported rather than counted as failed, and a dying worker neither hanging the run nor losing its siblings' pages.

smoke_test.py: 311 checks, up from 301.

Full detail in CHANGELOG.md.

v0.1.3 — the untested paths, run and fixed

If you built the Docker image from v0.1.0–v0.1.2, it did not work. proxy_pool.py was missing from the Dockerfile's COPY list, so the container died with ModuleNotFoundError on every invocation including --help. Rebuild from v0.1.3.

Three defects, all invisible to a green CI, all found by executing something rather than reading it.

The pyppeteer engine died on its first real page

detect_page_state was called on a line reached only while fetching, after that import had been removed in favour of page_flow.classify. The module imported cleanly, --help worked, compileall passed, the whole offline suite passed and CI was green — and the engine raised NameError the moment it loaded a page.

Byte-compiling proves a file parses. It says nothing about whether the names in it resolve, and the paths where they do not are exactly the ones an offline suite never executes. The suite now walks every module's AST for names that are never imported, defined or assigned.

The Docker image was broken on every invocation

Including --help, its own CMD. The Dockerfile COPYs an explicit module list — correct, so the image carries no test suite and no stray .env — and the list had fallen behind the imports. CI never builds the image, so nothing noticed. The suite now checks that list against the entrypoint's transitive imports, with no Docker required.

Verified, having previously been assumed

• pyppeteer live: 24 rows over 2 pages, 24/24 confirmed, and the same 24 skus the Playwright engine returned for the same category. The one price that differed had genuinely changed on the site between the two runs.
• Selenium's two documented refusals: a credentialled --cdp-endpoint exits 2 with the credential masked; a user:pass proxy is stripped without the password reaching the output.
• The canary, dispatched for the first time: with no proxy secret it takes the skip path and completes green in 12s with a notice explaining why.
• The canary's assertion block, run against a real 36-row output.

Still not run, stated rather than implied

--concurrency > 1 and --proxy-file rotation against the live site, scraper_api_client.py (no subscription), fingerprint_client.py (needs a key), and an actual docker build.

smoke_test.py: 301 checks, up from 284.

Full detail in CHANGELOG.md.

v0.1.2 — all ten country sites verified

mediamarkt.lu is no longer a supported host. It is a real MediaMarkt shop, but it runs on Shopify rather than on the MediaMarkt Saturn platform this scraper reads — no /category/ or /product/ paths, no product cards, no product JSON-LD. A run against it would have reported an empty category. It is now refused with that reason.

All ten supported country sites are live-verified, each from a residential exit in its own country. Every one returns twelve products a page with every row confirmed against its rendered tile.

Host	Currency	Host	Currency
mediamarkt.de	EUR	mediaworld.it	EUR
mediamarkt.at	EUR	mediamarkt.pl	PLN
mediamarkt.ch	CHF	mediamarkt.hu	HUF
mediamarkt.nl	EUR	mediamarkt.com.tr	TRY
mediamarkt.be	EUR	mediamarkt.es	EUR

Fixed: a discount badge read as a price on the Turkish site

Turkish puts the percent sign before its number and the currency symbol before its own — -%10,34 ₺25.999,– — so the pattern's trailing-symbol form matched 10,34 ₺: the badge's number wearing the next price's symbol. A 25,999 TRY air conditioner parsed as costing 10.34.

It reached the output only as an unconfirmed row, because the structured price disagreed with the tile and the parser kept the structured one — the guard worked, the parse did not. Percentages are now stripped before matching, in both word orders (German writes -16%, Turkish -%10,34), because merely rejecting the match still consumed the currency symbol and lost the real price with it.

The exit country has to match the site

v0.1.1 called this an assumption. It is now measured: one German residential address was accepted by .de, .es and .pl and refused with 403 by the other seven, each of which answered normally from an exit in its own country.

A residential IP is necessary and not sufficient. With a 2Captcha proxy the fix is one word (-region-de → -region-it); with the Scraping Browser API it is the country- segment of the endpoint.

Also

• A refused host now says why. mediamarkt.lu and the two Saturn domains each get their own reason instead of "is not a MediaMarkt site" — which was false for all three.
• smoke_test.py: 284 checks. The new Turkish fixture carries a prefixed currency, a percent sign before its number and an instalment line in one price block.

Full detail in CHANGELOG.md.

v0.1.1 — two silent locale bugs fixed

If you pulled v0.1.0 and ran it against mediamarkt.pl or mediamarkt.lu: your original_price and lowest_price_30d columns were empty on every row, and price_source was jsonld on every row. Nothing errored and the prices themselves were correct. Re-run those sites on v0.1.1.

Two bugs, both found by live-verifying a second country site rather than by reading the code. Neither could show on mediamarkt.de, and neither produces an error — rows, titles and prices stay correct throughout, because they come from the structured data. Only the columns that depend on reading the rendered page disappear.

mediamarkt.pl and mediamarkt.lu lost their DOM-only columns

Both answer without a www. prefix; their own hreflang entries say so. The parser rebuilt every product URL as https://www.{host}{path}, which never matched the page's own address, so the join between a structured row and its rendered tile failed on every row.

	v0.1.0	v0.1.1
rows / priced	12 / 12	12 / 12
confirmed	0	12
original_price, lowest_price_30d	always null	populated

URLs are now resolved against the page's own address, and the tile join ignores www., percent-encoding and trailing tracking parameters — three differences that do not change which product a URL addresses, and each of which empties the same two columns.

A product title was read as the catalogue counter

Polish writes "of" as a bare z. The real title "ELECTROLUX LVM8E08Z 44l" contains 8Z 44, read as "8 of 44" — so a listing reported a catalogue of 44 against its own printed 85. Only a text node that is the count is accepted now, and the engines pass the row count they actually parsed so the read is checked rather than scanned for.

Also in this release

• Three country sites are live-verified, up from one: .de, .es and .pl, each 12 products a page at 100% DOM confirmation, in EUR, EUR and PLN.
• A residential exit is necessary but not always sufficient. One German address was accepted by .de, .es, .pl and refused with 403 by the other eight. An exit in the target country is the obvious answer, and the README states it as an assumption because those eight were not retried locally.
• smoke_test.py: 269 checks, up from 255. The new Polish fixture is a second language, a non-euro currency and the non-www host in one, and it carries the decoy title beside a genuine counter so both regressions are pinned rather than described.

Full detail in CHANGELOG.md.

v0.1.0 — rewritten as a member of the 2scraper family

Before your first run: MediaMarkt answers every datacentre and VPN address with HTTP 403, on every URL of every country site — in a real headless Chromium as well as a plain HTTP client (measured 2026-09-09 on five hosts). A residential exit is a requirement here, not an optimisation. A 2Captcha key does not substitute for one: the block page carries no challenge to solve, and this scraper deliberately does not send it to the solver.

First release of the rewritten scraper. Replaces three standalone scripts that shared nothing with the rest of this family and whose site knowledge was largely wrong — the hard-coded category map pointed smartphones at category 486, which is actually Filme & Serien. Nothing from that version survives.

What is new

• Two modes. --mode listing (category grids and search results, which share tile markup exactly) and --mode product (adds brand, ean, description and the full image list).
• Eleven country sites, taken from mediamarkt.de's own hreflang declaration rather than guessed. The hostname decides which — no --country flag that could disagree with the URL.
• --concurrency, safe because pagination is addressable: page 1's link[rel=next] is exactly what the ?page=N convention builds, and the run verifies that agreement before planning ahead.
• lowest_price_30d, a column of its own for the EU Omnibus 30-day-low disclosure.
• price_source on every row, with per-page DOM-confirmation logging that warns below 90%.
• Three engines plus a remote browser over CDP, all agreeing on rows, exit codes and run status.

The trap this parser exists to avoid

MediaMarkt renders two struck-through prices in nearly identical markup. -type-rrp is the UVP and is a genuine old price; -type-lop is "Tiefstpreis (30 Tage)" and is normally below the current price. Reading both into original_price gives original=299 against price=349 — a negative discount on a product that is not discounted. They now go to different columns, and the canary fails on a crossed pair.

Also worth knowing

• Hub categories legitimately return nothing. /de/category/notebooks-680.html answers 200 with no product grid. That is exit 4, not exit 3, and an empty page is never retried.
• brand is null on every listing row — measured across 96 tiles, the markup has no brand line. Use --mode product.
• rating is null where nobody has rated the product. The star widget shows "0 von 5" when empty; that is not a score.
• If you use the Scraping Browser API with your own tooling: its auto-solve extension injects cf-turnstile into every page it loads. A marker-based detector will call a perfectly good page blocked. This repo strips extension-injected scripts first.

Verification

smoke_test.py: 255 offline checks, no network, no browser, no credentials — every field assertion pins a value read off a real capture rather than checking for not-null.

Live: 36 rows over 3 pages, 100% price coverage, 100% DOM confirmation, status=complete. sample_output.{json,csv} is cut from that run.

The canary skips rather than fails when MEDIAMARKT_PROXY is unset, because a GitHub runner is a datacentre address and would be red every day.

Full detail in CHANGELOG.md.