v0.1.4 — fingerprint_client reads .env

A one-defect patch, and the defect is that a documented mechanism was not
applied on one path.

fingerprint_client.py's --key defaulted to
os.environ.get("TWOCAPTCHA_KEY") and only that. So a key put in .env —
exactly as §3, the README and .env.example instruct — worked for every
engine and failed here with "No API key".

It now reads through env_config.env_value, calling load_env() itself
because this is a standalone entry point that no engine has necessarily run
first.

Why the loader and not os.environ

Measured both ways with TWOCAPTCHA_KEY=your_2captcha_api_key_here exported:

	What the user saw
before	Fingerprint API rejected the key (401) — note this is a separate subscription from captcha solving
after	TWOCAPTCHA_KEY is still set to the placeholder from .env.example — treating it as unset

The old message sent the reader off to check a subscription they never
needed.

Provenance

Found on a sibling repo's first live --fingerprint run, then checked
across the family before patching: five repos had it, one had already fixed
it. This release makes all seven identical.

Pinned by a check verified to fail on the old code — that it loads .env
itself, reads through the loader, does not read os.environ directly,
that a placeholder still reads as unset, and that the help string does not
interpolate its default, which is one substring away from printing a live
credential to anyone who types --help.

Nothing else changed. No flag, no default, no output column.

v0.1.3 — the fingerprint's device scale factor is applied

If you use --fingerprint, this one changes what the browser reports.

playwright_context_kwargs mapped the user agent, the locale, the timezone
and the screen onto the browser context and ignored deviceScaleFactor,
which the fingerprint API returns beside them.

Measured 2026-09-11 against the live API and a live browser:

	fingerprint says	browser reported, before
user agent	Chrome/146 on Windows	✅ same
locale	nl-NL	✅ same
timezone	Europe/Amsterdam	✅ same
screen width	1536	✅ same
devicePixelRatio	1.25	1 ❌

That is the mismatch the flag exists to prevent, on an axis any
fingerprinter reads for free and for nothing: the paid identity said one
thing and the browser said another, on every run, silently.

Playwright takes it as its own context option, so the fix is to pass it.
Verified in a live browser both ways and pinned in the offline suite.

Found while auditing a new sibling repo against the family notes — all
five repos in this family had it.

🤖 Generated with Claude Code

v0.1.2 — captcha detection: the three shapes the site's own reCAPTCHA would take

No captcha was encountered on this site — and it is configured for one on every page, which turns out to matter.

What 13 dumps say

Six captures (two page kinds, two exit countries, the empty-result page, a discovery hub, a detail page) plus every live run's snapshot and debug dump:

zero reCAPTCHA · zero hCaptcha · zero Cloudflare Turnstile · zero DataDome · zero PerimeterX · zero Imperva/Incapsula · zero Kasada · zero AWS WAF · no Akamai refusal page · no challenge iframe · no data-sitekey in served markup.

The apparent exception is ours. A dump fetched over --cdp-endpoint carries 16 chrome-extension://…/content/captcha/… scripts — the Scraping Browser's auto-solve extension injecting hunters for turnstile, recaptcha, arkoselabs, geetest, keycaptcha, lemin, captchafox, mt_captcha, amazon_waf and yandex into every page it loads. Strip those and the count is zero, which is why _strip_extension_tags runs before any marker scan and why cf-turnstile is deliberately not in this repo's marker set.

But the site ships the machinery anyway

On every page it serves:

<captcha-widgets></captcha-widgets>     an EMPTY custom element
"CAPTCHA_SITE_KEY":"6L…"                a reCAPTCHA key in its front-end config

So Tokopedia has reCAPTCHA wired and simply does not render it to an anonymous visitor reading listings. It presumably does on sign-in, checkout, or for an address it is unsure about — so the shapes it would take are worth detecting, and three of them were not.

Fixed

shape	before	now
the loader (recaptcha/api.js)	detected	detected
g-recaptcha class	detected	detected
a rendered widget's api2/anchor iframe	missed	detected
its api2/bframe challenge frame	missed	detected
a child of the mount point carrying only data-sitekey	missed	detected
the mount point populated with anything at all	missed	detected, structurally

The last one could not be a substring marker: the bare tag is on 13 of 13 dumps, so matching it would report a challenge on every page — the same mistake the bare string akamai made in 0.1.0, one release ago. captcha_widget_is_populated() checks the element for non-whitespace content instead.

All three are pinned in both directions — the empty mount point must NOT be a challenge, and whitespace-only content must not either — plus a re-check that all three committed fixtures still report clean. Every broadening of a marker set risks the "matches every page" failure, so it is verified rather than argued about.

A refusal, meanwhile, is still not a page: there is nothing to solve there whatever happens.

479 offline checks, all green.

v0.1.1 — pre-publication audit: five defects fixed

Two of these made a documented feature not work at all. If you are already using this, both are worth knowing.

A copied .env.example read as CONFIGURED. The placeholder check was a literal set, so it caught your_2captcha_api_key_here and missed the two credentialled URLs — which the file documents the way the vendor does, with the parts you fill in in braces. So cp .env.example .env and a run connected to cb.2captcha.com with the string {login}-zone-… as its username and got a 401 a long way from its cause. Any {…} left in a value now reads as unset, and python3 env_config.py names which one.

--fingerprint never worked. The engines' --fp-tags default was Windows,Chrome,Desktop, and the fingerprint API rejects it with HTTP 400 — while fingerprint_client.py's own --tags help has always said ONE OS-family tag, not a list. Measured against the live API: Windows succeeds; Windows,Chrome,Desktop, Chrome and Desktop each 400. Fixed and verified end to end. This one was in all four sibling repos too, and each has its own fix.

A pass over the family notes section by section before making this repo public. Five defects, every one found by checking rather than by reading.

Also fixed

• A no-results page could be reported as blocked. detect_page_state checked the served-by-Tokopedia heuristic (two or more asset references) before the site's own Oops, produk nggak ditemukan sentence, so a minimal real page carrying one reference instead of the measured 3-7 came back as exit 3 — a proxy hunt for a correct answer.
• RETRY_ON_BLOCKED was a policy nothing enforced. A constant with a paragraph of justification that no engine read; setting it False changed nothing. Now consulted by all three.
• Half the CLI was undocumented. Sixteen real flags appeared in no document. The README now has every flag with its default, and the three engines' flag sets are pinned against the family contract and against each other in both directions — so a new divergence fails a check, and closing a documented one does too.

Removed and documented

page_flow.page_bound (never called, always returned its input) and page_flow.page_number (a wrapper that only delegated) are gone.

proxy_pool.mask() takes a bare URL, not a sentence: given one it returns ?://? — the password is gone, which is the property that matters, but so is the host and port the log was written to show. Every call site passes the URL as its own argument for that reason, and the limitation is now pinned rather than half-guarded.

Two README figures that legitimately vary between runs are stated as ranges rather than as fractions that would go stale.

Audited and clean, for the record

No credential leaks on any error path — the fingerprint API, both captcha API versions, getBalance, the Scraper API and all three engines' maskers, each tested with a 32-hex key and a password in a URL. Git history clean across 54 blobs. .env untracked and ignored. ENV_KEYS matches .env.example in both directions with nothing mapped onto a defaulted flag. pyproject's py-modules and all three extras match the real modules and requirements files. The banned-wording set appears only inside the checker. The suite mutates no file. Every numeric claim in the README was re-verified against the run artefacts.

467 offline checks, all green.

v0.1.0 — rewrite as a member of the 2scraper family

This replaces the April 2026 prototype and shares no data contract with it. If anything consumed the old output it needs rewriting rather than adjusting: price was the string "Rp17.999.000" and is now 17999000.0 with currency: "IDR"; discount was "18%" and discount_pct is now a number computed from the two prices; sold was "2rb+" and is now 2000 with sold_is_floor: true beside it. There are exit codes and a <out>.meta.json sidecar where there were none, --mode shop and --country do not exist, and the prototype's twenty /p/<slug> "categories" all turn out to be hub pages with no product grid on them.

Scrapes Tokopedia search grids, category listings and product pages through Playwright, Selenium or pyppeteer, with JSON/CSV output, a run-metadata sidecar and the family's exit-code contract.

What you need before it works

A residential exit. The country does not matter. Tokopedia does not refuse an address it has scored — it ignores it: the connection is accepted, TLS completes, the HTTP/2 stream opens, and then it is reset. No status code, no interstitial, no challenge, no error page. A 2Captcha solving key does nothing about that, because there is nothing to solve.

Measured 2026-09-10: an Indonesian exit and a US one both returned HTTP 200 and 95 products, with identical markup, identical <html lang="id">, IDR prices both times and zero price differences across the 68 products both runs saw. So this is one storefront with one language and one currency, and what the site scores is the address rather than its geography.

A healthy run

category listing, 2 pages   119 rows, 100% price coverage
search grid                 155 rows, 100% price, 98% rating, 100% sold
--mode product              the site's real product id, the EXACT sold
                            count, review count, condition, weight

Three numbers look like defects and are not: image_url on ~37% of rows (Tokopedia lazy-loads tile images, and this parser reports null rather than the placeholder a tile below the fold carries), slug_id on ~70% (4 of 40 product URLs have no 19-digit tail — and that tail is not the product id anyway), and rating/sold/original_price null on every row of a category run (that tile kind prints none of them).

Four defects the first live runs found

Every one was invisible to a green offline suite, and every one is pinned by a check now.

1. The readiness wait crashed on /search — the site's Content-Security-Policy has no unsafe-eval, and Playwright's wait_for_function hands the browser a string to evaluate. Every readiness wait now polls querySelectorAll over CDP.
2. Two of the three engines crashed on their FIRST fetch — a signature mismatch invisible to import, --help, compileall, an AST walk and 400+ green assertions. The suite now binds every shared-module call in every engine against the real signature.
3. A /p/<slug> hub reported exit 3 — the bare string akamai was a challenge marker, and this site is fronted by Akamai: its own performance script names akamaihd.net on every page it serves.
4. page was 1 on every row of a two-page run, which made position ambiguous across pages.

Known limitations, measured

• --proxy does not work from a browser here, and that is documented as unexplained rather than papered over: the 2Captcha residential gateway reaches an Indonesian exit and plain requests goes through it, but every Chromium navigation through it times out — including a control site. Use --cdp-endpoint.
• A search listing has no per-page addresses. ?page=2 on a search URL empties the result set rather than advancing it, so --concurrency above 1 is refused for one, with that reason. A category listing paginates normally.
• The Scraper API path returns about 5 products where a browser returns 60 — the grid hydrates only as the page is scrolled, so a single browserless fetch sees the first paint.
• No challenge of any kind has been observed on this site, so a solving key buys nothing here today. The path is wired up and capped at one purchase per page.

Checks

439 offline checks, with fixtures cut from real captures and verified to parse identically to the untrimmed original. They assert values — the expected price, rating, sold count and title for named products — because a column can be 100% populated and entirely wrong. CI also builds the Docker image, runs its entrypoint, launches Chromium inside it and asserts no .env is baked in.

The canary is dispatch-only rather than nightly: a Scraping Browser credential on this account does not survive a day, and a check that is always red teaches everyone to ignore checks. It skips with a notice when the secret is absent.