Releases: 2scraper/transfermarkt-scraper
Release list
v1.1.0 — a live AWS WAF captcha, cleared by the primary path
Read this first if you already use
--cdp-endpointor a proxy — two defaults changed, and both change what you are billed.
- On a Scraping Browser session the engine now waits up to 180s for
Captcha.setAutoSolveto finish before calling the paid solver API. It used to call the solver immediately, which cost money, cleared nothing, and reloaded the page out from under the auto-solver. Expect a challenge to take longer and cost less.- An AWS WAF Challenge-action page (no captcha widget) is no longer sent to the solver at all. There is no puzzle on one, and
createTaskwill happily charge for a task built from it.- With a proxy in play, the AWS WAF task type is now
AmazonTask, carrying your exit, instead ofAmazonTaskProxyless. The proxyless type solves from 2Captcha's address, not the challenged one, and was returning tokens that cleared nothing.
The release in which the primary path was finally exercised: a live AWS WAF captcha, met and cleared by the Scraping Browser's own auto-solve, with a control proving the block does not lift by itself.
| Challenges met | Outcome | |
|---|---|---|
Control — --no-autosolve, nothing solving |
1 | 0 of 8 reloads cleared it |
| Primary path — auto-solve on | 5 | 5 of 5 → 25 rows, status=complete |
Added
tools/browser_profile_client.py— fetches a readyconnectionUrirather than assembling one, and refuses an account whoseproxyModeisnone, which is whatERR_TUNNEL_CONNECTION_FAILEDactually was.--proxy-sessions Nmints a pool from one credential instead of a file of them. Measured: ten sessions, ten distinct exits.--no-autosolve, which is what makes the control above possible at all.
Also corrected
- The "profile credentials live about a day" figure was inherited and wrong: the vendor documents 90 days from creation or until deleted, and they do not expire on a timer. The rule against publishing an endpoint stands, with the opposite reasoning.
- A duplicate key in
.envresolved to the FIRST value with the built-in parser and the LAST withpython-dotenv. Both now agree, and the loader warns with line numbers. mask_url()could raise on a malformed port — the defect that once put a proxy password in a public CI log.
Full detail in CHANGELOG.md.
v1.0.0 — every engine and the container verified live
First release in which every engine, and the container, have been run against the live site and compared row for row. The version number is that, not a feature list.
Two of the three browser engines had never completed a live fetch of this site. Both were broken, and neither failure was visible to a green offline suite.
Fixed
- Selenium could not load this site at all. Its default page-load strategy blocks until a
loadevent www.transfermarkt.com does not fire — 60s timeout on every attempt, whileexample.comreturned in 0.2s through the same driver. Noweager. - pyppeteer could not launch on Apple silicon. The Chromium it downloads for itself is x86_64 and dies under Rosetta. Added
--chromium-path.
Added
- AWS WAF parity across all three engines — previously only Playwright could act on the one challenge this site serves.
- A parity test that drives all three real handlers against the challenge fixture.
Measured 2026-09-16
Playwright, Selenium, pyppeteer and the Docker image each returned 25 rows at 100% price coverage; diff_runs.py reports 0 added, 0 removed, 0 changed across all six pairings. The Docker image was built and run for the first time in this repo's history.
Full detail in CHANGELOG.md.