Skip to content

Releases: 2scraper/transfermarkt-scraper

v1.1.0 — a live AWS WAF captcha, cleared by the primary path

Choose a tag to compare

@2ione 2ione released this 17 Sep 06:42

Read this first if you already use --cdp-endpoint or a proxy — two defaults changed, and both change what you are billed.

  • On a Scraping Browser session the engine now waits up to 180s for Captcha.setAutoSolve to finish before calling the paid solver API. It used to call the solver immediately, which cost money, cleared nothing, and reloaded the page out from under the auto-solver. Expect a challenge to take longer and cost less.
  • An AWS WAF Challenge-action page (no captcha widget) is no longer sent to the solver at all. There is no puzzle on one, and createTask will happily charge for a task built from it.
  • With a proxy in play, the AWS WAF task type is now AmazonTask, carrying your exit, instead of AmazonTaskProxyless. The proxyless type solves from 2Captcha's address, not the challenged one, and was returning tokens that cleared nothing.

The release in which the primary path was finally exercised: a live AWS WAF captcha, met and cleared by the Scraping Browser's own auto-solve, with a control proving the block does not lift by itself.

Challenges met Outcome
Control — --no-autosolve, nothing solving 1 0 of 8 reloads cleared it
Primary path — auto-solve on 5 5 of 5 → 25 rows, status=complete

Added

  • tools/browser_profile_client.py — fetches a ready connectionUri rather than assembling one, and refuses an account whose proxyMode is none, which is what ERR_TUNNEL_CONNECTION_FAILED actually was.
  • --proxy-sessions N mints a pool from one credential instead of a file of them. Measured: ten sessions, ten distinct exits.
  • --no-autosolve, which is what makes the control above possible at all.

Also corrected

  • The "profile credentials live about a day" figure was inherited and wrong: the vendor documents 90 days from creation or until deleted, and they do not expire on a timer. The rule against publishing an endpoint stands, with the opposite reasoning.
  • A duplicate key in .env resolved to the FIRST value with the built-in parser and the LAST with python-dotenv. Both now agree, and the loader warns with line numbers.
  • mask_url() could raise on a malformed port — the defect that once put a proxy password in a public CI log.

Full detail in CHANGELOG.md.

v1.0.0 — every engine and the container verified live

Choose a tag to compare

@2ione 2ione released this 17 Sep 06:42

First release in which every engine, and the container, have been run against the live site and compared row for row. The version number is that, not a feature list.

Two of the three browser engines had never completed a live fetch of this site. Both were broken, and neither failure was visible to a green offline suite.

Fixed

  • Selenium could not load this site at all. Its default page-load strategy blocks until a load event www.transfermarkt.com does not fire — 60s timeout on every attempt, while example.com returned in 0.2s through the same driver. Now eager.
  • pyppeteer could not launch on Apple silicon. The Chromium it downloads for itself is x86_64 and dies under Rosetta. Added --chromium-path.

Added

  • AWS WAF parity across all three engines — previously only Playwright could act on the one challenge this site serves.
  • A parity test that drives all three real handlers against the challenge fixture.

Measured 2026-09-16

Playwright, Selenium, pyppeteer and the Docker image each returned 25 rows at 100% price coverage; diff_runs.py reports 0 added, 0 removed, 0 changed across all six pairings. The Docker image was built and run for the first time in this repo's history.

Full detail in CHANGELOG.md.