Releases: 2scraper/catawiki-scraper
Release list
v0.1.0 — Catawiki auction scraper
First release. The original generated scripts are replaced by the
2scraper family architecture: one row schema,
one set of exit codes, one page_flow policy shared by three engines, 465
offline checks, and every engine and paid path run live against the site.
Read this first if you are used to the sibling repos:
--headfulis the
default here. Catawiki refuses a headless browser regardless of the exit
address — HTTP 403 and a 394-byte "Access Denied" from four residential
exits and one datacentre address, against HTTP 200 and the full catalogue
from those very same addresses with a real window. A proxy does not fix it,
--headlesswill report exit 3, andpage_flow.RETRY_ON_BLOCKEDisFalse
because rotating an exit spends the retry budget on a change that cannot
help.
What it reads
--mode listing— a category (/{loc}/c/{id}-{slug}), a search
(/{loc}/s?q=) or a whole auction (/{loc}/a/{id}-{slug}). An auction page
is the best source of the three: it carries every one of its lots in one
response (130 of 130 on the page measured) plus the auction's absolute
close time, which a category listing states nowhere.--mode lot— one lot in full: the expert's estimate, the seller and
their feedback score, the specifications, the bid-history depth and both
absolute bidding times.--mode auctions— the auctions index as a work list, one row per
auction.
Four back ends behind one schema: Playwright (primary), pyppeteer, Selenium,
and the 2Captcha Scraper API. Verified live on the same URL — 48 rows each,
identical sku sets, 44 identical columns in the same order.
What an auction row means
price is three different quantities and bid_kind says which: a live
current bid, a final bid on a closed lot (a hammer price only when
sold is also true — one measured lot reached €1,300 with its reserve unmet
and sold for nothing), or a starting floor nobody has bid.
A null price is normal: it is a lot whose reserve has not been met, and 57
of 57 blank prices across 13 captures carried reserve_price_set. So there is
no price-coverage threshold anywhere in this repo — the suite and the canary
assert that invariant instead.
bid_count is a floor: the site returns the last ten bids and states no
total.
Known limits, stated rather than discovered
- Pagination is capped at 100 pages by the site. Past the cap it returns
page 100's own lots under HTTP 200 rather than failing, so a run of an
11,681-lot category legitimately holds 2,400 lots — and now says so:
"386 page(s) deeper than the site will address". - A search that matches nothing returns 24 suggested lots reported as
total: 24. That state is detected from the payload's own flag and is not
parsed. - The Scraper API gets 24 of 24 lots with ids, titles and urls and zero
prices: the site fills the money in at hydration and a browserless fetch
gets the pre-hydration HTML. Every row says so viaprice_source. - Chromium cannot authenticate a SOCKS5 proxy and Selenium cannot
authenticate any proxy. 2Captcha's IP-whitelist mode hands out
credential-freehost:portexits, which is what makes a proxy usable in
every engine — and keeps the credential out of your config entirely.
Fixes to code inherited from the family
--fingerprintapplied the user agent, locale, timezone and screen but
droppeddeviceScaleFactor, so a fingerprint stating 1.25 produced a
browser reportingdevicePixelRatio === 1.fingerprint_client.pyread only the exported environment, so a key placed
in.envas documented failed at that one entry point.env_config.apply()refilled an explicitly empty flag from.env, so
--cdp-endpoint ''could not switch the file's value off.
Full detail in CHANGELOG.md and in PR #1.