Skip to content

Releases: 2scraper/catawiki-scraper

v0.1.0 — Catawiki auction scraper

Choose a tag to compare

@jehrr jehrr released this 11 Sep 05:21
efd4a09

First release. The original generated scripts are replaced by the
2scraper family architecture: one row schema,
one set of exit codes, one page_flow policy shared by three engines, 465
offline checks, and every engine and paid path run live against the site.

Read this first if you are used to the sibling repos: --headful is the
default here.
Catawiki refuses a headless browser regardless of the exit
address — HTTP 403 and a 394-byte "Access Denied" from four residential
exits and one datacentre address, against HTTP 200 and the full catalogue
from those very same addresses with a real window. A proxy does not fix it,
--headless will report exit 3, and page_flow.RETRY_ON_BLOCKED is False
because rotating an exit spends the retry budget on a change that cannot
help.

What it reads

  • --mode listing — a category (/{loc}/c/{id}-{slug}), a search
    (/{loc}/s?q=) or a whole auction (/{loc}/a/{id}-{slug}). An auction page
    is the best source of the three: it carries every one of its lots in one
    response (130 of 130 on the page measured) plus the auction's absolute
    close time, which a category listing states nowhere.
  • --mode lot — one lot in full: the expert's estimate, the seller and
    their feedback score, the specifications, the bid-history depth and both
    absolute bidding times.
  • --mode auctions — the auctions index as a work list, one row per
    auction.

Four back ends behind one schema: Playwright (primary), pyppeteer, Selenium,
and the 2Captcha Scraper API. Verified live on the same URL — 48 rows each,
identical sku sets, 44 identical columns in the same order.

What an auction row means

price is three different quantities and bid_kind says which: a live
current bid, a final bid on a closed lot (a hammer price only when
sold is also true — one measured lot reached €1,300 with its reserve unmet
and sold for nothing), or a starting floor nobody has bid.

A null price is normal: it is a lot whose reserve has not been met, and 57
of 57 blank prices across 13 captures carried reserve_price_set. So there is
no price-coverage threshold anywhere in this repo — the suite and the canary
assert that invariant instead.

bid_count is a floor: the site returns the last ten bids and states no
total.

Known limits, stated rather than discovered

  • Pagination is capped at 100 pages by the site. Past the cap it returns
    page 100's own lots under HTTP 200 rather than failing, so a run of an
    11,681-lot category legitimately holds 2,400 lots — and now says so:
    "386 page(s) deeper than the site will address".
  • A search that matches nothing returns 24 suggested lots reported as
    total: 24. That state is detected from the payload's own flag and is not
    parsed.
  • The Scraper API gets 24 of 24 lots with ids, titles and urls and zero
    prices
    : the site fills the money in at hydration and a browserless fetch
    gets the pre-hydration HTML. Every row says so via price_source.
  • Chromium cannot authenticate a SOCKS5 proxy and Selenium cannot
    authenticate any proxy. 2Captcha's IP-whitelist mode hands out
    credential-free host:port exits, which is what makes a proxy usable in
    every engine — and keeps the credential out of your config entirely.

Fixes to code inherited from the family

  • --fingerprint applied the user agent, locale, timezone and screen but
    dropped deviceScaleFactor, so a fingerprint stating 1.25 produced a
    browser reporting devicePixelRatio === 1.
  • fingerprint_client.py read only the exported environment, so a key placed
    in .env as documented failed at that one entry point.
  • env_config.apply() refilled an explicitly empty flag from .env, so
    --cdp-endpoint '' could not switch the file's value off.

Full detail in CHANGELOG.md and in PR #1.