Skip to content

v0.1.0 — the family-architecture rebuild

Choose a tag to compare

@jehrr jehrr released this 16 Sep 18:44
· 20 commits to main since this release
6edee2c

If you used the previous version of this repo, read this first. It is
gone, and what replaces it behaves differently in a way you will notice in
your row count. The old scripts called Polymarket's Gamma API and could walk
~35,000 markets by paging an endpoint. This one scrapes the site's own
pages, as an anonymous visitor is served them — and a Polymarket listing
page renders twenty events and publishes no address for the
twenty-first. A run that returned tens of thousands of rows will now return
about ninety per listing. --mode events and more --category runs are how
you get volume; the trade is depth and honesty about the source, not
breadth. The CLI, the column names and the output files all changed too.

The first release of this repo as a member of the 2scraper family: the same
row schema, exit codes, run metadata, proxy pool, captcha path and offline
suite as its siblings.

Three modes, one row per market

mode fetches fields per market
markets (default) one listing page 11–12
events the listing, then every event page under it 111–115 on the upgraded rows
event one /event/{slug} 111–115

Forty columns, including the order book (best_bid, best_ask, spread,
last_trade_price), the volume windows, the event breadcrumb, and — from an
event page — condition_id and both clob_token_ids, which are what
Polymarket's own CLOB API keys on.

Measured 2026-09-16, two addresses

From a datacentre exit in Finland and, independently, from a bare GitHub
runner on the canary's first dispatch — both with no proxy and no key:

  • /predictions → 92–94 markets from 20 events in about 8 seconds;
  • a listing is ONE page. ?page=2, ?_p=2 and ?offset=20 each answer
    with the same twenty events, twelve scroll rounds add none, and the site's
    own state says totalCount: 21,585 behind a cursor the interface never
    spends. A run asked for more reports complete with
    stop_reason: listing_has_one_page and records the site's total in its
    sidecar;
  • the same page went to curl/8.0 byte for byte, so the site does not read
    the client, and headless and headful were identical;
  • no captcha was met at all — which is not the same as none being
    possible. The site is behind Cloudflare, and this repo implements
    TurnstileTaskProxyless (with the render interception a Challenge page
    requires) and RecaptchaV2EnterpriseTaskProxyless;
  • all three browser engines return identical rows, sidecars and exit codes.

Found by running it

An event page parsed to zero rows when its payload shipped no event object —
a brace inside a single-quoted string in the site's own inlined bootstrap
desynced the parser's brace stack. An event page returned its rails' markets
as its own. --mode events threw away every deep row as a duplicate of a
shallow one. Selenium crashed on its first fetch. All four are fixed, and the
capture that caused the first is a committed fixture.

Full detail in CHANGELOG.md.