Repository navigation
v0.1.0 — the family-architecture rebuild
If you used the previous version of this repo, read this first. It is
gone, and what replaces it behaves differently in a way you will notice in
your row count. The old scripts called Polymarket's Gamma API and could walk
~35,000 markets by paging an endpoint. This one scrapes the site's own
pages, as an anonymous visitor is served them — and a Polymarket listing
page renders twenty events and publishes no address for the
twenty-first. A run that returned tens of thousands of rows will now return
about ninety per listing.--mode eventsand more--categoryruns are how
you get volume; the trade is depth and honesty about the source, not
breadth. The CLI, the column names and the output files all changed too.
The first release of this repo as a member of the 2scraper family: the same
row schema, exit codes, run metadata, proxy pool, captcha path and offline
suite as its siblings.
Three modes, one row per market
| mode | fetches | fields per market |
|---|---|---|
markets (default) |
one listing page | 11–12 |
events |
the listing, then every event page under it | 111–115 on the upgraded rows |
event |
one /event/{slug} |
111–115 |
Forty columns, including the order book (best_bid, best_ask, spread,
last_trade_price), the volume windows, the event breadcrumb, and — from an
event page — condition_id and both clob_token_ids, which are what
Polymarket's own CLOB API keys on.
Measured 2026-09-16, two addresses
From a datacentre exit in Finland and, independently, from a bare GitHub
runner on the canary's first dispatch — both with no proxy and no key:
/predictions→ 92–94 markets from 20 events in about 8 seconds;- a listing is ONE page.
?page=2,?_p=2and?offset=20each answer
with the same twenty events, twelve scroll rounds add none, and the site's
own state saystotalCount: 21,585behind a cursor the interface never
spends. A run asked for more reports complete with
stop_reason: listing_has_one_pageand records the site's total in its
sidecar; - the same page went to
curl/8.0byte for byte, so the site does not read
the client, and headless and headful were identical; - no captcha was met at all — which is not the same as none being
possible. The site is behind Cloudflare, and this repo implements
TurnstileTaskProxyless(with the render interception a Challenge page
requires) andRecaptchaV2EnterpriseTaskProxyless; - all three browser engines return identical rows, sidecars and exit codes.
Found by running it
An event page parsed to zero rows when its payload shipped no event object —
a brace inside a single-quoted string in the site's own inlined bootstrap
desynced the parser's brace stack. An event page returned its rails' markets
as its own. --mode events threw away every deep row as a duplicate of a
shallow one. Selenium crashed on its first fetch. All four are fixed, and the
capture that caused the first is a committed fixture.
Full detail in CHANGELOG.md.