v0.2.0 — the audit of 8 October, worked

A third-party audit of 8 October, worked end to end. All four of its findings were real. Investigating the first found a second defect underneath it that the audit had not seen, and the two compounded — one was manufacturing the condition the other mistook for the end of a walk.

If you use --mode events, this release changes what you get and what the sidecar calls it. A walk used to stop at the first event page that produced no markets and still report status: complete with exit 0. Expect row counts to rise sharply, expect complete to start meaning what it says, and note that no_new_products is no longer a stop reason this project emits.

The walk stopped early and called itself complete

Reproduced end to end against the audited commit: one empty event page at position 4 of 20 left 17 event pages never fetched, with status: complete, exit 0, and pages_completed: 4 of 21 in the same sidecar.

The rule came from a sibling repo whose pages are consecutive days, where stopping early genuinely saves fetches. Here pages 2..N are the event URLs page 1 named, capped at --pages - 1 — a finite list, nothing to exhaust, nothing saved. The walk now visits every planned URL and records the empty ones as pages_without_rows.

Every page the site serves was classified as its own 404

Next.js inlines its 404 template into the payload of every page. The not-found check scanned the first 200,000 bytes for that template's wording. Counted on 2026-10-09 across the twenty event pages /predictions named, as the site serves them:

carry the wording	20 of 20
…inside the scanned window	17 of 20, always at ~134,000 bytes
…rendered into the body markup	0 of 5 checked, bogus pages included
misclassified while holding markets	12 of 20, holding 5 to 53 each

empty carries parse: False, so those rows were discarded — which produced exactly the zero-row page that ended the walk above.

A marker on every good page is not a marker, so the text markers are gone rather than narrowed, and the content check moved to the top: a page whose payload names markets is content, whatever templates it also ships. Two measurements shaped the fix — a slug that does not exist answers HTTP 200 (4 of 4), so the docstring promising "a real HTTP 404" was wrong; and digest, which looked like a clean replacement, is on 2 of 10 good pages and was not adopted.

The browser engines escaped this by byte position rather than by design: the template sits at 511,547 in a browser's DOM and 134,033 in the served bytes of the same URL.

Also in this release

• Atomic writes. open(path, "w") truncates before a byte is written, so an interrupted run left a shorter file where a complete one was — measured here, a 204,292-byte out.json came back 28 bytes and invalid. All three writers go through one _atomic now. The mode is the part eight sibling repos get wrong, measured by calling each one: a new file gets what open() would have given it, and a target somebody tightened keeps its own.
• The canary runs all three engines daily, compared on ids and schema rather than on prices, since the runs are minutes apart and a prediction market reprices continuously. Verified before shipping: the same 5 markets, 40 columns and 22 stable fields, with scraped_at the only field that differed.
• The Docker base is pinned by digest, with the tag kept beside it.
• events_on_page_1 was counting the events whose rows were not upgraded by a deeper read — 3 reported where page 1 had named 11, printed beside total_events: 23475.
• A check that failed a fresh clone's first command, found by cloning the published repo and typing what the README says.

Declined, with reasons

• Lock files. They would stop the daily canary testing new releases of the libraries, which is what catches a dependency change the morning it happens. The image is a build artefact and should be reproducible; the canary is a detector and should be exposed.
• A run identifier across the JSON, CSV and sidecar. A failed run writes no sidecar at all, save refuses to overwrite good output with an empty result, and each file is now replaced by an atomic rename. An id would add a second thing to keep consistent.
• A queue, scheduler, storage and webhook layer. A product decision rather than a defect; this is a scraper with a file output contract, and the README says so.

Every change in this release was verified by planting the fault it prevents and requiring the suite to go red on a named check — 22 controls in all. Five of them behaved unexpectedly on the first attempt, and every one of those exposed a hole in the checking rather than in the code — a control that crashed instead of naming a check, three that left the suite green because the guard they targeted could not fail as written, and a crash detector that called every control a crash because a passing run of this suite prints a traceback on purpose.

v0.1.1 — the checklist audit

Correction to v0.1.0's notes. That section was edited after the tag was cut, adding a paragraph about the credential-gated paths. A released section is history — git show v0.1.0:CHANGELOG.md is what a reader can check it against — so it has been restored verbatim and the paragraph moved into this release, where it belongs.

An audit of v0.1.0 against this family's own checklist, done by mutation — breaking one thing at a time and watching whether the offline suite noticed — rather than by re-reading the code. Eight defects this family has shipped before were injected. Six were caught immediately; two were not, and both turned out to be real.

Fixed

• diff_runs.py compared columns that do not exist here. It arrived from a sibling tracking seven fields — none of them a column in this repo's schema — so two runs diffed against each other would have reported "no changes" for ever, on any input, while exiting 0. TRACKED_FIELDS now names what a change is on a prediction market: the price, the order book around it, the volume windows, the state transitions that end a market's life, and the question being re-worded. The suite asserts every name in it is a real column.
• A markets run diffed against an events run reported a collapse that never happened. volume_scope moves between the two while data_source stays flight on both, so volume went from the event's 199,000,000 to the market's 19,000,000 and read as a change. volume_scope now decides "which view" alongside data_source.
• A test that could not fail. Deleting the event-scoping rule outright left the suite green — the fixture generator had trimmed away the very rails the rule exists to scope out. It now keeps two of them on purpose, and the same mutation is caught by ten checks.
• scraper_api_client.py logged "Parsed 70 stor(ies)" through a full live run: a sibling's vocabulary surviving in a shipped file.

Added

• A vocabulary guard — the file-describes-another-site check now also bans the words a copied paragraph keeps after the site's name has been swapped out. Each was counted across the repo before being banned, so words that occur legitimately here are not in the list.
• A check that every field diff_runs.py tracks is a real column, which is what would have caught the first defect above.

Measured

Every credential-gated path was run end to end against this site, because those are the paths nobody runs. All four return the same 70 markets a local browser does:

path	result
Scraper API	$0.0005, 766,857 bytes, 70 markets, 11s
Scraping Browser over CDP	70 markets, identical
Fingerprint API	fetched and applied — user agent, locale en-US, timezone America/New_York
solver	balance call OK; no challenge existed to solve

The Fingerprint API's documented multi-tag example still answers HTTP 400, and this repo's default is the single tag that works.

v0.1.0 — the family-architecture rebuild

If you used the previous version of this repo, read this first. It is
gone, and what replaces it behaves differently in a way you will notice in
your row count. The old scripts called Polymarket's Gamma API and could walk
~35,000 markets by paging an endpoint. This one scrapes the site's own
pages, as an anonymous visitor is served them — and a Polymarket listing
page renders twenty events and publishes no address for the
twenty-first. A run that returned tens of thousands of rows will now return
about ninety per listing. --mode events and more --category runs are how
you get volume; the trade is depth and honesty about the source, not
breadth. The CLI, the column names and the output files all changed too.

The first release of this repo as a member of the 2scraper family: the same
row schema, exit codes, run metadata, proxy pool, captcha path and offline
suite as its siblings.

Three modes, one row per market

mode	fetches	fields per market
markets (default)	one listing page	11–12
events	the listing, then every event page under it	111–115 on the upgraded rows
event	one /event/{slug}	111–115

Forty columns, including the order book (best_bid, best_ask, spread,
last_trade_price), the volume windows, the event breadcrumb, and — from an
event page — condition_id and both clob_token_ids, which are what
Polymarket's own CLOB API keys on.

Measured 2026-09-16, two addresses

From a datacentre exit in Finland and, independently, from a bare GitHub
runner on the canary's first dispatch — both with no proxy and no key:

• /predictions → 92–94 markets from 20 events in about 8 seconds;
• a listing is ONE page. ?page=2, ?_p=2 and ?offset=20 each answer
  with the same twenty events, twelve scroll rounds add none, and the site's
  own state says totalCount: 21,585 behind a cursor the interface never
  spends. A run asked for more reports complete with
  stop_reason: listing_has_one_page and records the site's total in its
  sidecar;
• the same page went to curl/8.0 byte for byte, so the site does not read
  the client, and headless and headful were identical;
• no captcha was met at all — which is not the same as none being
  possible. The site is behind Cloudflare, and this repo implements
  TurnstileTaskProxyless (with the render interception a Challenge page
  requires) and RecaptchaV2EnterpriseTaskProxyless;
• all three browser engines return identical rows, sidecars and exit codes.

Found by running it

An event page parsed to zero rows when its payload shipped no event object —
a brace inside a single-quoted string in the site's own inlined bootstrap
desynced the parser's brace stack. An event page returned its rails' markets
as its own. --mode events threw away every deep row as a duplicate of a
shallow one. Selenium crashed on its first fetch. All four are fixed, and the
capture that caused the first is a committed fixture.

Full detail in CHANGELOG.md.