v0.1.0 — indiegogo.com, three modes, three engines

First release.

Scrapes indiegogo.com project search listings, campaign pages and reward
tiers to JSON or CSV. Playwright is the primary engine, with Selenium and
pyppeteer at flag-for-flag parity, plus an HTTP-only client that needs no
local browser.

pip install -r requirements.txt -r requirements-playwright.txt
playwright install chromium
python3 playwright_scraper.py --pages 3

Do you need a 2Captcha account?

For most of this, no — and that is measured, not assumed.

Every HTML route on indiegogo.com answers an HTTP client with a Cloudflare
managed challenge: HTTP 403 on 8 routes out of 8 from a datacentre
address on 2026-09-17, robots.txt and the site's own JSON API included. A
plain requests call cannot read this site at all.

A real browser can. From that same datacentre address a local headless
Chromium returned a full page of results on 4 of 4 runs with no key, no
proxy and no CDP endpoint — and 2 of those 4 met the challenge first and
cleared it themselves on the next navigation, with 0 solves attempted and
the 2Captcha balance unchanged to five decimal places. The canary's first
dispatch, from a bare GitHub Actions runner with no secret configured,
returned 72 rows across 3/3 pages with the first response already
unchallenged.

The paid products buy volume from many addresses, a specific exit country,
no browser infrastructure to run, and a solver if the challenge ever stops
self-clearing. Cloudflare Turnstile is implemented
(TurnstileTaskProxyless), including the turnstile.render interception a
Challenge page requires — such a page publishes no sitekey anywhere in its
markup, so no static read of the HTML can produce a solvable task.

Modes

mode	what you get
--mode search (default)	one row per campaign, 24 per page
--mode campaign	one campaign page, including its ISO 4217 currency
--mode rewards	one row per reward tier, with prices, stock and delivery

Things about this site worth knowing before you start

• ?page=2 silently returns page 1. The listing has no per-page URL at
  all; real pagination is a 0-based pageIndex in a POST body.
  product_parser.page_url() raises rather than returning a URL, because a
  ?page=N convention here reports a complete run holding 24 rows.
• Every search is capped. The site claims 10,000 matches across 417
  pages, but pageIndex 416 is empty while 415 is full — a real ceiling of
  9,984 rows. A run that fetches everything is genuinely complete and
  also a sample; the sidecar records both.
• Most search results are not Indiegogo campaigns. 110 of 432 sampled
  rows were hosted on gamefound.com. Every row carries a platform column,
  and the mix depends heavily on --sort (a-z, z-a and oldest
  returned zero Gamefound rows across 120 each; default returned 49).
• campaignOutcome is not a success flag — 20 of 60 sampled rows with
  outcome=0 had already passed their goal.
• Zero JSON-LD on four page kinds. Extraction reads the site's own JSON.

See the README for the rest, including why cf-turnstile is deliberately
not used as a challenge marker.

Verification

85 offline checks, green with no engine library installed and again in each
engine's own virtualenv. CI runs Python 3.9 and 3.12, builds and runs the
Docker image, and a daily canary does a real 3-page run.