v0.2.0

Behaviour changes for scripts.

• A search the site answers with its own "nothing matched"
  (search-count:0) now exits 4 (no_results, an empty result) in all
  three browser engines. It exited 5 ("never obtained"), which said the
  opposite of what happened.
• A --concurrency run in which some planned pages were never fetched by
  any worker now ends partial (exit 6), listing them in pages_failed
  and the sidecar's pages_unattempted. It used to report complete and
  exit 0 with those pages missing.
• The Selenium and pyppeteer engines now exit 5 when every navigation
  failed (a dead network, a timeout), like the Playwright engine. They
  reported exit 3 (blocked), sending the reader after a proxy problem.
• A paid captcha solve is now bought at most once per page. With the
  default --retries 2, one gated page used to buy a token on every
  attempt (three solves); the run now stops at exit 3 after one.
• --retries, --delay and --retry-delay below 0, and --pages or
  --concurrency below 1, are now usage errors (exit 2) naming the flag,
  in all engines and scraper_api_client.py. --retries -1 used to run
  no navigation at all and report exit 3. --retries 0 (try once) is
  unchanged.
• Output files are written to a temporary file in the same directory and
  renamed into place. A new file gets the mode open() would have given it
  under your umask; an existing file keeps its mode.

Fixed

• An interrupted write (a crash, a kill, a full disk) no longer destroys the
  previous good output: out.json, out.csv and out.meta.json are each
  written atomically. Measured before the fix: an 852-byte out.json came
  back as 12 bytes of invalid JSON.
• In the Selenium and pyppeteer engines, rotating to the next proxy exit now
  relaunches the browser. --proxy-server is fixed at launch, so the old
  code advanced the pool and kept using the first exit. Both engines also
  rotate on a blocked page 1 when a pool is given, as Playwright does.
• diff_runs.py now has the source_changed bucket its documentation
  described: a price difference that comes with a price_source difference
  (the API's exact figure against a card's rounded one) is reported there and
  does not trip --fail-on-change.
• The credential scan in .github/ci_checks.py tested every ancestor of the
  checkout against its skip list, so a clone under a directory named venv
  scanned 0 files and reported clean. It now tests paths relative to the
  repository, fails when it scanned nothing, and also reads .json, .csv,
  .toml, .cfg and .sh files.
• CONTRIBUTING.md, the bug-report and site-change issue templates and the
  review workflow's prompt now describe Indiegogo (modes search,
  campaign, rewards, the inlined search payload and the
  searchProjectsForCards POST) instead of another site's pages, flags and
  modes.
• The exit-code tables in README.md and TROUBLESHOOTING.md now say what exit
  4 and 5 mean in the code: 4 is "the site answered and nothing matched", 5
  is "the content was never obtained", including a served page that parsed
  to nothing (parser_found_nothing).
• The help text and docs no longer say Scraping Browser credentials last
  "about a day"; how long one lasts varies, and a 401 is the signal to fetch
  a fresh endpoint.
• --proxy-block-retries is accepted for compatibility with the other
  scrapers in this family; its help now says it has no effect here.
• The daily canary was red from 2026-10-07 to 2026-10-12 although the run
  itself was healthy (72 rows, complete, 3 of 3 pages). Its sanity check
  rejected pct_funded <= 0, and a real campaign that raised 10,000 against
  a goal of 900,000,000 rounds to 0.0. The check now fails on a negative
  percentage, or one computed from a goal that is not positive; zero is a
  figure. The parser was right and is unchanged, with a check pinning the case.
• Scraper API: waitFor is now sent as a JSON object. Measured
  2026-09-23 against scraper.2captcha.com/tasks/sync: the JSON-encoded
  string this client sent (the form older docs described) is refused with
  HTTP 422 "params.waitFor must be an object" -- and the task is still
  billed ($0.0005) -- so every run with --wait-text or its sibling wait flags
  failed with exit 5. The object form answers HTTP 200.
• Scraper API: the target page's status is read from http_code. The
  response's status field is the API's own verdict string ("success"),
  not the target site's HTTP code, so a target 403/503 was never seen by
  this client. http_code (an int) is read now, with status kept as a
  fallback only when it is an int.
• A rendered enterprise widget went undetected. The iframe rung matched
  recaptcha/api2/anchor; this site serves recaptcha/enterprise/anchor,
  where the sitekey can be only in the frame's k= parameter.
• A sitekey inside the site's own <captcha-*> mount went undetected.
• category is localised and differed BETWEEN MODES. The same German
  campaign read Produktivität from --mode search (the API's localised
  name) and Productivity from --mode campaign (a numeric code resolved
  through an English table).
• pyproject extras had drifted from requirements-*.txt
  (playwright>=1.44 against >=1.40.0, selenium>=4.20 against
  >=4.15.0). CI installs the requirements files, so the drift was invisible.
• The offline suite's credential check scanned gitignored files and
  failed on a --dump-html capture containing a site nonce — a file that
  cannot be committed. .github/ci_checks.py had already learned this;
  the suite now matches it.
• .gitignore did not cover this repo's own default output prefixes.
• Leftovers from the repo this one was bootstrapped from. .gitignore
  listed transfermarkt-scraper's transfermarkt_products.* output prefixes
  (this repo's own indiegogo_projects.* was already listed) and described
  sample_output.* as a --mode market-values run; it is --mode search.
  tools/waf_probe.sh and tools/verify_browser_api.sh were transfermarkt's
  AWS WAF probes — transfermarkt URLs, TRANSFERMARKT_* variables that this
  repo's loader does not read, and a --mode market-values this repo does
  not have. Nothing referenced either, and this site's gate is a Cloudflare
  managed challenge rather than AWS WAF, so they are removed rather than
  ported.

Added

• CSV output prefixes any text cell that begins with =, +, -, @, a
  tab or a line break with an apostrophe, so a spreadsheet does not run it as
  a formula. Numbers are untouched, JSON keeps the site's text, and the
  sidecar's csv_cells_escaped counts the cells that differ.
• category_code on Campaign — the site's numeric, locale-independent
  catalogue id. Join on it; category is display text.
• Checks pinning all of the above, each verified to fail when its fix is
  reverted.

Changed

• The daily canary no longer skips green when the runner is blocked (exit
  3): the README's claim is that a local browser needs no key and no proxy
  here, and bare runners have been served on every scheduled run checked, so
  a block is now a failure. Its exit 4 and 5 messages match the codes above.
• scraper_api_client.py warns on stderr when the key is passed with --key
  (visible to ps, kept in shell history). TWOCAPTCHA_KEY in .env is
  unchanged and remains the documented way.

Correction to v0.1.0's release notes. They said the 2Captcha balance
was "unchanged to five decimal places". That was measured across the four
bare-browser runs and is true of them, but it is not true of the session as
a whole: the balance moved $0.0024 over the day, which is Scraping
Browser session usage from --cdp-endpoint runs. The claim that matters is
narrower and now stated that way — 2Captcha's own statistics endpoint
reports 0 captcha solves and $0.00000 for the day. A run without
--cdp-endpoint still costs nothing at all.

The site has a SECOND captcha, configured but never rendered

Asked plainly which captcha this site uses, and answered by measuring. There
are two, at different layers:

1. Cloudflare Turnstile, in a managed challenge at the edge. This is the
   one v0.1.0 documents: met, self-cleared, never solved.
2. reCAPTCHA Enterprise, wired into the application and enabled on every
   page the site serves — site key 6LeRruUr…, the enterprise loader on
   recaptcha.net (not google.com), and an empty <captcha-widgets> mount
   point. It is never shown to an anonymous reader.

"No challenge rendered" is not "no captcha configured", so the useful
question was whether this repo would RECOGNISE it. Constructing the shapes
this site's own captcha would take found two the static detector missed.

The proxy path, now exercised live

v0.1.0 shipped with --proxy untested against a real gateway -- there was
no credential to test with. There is now, and the path works, but testing it
found a defect first.

• Fixed: a pasted proxy-LIST line exited 1 (crash) instead of 2 (bad
  usage). Proxy lists are scheme://host:port:login:password; this
  expects http://login:password@host:port, and the extra colons land in
  the port. proxy_pool already validated it and raised with a message
  naming the fix -- but no engine caught ProxyError at its entry point, so
  it reached the interpreter as a traceback. proxy_pool's own comment said
  this should be exit 2; now it is, in all three engines, with the password
  absent from the message.

Measured 2026-09-17 through a 2Captcha residential gateway:

• The site served content on the FIRST response, with no challenge --
  better than the datacentre path, which met one on 2 of 4 runs.
• -region-XX is honoured: de → Munich (M-net), us → Reston (AT&T),
  gb → Glossop (Plusnet), tr → Istanbul (Türk...

v0.1.0 — indiegogo.com, three modes, three engines

First release.

Scrapes indiegogo.com project search listings, campaign pages and reward
tiers to JSON or CSV. Playwright is the primary engine, with Selenium and
pyppeteer at flag-for-flag parity, plus an HTTP-only client that needs no
local browser.

pip install -r requirements.txt -r requirements-playwright.txt
playwright install chromium
python3 playwright_scraper.py --pages 3

Do you need a 2Captcha account?

For most of this, no — and that is measured, not assumed.

Every HTML route on indiegogo.com answers an HTTP client with a Cloudflare
managed challenge: HTTP 403 on 8 routes out of 8 from a datacentre
address on 2026-09-17, robots.txt and the site's own JSON API included. A
plain requests call cannot read this site at all.

A real browser can. From that same datacentre address a local headless
Chromium returned a full page of results on 4 of 4 runs with no key, no
proxy and no CDP endpoint — and 2 of those 4 met the challenge first and
cleared it themselves on the next navigation, with 0 solves attempted and
the 2Captcha balance unchanged to five decimal places. The canary's first
dispatch, from a bare GitHub Actions runner with no secret configured,
returned 72 rows across 3/3 pages with the first response already
unchallenged.

The paid products buy volume from many addresses, a specific exit country,
no browser infrastructure to run, and a solver if the challenge ever stops
self-clearing. Cloudflare Turnstile is implemented
(TurnstileTaskProxyless), including the turnstile.render interception a
Challenge page requires — such a page publishes no sitekey anywhere in its
markup, so no static read of the HTML can produce a solvable task.

Modes

mode	what you get
--mode search (default)	one row per campaign, 24 per page
--mode campaign	one campaign page, including its ISO 4217 currency
--mode rewards	one row per reward tier, with prices, stock and delivery

Things about this site worth knowing before you start

• ?page=2 silently returns page 1. The listing has no per-page URL at
  all; real pagination is a 0-based pageIndex in a POST body.
  product_parser.page_url() raises rather than returning a URL, because a
  ?page=N convention here reports a complete run holding 24 rows.
• Every search is capped. The site claims 10,000 matches across 417
  pages, but pageIndex 416 is empty while 415 is full — a real ceiling of
  9,984 rows. A run that fetches everything is genuinely complete and
  also a sample; the sidecar records both.
• Most search results are not Indiegogo campaigns. 110 of 432 sampled
  rows were hosted on gamefound.com. Every row carries a platform column,
  and the mix depends heavily on --sort (a-z, z-a and oldest
  returned zero Gamefound rows across 120 each; default returned 49).
• campaignOutcome is not a success flag — 20 of 60 sampled rows with
  outcome=0 had already passed their goal.
• Zero JSON-LD on four page kinds. Extraction reads the site's own JSON.

See the README for the rest, including why cf-turnstile is deliberately
not used as a challenge marker.

Verification

85 offline checks, green with no engine library installed and again in each
engine's own virtualenv. CI runs Python 3.9 and 3.12, builds and runs the
Docker image, and a daily canary does a real 3-page run.