v0.3.0

Behaviour change: --retries 0 is now a usage error (exit 2). It used
to be accepted and made the page-load loop unreachable, so nothing was
fetched and the engine reported the browser's own blank page as a result:
exit 3 or 4 with a 39-byte "capture", in tens of milliseconds. If a
script of yours passes 0, pass 1: the flag counts ATTEMPTS, so 1 means
"try once and do not retry".

Behaviour change: --pages 0, --concurrency 0, and a negative
--pages, --concurrency, --delay, --retry-delay or
--proxy-block-retries are now usage errors (exit 2) in all three engines,
and a negative --retries/--retry-delay in scraper_api_client.py.

Behaviour change: a page that is served WITH vendor tiles and parses to
zero rows now fails the page with stop_reason: parser_found_nothing:
exit 5 when nothing was gathered, exit 6 after earlier pages, where it used
to be exit 4 ("empty listing") or a complete run. The Playwright and
pyppeteer engines now also pass the navigation's HTTP status to the
classifier, so a refusal with no vendor marker on it under 401/403/429/5xx
is exit 3 rather than a 25-second wait and an exit-4 "empty listing".

Behaviour change: CSV cells that begin with =, +, -, @, tab, CR
or LF are prefixed with an apostrophe so a spreadsheet reads them as text.
JSON is unchanged; the sidecar records csv_cells_escaped.

Behaviour change: the canary FAILS when neither FOODPANDA_PROXY nor
FOODPANDA_CDP_ENDPOINT is set, and on exit 5; it used to skip green.

Fixed

• Output files are replaced atomically. JSON, CSV and the
  <out>.meta.json sidecar are written to a temporary file in the same
  directory, flushed, fsynced and renamed over the old one. A write that
  failed half-way used to truncate the previous good file into invalid
  JSON. A new file gets the mode open() would have given it under the
  current umask, and an existing file keeps its own mode.
• --allow-empty on a complete, genuinely empty listing writes
  status: "complete", not "failed", so diff_runs.py can compare it.
• The credential scan reads what it used to skip. Exclusions are judged
  on the path inside the repository, so a clone under a directory called
  venv no longer scans 0 files and reports clean; a scan that read nothing
  now fails. Untracked .json dumps and .env.* backups are scanned
  (.env itself, the user's own file, is not).
• pyppeteer no longer re-runs a plain load failure for every block
  retry. Only an unusable proxy exit, with a pool to rotate to, earns
  another round — the same rule as the other two engines.
• The Scraping Browser credentials claim is corrected. README,
  TROUBLESHOOTING, .env.example and the canary said a profile "lives about
  a day". That was never a measured constant (one endpoint was still
  accepted ten days after it was set); they now say the credentials expire
  and that HTTP 401 deny_no_user is the signal.
• The Turnstile log no longer calls a user-agent mismatch the likeliest
  cause of a refused token; on this site a token minted under a different
  user agent was accepted.
• .gitignore covers --dump-html page dumps of any name (*.page[0-9]*),
  a root-level HTML dump and scratch directories (live/, out/, …).
• Removed the unused single_page_mode stop reason.
• Comments, docs and release notes no longer cite section numbers of an
  internal document readers cannot open; a suite check keeps them out.
• --retries 0 fetched nothing and called it a result. The load loop is
  range(1, retries + 1), so 0 never runs it and the engine judged
  about:blank. Three third-party audits of this family ran exactly that and
  reported a working scraper as broken. --retries now goes through
  page_flow.attempts in every entry point; a check reads the source of each
  one and calls the validator, so a new engine declaring the flag with a
  plain int fails the suite.
• A challenge that never clears no longer buys a solve per retry.
  SOLVES_PER_PAGE was enforced on only one of the two
  handle_captcha_if_present calls in a fetch attempt, so with no proxy pool
  a persistent challenge bought one solve on every block-retry: 6 solver
  invocations against a limit of 1, reproduced with the solver stubbed. The
  budget (page_flow.SolveBudget) is now spent inside the handler, right
  before the purchase, shared by both call sites and not reset by a rotation.
  All three engines.
• --delay now applies between page 1 and page 2. The pause sat at the
  tail of the loop, behind a "more pages remain" test, so it only ran between
  pages that had both been fetched already and a two-page run never paused.
  It now runs before every navigation after page 1, and once before the
  --concurrency workers start.
• diff_runs.py refuses two runs that read different listings. Both
  runs could be complete with the same country source and a different
  city or area, and the diff then reported the AREA change as vendors
  appearing and disappearing. The sidecar's start_url (host, path, query
  without page) must now agree, and a run that stopped at its own --pages
  limit is refused against a deeper run. --force still overrides, and a
  run with no start_url in its sidecar is let through as before.
• Scraper API: waitFor is now sent as a JSON object. Measured
  2026-09-23 against scraper.2captcha.com/tasks/sync: the JSON-encoded
  string this client sent (the form older docs described) is refused with
  HTTP 422 "params.waitFor must be an object" -- and the task is still
  billed ($0.0005) -- so every run with --wait-text or its sibling wait flags
  failed with exit 5. The object form answers HTTP 200.
• Scraper API: the target page's status is read from http_code. The
  response's status field is the API's own verdict string ("success"),
  not the target site's HTTP code, so a target 403/503 was never seen by
  this client. http_code (an int) is read now, with status kept as a
  fallback only when it is an int.
• scraper_api_client.py --help described two other sites. Its
  description said the pages need JavaScript and a single fetch returns
  about 5 products, and --url told you to wait for a
  lodging-card-responsive element that arrives over GraphQL. Neither is
  true here: foodpanda server-renders its grid, and the docstring's own
  measurement is 48 of 48 vendors from one request. --category no longer
  claims to default to a URL segment — this client does not use it.
• Issue templates described an auction site. The bug-report hints talked
  about lots, reserve prices, a page-100 cap and /en/ locales, and the
  site-change template's anchors were __NEXT_DATA__ and lot cards.
  Rewritten from this repo's README, CONTRIBUTING and TROUBLESHOOTING:
  headless refusal, PerimeterX retries, directory URLs, the (100+) and
  Up to flags, home-page-only delivery columns.
• The pyppeteer and Selenium engines' docstrings listed a --mode product
  and gave a Tokopedia path as their --url example; both now describe the
  one listing mode and a real area listing. Their unreachable
  --mode product sidecar branch (a shop's id and slug, which nothing here
  produces) is gone; the listing branch is unchanged.
• Comments that told another site's story as this one's: /p/<slug> hubs,
  "hub category", a search grid over GraphQL, a keyword=kopi tracking tail
  and a detail page's buy box in the engines; an Akamai 394-byte refusal and
  auction lots in captcha_solver.py; a Tokopedia no-results string, an
  HTTP/2 stream reset, an Orlando search and shop_rating in
  output_writer.py. Replaced with what this repo measured, or attributed
  to the sibling where it happened. captcha_solver.py no longer points at
  a "No DataDome solver" section that does not exist.
• SECURITY.md said this project has no releases or tags; it has both.
• The access badge now says what the README measures: no account needed,
  from a residential IP.
• The README was missing its canary badge. The workflow has been
  running all along; the badge that reports it was never added, so the
  one signal that says whether this scraper still works against the live
  site was invisible above the fold.
• Fifteen lines of unreachable code removed from playwright_scraper.py.
  A function's def line had been lost at some point before this repo's
  first commit, leaving its docstring and its try: return page.content()
  body indented into the end of _mask_credentials, where the control flow
  can never arrive. Nothing called it — _content_when_settled below it
  does the job — so no behaviour changes. The same fifteen lines, byte for
  byte, were in six repos of this family.

Added

• A check for a statement the control flow can never reach. The
  undefined-name walk beside it cannot see this class by design: it pools
  every binding in a file rather than tracking scopes, so a name used inside
  dead code passes as long as anything else in the module binds it. The new
  one is narrow — a statement after a return/raise/break/continue in
  the SAME block — and measured across the eighteen repos of this family it
  found six real problems and zero false positives. Verified by control:
  appending return 1 followed by a statement turns the suite red.

v0.2.1 — the help text was still lying

--help was still lying. 0.2.0 implemented both of this site's
captchas and corrected the README — but not the engines' own text, which is
what a user reads first.

Two engines were still telling readers "NO challenge has ever been observed
on this site … neither setting has anything to act on today, and neither
helps with a refusal". Playwright's said Cloudflare's challenge "is reported
as blocked instead, so no solve is attempted or billed for it". Both describe
a repo that stopped existing in 0.2.0.

Fixed

• Every user-facing claim that a captcha cannot be solved is gone — from
  --help, from four docstrings and from page_flow.block_advice. What
  replaced them is what was measured: reCAPTCHA Enterprise ~55s /
  $0.00299, Turnstile 11s / $0.00145, and PerimeterX's widget-less stub
  as the one refusal nothing can buy.

• --help named a host the parser refuses. .co.th was dropped in 0.1.0
  — it redirects to robinhood.co.th, a different company — and HOSTS has
  held ten entries ever since, while three engines and output_writer.py
  went on saying "eleven country sites" and playwright's --url help listed
  .co.th among the supported ones.

• The README contradicted itself about Cloudflare. One section said a
  managed challenge is what a non-browser client gets and that a browser
  engine seeing one means "something has stripped the context" — three
  sections above the Turnstile section explaining that foodpanda.com serves
  one to a real browser, which is how this repo solved one. Same
  contradiction in TROUBLESHOOTING.md.

Added

• A flag reference in the README. There was none: sixteen real flags were
  documented nowhere but --help. Four tables — core, proxies, 2Captcha, and
  the engine differences — with defaults and what each flag actually does.

• Three checks, because prose does not fail a build on its own: a
  banned-phrase check over every shipped .py and .md; a check that the
  documented host count equals len(HOSTS) and that no user-facing text
  advertises a refused host; and a check that every flag an engine accepts
  appears in the README, that every in-page link resolves, and that each exit
  code is documented.

  Each was verified to fail when its fix is reverted. A check that cannot
  fail is not a check.

532 offline checks, up from 493.

v0.2.0 — the captchas this site serves are solved now

If you read 0.1.0's notes, two claims in them are now false. A 2Captcha
key does clear this site's challenges — both of them — and the canary is
dispatch-only rather than daily.

Everything in this release is about the same thing: what this scraper does
when foodpanda refuses it. 0.1.0 shipped saying a key would not help. That
was wrong in the way that matters most — it reported a limit of this code as
a limit of the product.

Both of this site's challenges are solved now

	task type	measured
reCAPTCHA Enterprise on PerimeterX's denial page	RecaptchaV2EnterpriseTaskProxyless	~55s, $0.00299, page came back with its full grid
Cloudflare Turnstile, Challenge page included	TurnstileTaskProxyless	11s, $0.00145, 837-char token, page became the real site

The enterprise solve has a control, because a solve that coincides with a
block expiring proves nothing: a plain same-session reload cleared the same
block 0 of 8 times. The token is what got in.

The Turnstile half is harder than it looks. A Challenge page publishes no
sitekey at all — Cloudflare calls turnstile.render(container, params) once
and keeps nothing, so cData, chlPageData and action exist only inside
that call. Every engine now installs an interception script on the context
before any page script runs: add_init_script in Playwright,
evaluateOnNewDocument in pyppeteer, CDP
Page.addScriptToEvaluateOnNewDocument in Selenium — the one thing the three
cannot share.

Still unsolvable, and honestly so: PerimeterX's widget-less stub. 4.7 KB,
24 px-captcha references, no widget at all. Nothing there for any solver at
any price, so nothing is attempted or billed for it.

Changed

• A page with a solvable widget is challenge, not blocked. Both of
  this site's refusals carry one now.
• The canary is dispatch-only: no schedule, no badge. This repo runs
  nothing live on a timer, and a green badge for a check that is not
  happening is as unreadable as one that is always red.

Fixed

• A wrong captcha variant, found by spending real money on the first
  attempt — no rung for "challenge frame present, no size, no render",
  so it fell through to v3 and got ERROR_CAPTCHA_UNSOLVABLE after 87
  seconds. v3 renders no challenge frame at all, so a frame present now
  settles the variant.
• The static detector could not see this site's widget — it bailed unless
  the markup contained grecaptcha, and the denial page carries only the
  hyphenated class g-recaptcha. Anyone debugging from a --dump-html
  capture was told there was no captcha at all.
• cf-turnstile was tried as a marker and thrown away. 2Captcha's
  Scraping Browser injects a cf-turnstile-response hunter into every page it
  loads, so the string appears once on a served listing and zero times
  on the real challenge — a marker that fires on good pages and misses the bad
  one. challenges.cloudflare.com is the one that works: 0 on every served
  page, 5 on the challenge.
• A sitekey-less Turnstile detection raises instead of building a task.
  Paying for a request 2Captcha will reject is worse than reporting the page
  unsolved.

Tests

493 offline checks, up from 470, none of which touch the site or a
credential.

v0.1.0 — rewritten on the family architecture

If you used the April 2026 prototype, nothing carries over. Not the
command line, not the row schema, not the exit codes. The three monolithic
scripts are replaced by a parser, a page policy, a shared output contract
and three engines that must agree. Old foodpanda_*.json files are not
comparable with this one's output.

Scrapes foodpanda vendor listings — every restaurant and shop tile on a
country home page, a /city/{city} listing or a /city/{city}/area/{area}
listing — with rating, review count, cuisines, deal labels, delivery data and
whether the vendor is open right now. Ten country sites, one schema, JSON and
CSV.

A real run, from an ordinary residential address with no proxy and no key:

python3 playwright_scraper.py \
  --url "https://www.foodpanda.pk/city/lahore/area/gulberg" \
  --pages 3 --delay 45 --retry-delay 30 --out gulberg

[+] Saved 143 products -> gulberg.json
[+] Wrote run metadata -> gulberg.meta.json (status=complete)

Things worth knowing before you run it

• You do not need to buy anything. Nine of the ten country sites served a
  plain headful Chromium on a residential address. What you need is patience:
  10 of 30 cold navigations were served across a 20-minute sweep, and the
  refusal is per-session — a fresh browser and a slower rate is what clears
  it, which is what --retries does here.
• --headful is the default, unusually for this family. Headless was
  served 0 of 4 times against headful's 4 of 4.
• Do not use --fingerprint on this site. Measured harmful: both
  fingerprinted arms were refused 4 of 4 while both plain arms were served on
  the first attempt — including a fingerprint whose country matched the exit.
  The engine warns at runtime.
• A 2Captcha solver will not get you past the block. foodpanda's refusal
  renders reCAPTCHA Enterprise, which this project does not implement, so
  no solve is attempted and your key is never charged. The other three
  2Captcha paths were each run end to end and all work — the Scraping Browser
  took three pages, 144 vendors, zero refusals, in 29 seconds.
• foodpanda.co.th is not a foodpanda site. It redirects to
  robinhood.co.th, a different company. foodpanda has left Thailand. The
  scraper refuses that host with that reason rather than reporting a block.

Output

One row per vendor. Five family columns are absent because a foodpanda tile
has no price — 0 price nodes across 1,952 tiles on two country sites — and
each absence is a measurement written down in output_writer.py.

Three columns are populated only on the HOME page, which is the one listing
foodpanda renders with a delivery address in play: delivery_time,
delivery_fee and price_level. page_kind says which listing you have.

Read the flags beside the numbers: review_count_is_floor (the site caps at
100+, 500+, 1000+) and discount_is_upper_bound (Up to 35% off is a
ceiling, and 21% of measured labels are that shape).

Verification

457 offline checks, fixtures cut from real captures across three country
sites, all three engines run live, and a Docker image built and run in CI.
Six defects were found by running it rather than by reading it — including a
refused page being reported as an exhausted listing with status: complete
and exit 0. See CHANGELOG.md for the full list.