Skip to content

v0.1.0 — rewritten on the family architecture

Choose a tag to compare

@jehrr jehrr released this 16 Sep 10:17
· 10 commits to main since this release
35b095d

If you used the previous scripts, nothing you have will keep working.
quora_scraper_playwright.py, quora_scraper_selenium.py and
quora_scraper_puppeteer.js are gone. The replacements take different
flags, write a different (stable) schema, and return exit codes a pipeline
can branch on. Start from the README rather than from your old command
line.

Scrapes Quora answers — off a topic feed, a question page or a profile —
into JSON or CSV with a stable 30-column schema, a run-metadata sidecar, and
exit codes that tell "blocked" from "empty" from "partial". Works on all
twenty-four Quora language sites.

What to know before your first run

Point it at a question or a profile, not a topic, if you want the counts.
Quora renders nothing on the server; what it inlines into the page is its own
GraphQL results, and those carry the numeric ids, the full answer text,
upvotes, views and creation times. A question page and a profile have them. A
topic page has zero — its answers arrive over a later XHR — so those
columns are null on every topic row. The data_source column on each row
says which view built it.

--pages counts scroll batches, not addresses. Quora publishes no
per-page URL in any mode, and ?page=2 does not fail — it is ignored and the
feed returns its first items again. --concurrency above 1 is refused with
that reason.

A short run is usually a short session, not a bug. Four loads of one
question URL gave 5, 13, 12 and 259 answers; in the short ones the scroll
reached the document's end and the site never fetched more. Re-run it — the
run warns rather than reporting a short result as complete.

No paid product is needed to get data. Playwright's own bundled Chromium,
from a datacentre address with no key and no proxy, was served the full feed
on eight of its first fourteen fetches. What limits a long run is the
address's request RATE, not the browser and not the address type.

A captcha solve buys nothing here. Quora's refusal is Cloudflare's
managed challenge, which publishes no sitekey — 0 data-sitekey
attributes, 0 Turnstile iframes. Nothing is ever attempted or charged.

Included

Four engines behind one schema — Playwright (primary), Selenium, pyppeteer,
and a 2Captcha Scraper API client, which is the richest path per row on a
question URL (6 answers, 6 of 6 payload-backed, $0.0005) and returns nothing
at all on a topic one.

diff_runs.py keyed on sku, a proxy pool with credential masking and argv
safety, .env loading, fingerprint support, a Docker image built and
exercised in CI, and 562 offline checks that pass with no engine library
installed.

Every number in the README, the CHANGELOG and the code comments is measured,
with the date it was measured on. Two paths are marked unverified —
proxies and the Scraping Browser — with the protocol-level evidence for why
in CHANGELOG.md, rather than being quietly claimed.

Full detail: CHANGELOG.md.