Bulk URL Scraper in Python — Async, Rate-Limited, Fault-Tolerant

Scrape hundreds of URLs concurrently with asyncio + a scraping API: bounded concurrency, automatic retries, partial-failure tolerance, results streamed to disk as they land. The boring production details, done.

The core pattern

import asyncio, httpx

API = "https://app.quantumproxies.io/api/v1/scraper/extract"
SEM = asyncio.Semaphore(10)          # concurrency cap

async def scrape(client, url):
    async with SEM:
        for attempt in range(3):
            try:
                r = await client.post(API, json={"url": url, "format": "markdown"}, timeout=120)
                r.raise_for_status()
                return url, r.json()["content"]
            except Exception:
                await asyncio.sleep(2 ** attempt)
        return url, None             # keep going — one dead URL shouldn't kill the batch

async def main(urls):
    headers = {"Authorization": "Bearer qp_live_YOUR_KEY"}
    async with httpx.AsyncClient(headers=headers) as client:
        return await asyncio.gather(*(scrape(client, u) for u in urls))

Why the pieces matter

• Semaphore, not gather alone — unbounded fan-out trips rate limits and exhausts sockets. 10 in flight is a sane default.
• Retry with backoff — transient failures are normal at scale; three attempts recovers almost all of them.
• Return None, don't raise — a 500-URL batch with 3 dead links should produce 497 files, not a stack trace.
• Write as you go — the full script streams results to out/ so a crash at URL 400 loses nothing.

The API side handles the genuinely hard part (JS rendering, bot walls, IP rotation through a residential pool), so the client stays ~60 lines: bulk_scrape.py.

pip install httpx
QP_API_KEY=qp_live_... python bulk_scrape.py urls.txt

Backend: the QuantumProxies Scraper API — per-successful-request billing, so failed fetches in your batch don't cost anything. Keys at app.quantumproxies.io/api-keys.