scrapequeue

Polite, resumable URL queues for bounded Python scraping jobs.

scrapequeue is for developers who already know what URLs they want to fetch and want reliable queue management without adopting a crawler framework. You provide a fetch(url) function; scrapequeue handles URL ingestion, dedupe, rate limiting, retries, SQLite checkpointing, resume, failed URL inspection, and CSV export.

Why scrapequeue exists

A lot of practical scraping jobs are not full crawlers. You have a CSV of URLs, a custom fetch function, and a painful loop that slowly grows retries, sleeps, checkpoints, failure logs, and export code.

scrapequeue keeps that boring operational layer in one small library:

from scrapequeue import Queue

queue = Queue(rate_limit="1/s", retries=3, checkpoint="jobs.sqlite")
queue.add_urls("urls.csv")
queue.run(fetch, progress=True)
queue.export_csv("results.csv")

Built from community pain signals

This package started from a public developer-pain discovery sprint: repeated posts and issues around scraping thousands of URLs, handling rate limits, restarting failed jobs, and avoiding heavyweight crawler setup for bounded URL lists.

The goal is deliberately narrow: queue management for known URLs, not a new scraping framework.

When to use it

Use scrapequeue when you have:

• hundreds to low-hundreds-of-thousands of known URLs
• a script/notebook that keeps growing retry/checkpoint code
• a need to stop/restart jobs without re-fetching successes
• a custom fetcher using httpx, requests, Playwright, or another client

When not to use it

Use something else when you need:

• link discovery / crawling / spider frontiers
• distributed queues across machines
• proxy rotation, CAPTCHA solving, or anti-bot evasion
• HTML parsing helpers baked into the queue

For those, consider Scrapy/Crawlee/Playwright directly. scrapequeue intentionally stays smaller.

Install

pip install scrapequeue

For local development:

pip install -e '.[dev]'

Five-line quickstart

from scrapequeue import Queue

queue = Queue(rate_limit="1/s", retries=3, checkpoint="jobs.sqlite")
queue.add_urls(["https://example.com/a", "https://example.com/b"])
queue.run(lambda url: {"url": url, "body": "..."})
queue.export_csv("results.csv")

Real fetch function

import httpx
from scrapequeue import Queue

queue = Queue(rate_limit="30/m", retries=3, checkpoint="jobs.sqlite")
queue.add_urls("urls.csv")  # requires a `url` column


def fetch(url: str) -> dict:
    response = httpx.get(url, timeout=10, follow_redirects=True)
    response.raise_for_status()
    return {
        "status": response.status_code,
        "content_type": response.headers.get("content-type", ""),
        "body": response.text,
    }


queue.run(fetch, progress=True)
queue.export_csv("results.csv")
queue.export_failed("failed.csv")

Input sources

queue.add_urls(["https://a.com", "https://b.com"])
queue.add_urls("urls.txt")
queue.add_urls("urls.csv")  # CSV must contain a `url` column

Duplicate URLs are ignored, so re-adding the same source is safe.

State machine

Every URL is persisted in SQLite with one of these states:

pending -> in_progress -> success
pending -> in_progress -> retrying -> pending
pending -> in_progress -> failed

On startup, stranded in_progress / retrying rows are reset to pending so interrupted jobs can resume.

Retry failed URLs

queue.export_failed("failed.csv")
queue.retry_failed()
queue.run(fetch, only_failed=True)

This lets you inspect failures, fix your fetcher/network issue, then rerun only failed URLs.

Backoff and progress

queue = Queue(backoff="exponential", backoff_seconds=0.5)
queue.run(fetch, progress=True)

Backoff modes:

• fixed
• linear default
• exponential
• jitter

Progress can also be a callback:

events = []
queue.run(fetch, progress=events.append)

CLI

scrapequeue add urls.csv --checkpoint jobs.sqlite
scrapequeue status --checkpoint jobs.sqlite
scrapequeue run ./fetcher.py:fetch --checkpoint jobs.sqlite --retries 3 --rate-limit 1/s
scrapequeue run ./fetcher.py:fetch --checkpoint jobs.sqlite --only-failed --progress
scrapequeue export results.csv --checkpoint jobs.sqlite
scrapequeue export-failed failed.csv --checkpoint jobs.sqlite
scrapequeue retry-failed --checkpoint jobs.sqlite

run accepts a fetch callable as either:

module:function
/path/to/file.py:function

Why not Scrapy?

Scrapy is excellent when you need spiders, crawling, middleware, item pipelines, downloader settings, and project structure.

scrapequeue is for the smaller job:

“I already have the URLs. I just need to fetch them politely, survive failures, resume after interruption, and export results.”

That boundary keeps the API small and makes it easy to drop into a script or notebook.

Public API

Queue(...)

Queue(
    checkpoint="jobs.sqlite",
    retries=3,
    rate_limit="1/s",
    backoff_seconds=0.25,
    backoff="linear",
)

Methods

queue.add_urls(urls)
queue.run(fetch, only_failed=False, progress=None)
queue.export_csv(path)
queue.export_failed(path)
queue.retry_failed()
queue.counts()

Comparison

Tool	Best for	Difference
Scrapy	full crawlers/spiders	much larger framework; scrapequeue is bounded URL queue only
Crawlee	browser/crawling automation	scrapequeue avoids crawler abstraction and external complexity
tenacity	retrying functions	scrapequeue adds URL persistence, resume, export, and state tracking
requests-cache	HTTP caching	scrapequeue is queue/checkpoint oriented, not response caching

Examples

See:

examples/basic.py
examples/httpx_fetch.py
examples/playwright_fetch.py
examples/urls.csv

Contributor-friendly starter issues

Good first areas for contributors:

• more example fetchers
• docs improvements
• export_jsonl()
• failed URL inspection helpers
• per-domain rate limit design discussion

Development

pytest -q
ruff check .
ruff format --check .
python -m build
twine check dist/*

Non-goals

• HTML parsing
• browser automation abstraction
• distributed queues
• proxy rotation / anti-bot evasion
• crawling/link following

License

MIT