reqstorm 2.5.0

• concurrency="auto" finds how many requests at once a server handles well, like TCP congestion control: it starts at 8, doubles while responses stay healthy, backs off on 429/5xx, timeouts or slowdowns, and stops just below the level that caused pushback. Against a server that rejects anything over 12 at a time it settled at 12 with 1 % extra attempts, where a fixed 100 needed 46 % more. max_concurrency caps it; the CLI takes -c auto.
• Open file limit: a concurrency that needs more connections than ulimit -n allows no longer fails with "Too many open files": the limit is raised if the system permits, otherwise the concurrency is lowered with a warning.
• The progress line shows active 12/16 (in flight / allowed) with automatic concurrency.
• Fixed: the built-in TimeoutError is retried on Python 3.9 and 3.10.

Guide: https://reqstorm.github.io/guide/limits/#finding-the-concurrency-automatically-concurrencyauto

reqstorm 2.4.3

• Fixed: applications that never configure logging no longer see reqstorm's warnings and errors on stderr; the reqstorm logger now has a NullHandler.
• Docs: writing JSON into typed columns is easy to find: Schemas: JSON to typed columns, with a comparison and an example at the top of Writing to databases.

reqstorm 2.4.2

• Docker image: docker run --rm -v "$PWD:/data" ghcr.io/melihcolpan/reqstorm urls.txt -o results.jsonl (linux/amd64 and linux/arm64).
• Homebrew: brew install melihcolpan/tap/reqstorm on macOS and Linux.
• Installation docs show every way to install, as a library or as a command.
• Clearer description, keywords and classifiers on PyPI.

reqstorm 2.4.1

Fixed: a SOCKS proxy that cannot be reached, times out or refuses a connection is now retried like an HTTP proxy failure. With retries and a proxy pool, the next attempt goes through the next proxy instead of the request failing on the first one (aiohttp-socks raises its own exception classes, which were not recognised).

Homebrew: brew install melihcolpan/tap/reqstorm

reqstorm 2.4.0

See what a long batch is doing while it runs.

• Logging: each retry with its reason and wait, hosts paused after a 429, token refreshes, retry rounds, rejected records and final failures; every attempt at DEBUG. Without options it follows your application's logging setup through the "reqstorm" logger.
• Per-run logging, independent of the application: log_level="INFO", log_file="logs/", log_format="json". Log files are never overwritten: a directory or {time} gives a time-stamped file per run, and an existing run.log becomes run-2.log.
• A progress line that keeps moving: refreshed on a timer, with requests in flight, retries, the rate over the last minute, retry rounds, paused hosts and the time left. progress= also takes a function (ProgressInfo), stream() supports it, and total= gives generators a percentage.
• Read the output during the run: results are written at least once a second (flush_interval), never with half a line, and SQLite files use WAL mode so they can be read meanwhile.
• Command line: -v / -vv, -q, --log-file, --log-json, --flush-interval; --help now explains every option with its default, examples, exit codes and a link to the guide.

Guide: https://reqstorm.github.io/guide/observability/

reqstorm 2.3.0

• SOCKS proxies: proxy= and Request(proxy=...) accept socks5://, socks5h://, socks4:// and socks4a:// URLs, with credentials. The h / a variants let the proxy resolve host names (Tor, internal networks). SOCKS and HTTP proxies can be mixed in one pool. Install with pip install "reqstorm[socks]".
• Capped retry backoff: max_backoff (default 30 s). The wait between retries used to double without limit; retries=10 waited 256 s before the last attempt.
• Jitter: waits are now a random time between half and all of the computed delay (jitter=True by default), so requests that failed together do not retry together.
• Retry-After is still followed as given.
• Command line: --max-backoff, --no-jitter, SOCKS URLs for --proxy.

Behaviour change: default waits between retries are now capped and jittered. Pass max_backoff=float("inf"), jitter=False for the previous behaviour.

Guides: https://reqstorm.github.io/guide/retries/ · https://reqstorm.github.io/guide/auth/#socks-proxies

reqstorm 2.2.0

Nine additions for long batch jobs against real APIs. No breaking changes.

• rate_limit="auto": follows the server. Slows down on 429 + Retry-After, spreads requests by X-RateLimit-Remaining / Reset, speeds back up when the server stops pushing back.
• Pagination: paginate=NextLink("links.next"), LinkHeader(), Cursor(...) or PageNumber(...); works with schemas, so every page becomes typed rows.
• Command line: reqstorm urls.txt -o results.jsonl --rate 100/min --retries 2, plus --estimate, --template, --paginate, --schema, --cache, --proxy, --report.
• Token refresh: auth=BearerAuth(token, refresh=get_token) renews an expired token on 401 and resends; concurrent 401s share one refresh.
• Requests from data: from_template(url, rows) with read_csv / read_sql.
• Run report: results.report() with p50/p90/p95/p99, statuses, errors and per-host figures; also in summary() and Summary.report.
• Caching: Cache("responses.sqlite", ttl=...) revalidates with If-None-Match; unchanged resources cost a 304.
• Proxies: one URL, a rotating list, or Request(proxy=...).
• Pydantic: pass a Pydantic 2 model as schema= (pip install "reqstorm[pydantic]").

Guides: https://reqstorm.github.io

reqstorm 2.1.0

Structured data: JSON responses to typed columns

Give a schema and each JSON field goes to its own typed column, checked before it is written:

from reqstorm import Field

schema = {
    "id": Field("id", int, required=True, key=True),
    "price": Field("pricing.amount", float),
    "tags": Field("tags", "json"),
    "page": Field("$.meta.page", int),
}
reqstorm.fetch_to_db_sync(urls, connection, table="products", schema=schema,
                          explode="items", rejects_table="products_rejects")

• Typed columns: int, float, str, bool, "datetime" and "json" become BIGINT, DOUBLE PRECISION, TEXT, BOOLEAN, TIMESTAMPTZ and JSONB in PostgreSQL, with equivalents in MySQL and SQLite.
• Strict checks: NaN and Infinity are always rejected. coerce=True accepts compatible values such as "12.5".
• Nested JSON: dotted paths, "json" columns, and explode for one row per array element.
• Rejected records: never written; listed with their reasons in Summary.errors and the optional rejects table.
• Upserts and resume: upserts on key=True fields; resume through the source columns.
• Same schema for files: works for SQLite files, JSON Lines and CSV.
• New helpers: reqstorm.extract(results, schema) and reqstorm.infer_schema(samples).

Guide: https://reqstorm.github.io/guide/structured-data/

reqstorm 2.0.1

reqstorm is the new name of reqt. This first reqstorm release has the same features as reqt 2.0.0, under a name that is easier to find. Another project has used "reqt" since 2013.

$ python -m pip install reqstorm

import reqstorm

print(reqstorm.estimate(7000, rate_limit="100/min"))   # 7000 requests: about 1h 10m (limited by rate_limit)
results = reqstorm.fetch_all_sync(urls, rate_limit="100/min", retries=2, progress=True)
print(results.summary())

• Documentation: https://reqstorm.github.io
• Coming from reqt: replace import reqt with import reqstorm. The reqt package now installs reqstorm and re-exports it with a deprecation warning, so existing code keeps working.
• What reqstorm does: per-host rate limits in any unit, retries and retry rounds, results with attempt history and reports, output to JSONL, CSV, SQLite, PostgreSQL or MySQL with resume, a blocking API for scripts and notebooks, and progress with an ETA. See the changelog.

reqt 2.0.0

reqt 2.0 sends large numbers of HTTP requests concurrently and gives you one result per request, with rate limits, retries, progress and output to files or databases built in.

import reqt

print(reqt.estimate(7000, rate_limit="100/min"))   # 7000 requests: about 1h 10m (limited by rate_limit)
results = reqt.fetch_all_sync(urls, rate_limit="100/min", retries=2, progress=True)
print(results.summary())

Highlights

• Results: one Result per request, in order, with the history of every attempt. A failure never stops the other requests. summary(), errors() and to_dicts() report on the batch.
• Limits:
  • Per-host rate limits in any unit: "10/s", "100/min", "1000/h".
  • Overall and per-host concurrency.
  • reqt.estimate predicts how long a batch takes.
• Retries: per-attempt timeouts, immediate retries with backoff and Retry-After, and retry rounds for what still failed at the end.
• Output:
  • Write results straight to JSONL, CSV, SQLite, PostgreSQL or MySQL, in completion order or in input order.
  • Resume an interrupted run where it stopped.
• Usage:
  • Blocking *_sync functions for scripts and Jupyter.
  • stream for results as they finish.
  • Progress with an ETA.

Upgrading from 1.x

• TLS certificates are now verified. reqt 1.x silently skipped verification.
• The 1.x call still works. fetch_all(urls=..., method=callback) runs with a DeprecationWarning, and will be removed in 3.0.
• License: reqt is now licensed under MIT; the 1.x license file was empty.

See the changelog for the full list and the README for usage.

$ python -m pip install --upgrade reqt