reqt 2.0.0

reqt 2.0 sends large numbers of HTTP requests concurrently and gives you one result per request, with rate limits, retries, progress and output to files or databases built in.

import reqt

print(reqt.estimate(7000, rate_limit="100/min"))   # 7000 requests: about 1h 10m (limited by rate_limit)
results = reqt.fetch_all_sync(urls, rate_limit="100/min", retries=2, progress=True)
print(results.summary())

Highlights

• Results: one Result per request, in order, with the history of every attempt. A failure never stops the other requests. summary(), errors() and to_dicts() report on the batch.
• Limits:
  • Per-host rate limits in any unit: "10/s", "100/min", "1000/h".
  • Overall and per-host concurrency.
  • reqt.estimate predicts how long a batch takes.
• Retries: per-attempt timeouts, immediate retries with backoff and Retry-After, and retry rounds for what still failed at the end.
• Output:
  • Write results straight to JSONL, CSV, SQLite, PostgreSQL or MySQL, in completion order or in input order.
  • Resume an interrupted run where it stopped.
• Usage:
  • Blocking *_sync functions for scripts and Jupyter.
  • stream for results as they finish.
  • Progress with an ETA.

Upgrading from 1.x

• TLS certificates are now verified. reqt 1.x silently skipped verification.
• The 1.x call still works. fetch_all(urls=..., method=callback) runs with a DeprecationWarning, and will be removed in 3.0.
• License: reqt is now licensed under MIT; the 1.x license file was empty.

See the changelog for the full list and the README for usage.

$ python -m pip install --upgrade reqt