v4.0.0

fast-s3 4.0.0

Rewrite of the transfer engine for many small files. boto3 is gone; transfers run on a single asyncio event loop with an aiohttp-based S3 client and hand-rolled SigV4 signing, keeping hundreds of requests in flight from one process at ~0.1 ms CPU per object.

Measured on Wasabi eu-central-1 over 1 Gbit ethernet with 27 KB files: ~1,800 files/s download and ~2,200 files/s upload at concurrency 256, versus ~1,600 / ~1,400 for rclone.

Breaking changes

• boto3, botocore and s3transfer are no longer dependencies.
• n_workers is deprecated in favour of concurrency (default 256); worker_batch_size and transfer-manager kwargs were removed.
• buffer_size now bounds memory in ordered mode as well. Defaults to 4x concurrency, or 32x concurrency when ordered=True, because one slow object stalls ordered iteration.
• Requires Python >= 3.9.

Added

• Retries with exponential backoff on connection errors and 429/5xx; 404/403 fail immediately and are reported per file, never raised from iteration.
• Hedged GET/HEAD requests (hedge=) to cut the latency tail.
• Uploader accepts bytes, file paths or file-like objects and reads files lazily under the concurrency limit; usable as a context manager.
• AsyncS3Client exported for direct asyncio use.
• Tests: AWS signature vectors, retry behaviour against a fake server, optional integration tests (FAST_S3_TEST_*).
• benchmarks/compare_rclone.py and benchmarks/compare_rclone_upload.py.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PypmjWnWFCaqquVV7k7GU9