pyhaul — resumable, crash-safe async downloads with aiohttp.ClientSession

[chad-loder]

I built pyhaul, a pure-Python library for resumable HTTP downloads. It has first-class aiohttp support — pass your aiohttp.ClientSession directly and pyhaul handles byte-range negotiation, ETag validation, and crash-safe checkpointing.

pip install pyhaul[aiohttp]

Basic usage

import asyncio
import aiohttp
from pyhaul import haul_async, PartialHaulError

async def main():
    async with aiohttp.ClientSession() as session:
        for _ in range(10):
            try:
                result = await haul_async(
                    "https://example.com/big.zip", session, dest="big.zip"
                )
                break
            except PartialHaulError:
                pass  # progress saved to disk; next call resumes

asyncio.run(main())

One haul_async() = one HTTP request. It either succeeds, or it saves progress so the next call resumes from the last durable byte.

Concurrent downloads with TaskGroup

import asyncio
from pathlib import Path
import aiohttp
from pyhaul import haul_async, PartialHaulError

URLS = [
    ("https://data.example.edu/shard-001.bin", Path("data/shard-001.bin")),
    ("https://data.example.edu/shard-002.bin", Path("data/shard-002.bin")),
    ("https://data.example.edu/shard-003.bin", Path("data/shard-003.bin")),
]

async def download_one(session: aiohttp.ClientSession, url: str, dest: Path):
    for attempt in range(1, 11):
        try:
            await haul_async(url, session, dest=dest)
            return dest
        except PartialHaulError:
            if attempt == 10:
                raise
            await asyncio.sleep(min(2**attempt, 30))

async def main():
    Path("data").mkdir(exist_ok=True)
    async with aiohttp.ClientSession() as session:
        async with asyncio.TaskGroup() as tg:
            tasks = [
                tg.create_task(download_one(session, url, dest))
                for url, dest in URLS
            ]
    for task in tasks:
        print(f"done: {task.result()}")

asyncio.run(main())

How it works with aiohttp

• Your session, your config. pyhaul borrows your ClientSession — it never creates, configures, or closes it. Auth, proxies, timeouts, connector settings all pass through unchanged.
• Real async. The async engine uses async with / async for natively. It's not a sync wrapper in a thread pool.
• Raw streaming for correctness. pyhaul sets auto_decompress=False on requests internally to get raw bytes for accurate byte-range resume. Your session's other settings are untouched.
• aiohttp errors pass through unwrapped. aiohttp.ClientError stays aiohttp.ClientError. You catch the types you already know.

What pyhaul handles

The "resumable download" problem is deceptively complex at the protocol level. Some things pyhaul takes care of:

• ETag validation on resume. If the remote file changes between attempts, pyhaul detects the mismatch via If-Range and starts over cleanly instead of splicing mismatched data.
• Servers ignoring Range headers. Servers are allowed to respond 200 instead of 206. pyhaul detects this, rewinds the on-disk cursor, and starts clean.
• Crash-safe file handling. The destination file doesn't exist until download is complete — incomplete data lives in a .part file that gets atomically renamed on success. Checkpoint metadata is flushed to disk after data, never before.
• Compression edge cases. pyhaul sends Cache-Control: no-transform to prevent proxies from re-compressing range responses.

I wrote up a deep-dive comparing pyhaul's behavior to curl, wget, and aria2 (verified against source code) if anyone's interested in the protocol details.

Links

• PyPI: https://pypi.org/project/pyhaul/
• GitHub: https://github.com/chad-loder/pyhaul
• Docs (aiohttp adapter guide): https://chad-loder.github.io/pyhaul/guides/adapters/#aiohttp
• Full docs: https://chad-loder.github.io/pyhaul/

Pure Python, zero required dependencies, MIT licensed, Python 3.12+. Also supports httpx, requests, niquests, and urllib3 if you use those. Feedback welcome.

[chad-loder]

PS: Here's how I designed the checkpoint file spec. It would have been a bit nicer to use BLAKE3 for this but I wanted pyhaul to be a zero dependencies library so I had to go with what was in stdlib. Still, the format is solid IMHO, it scales well for large files, the headers are aligned on word boundaries to make it super easy to read into a C or Rust struct.

It's a tree hash of sha256 in 8mb chunks, so when re-reading after a resume you only need to rehash the tail chunk rather than read an entire 100GB file.

Also some careful file handling and I/O sequencing to get the atomicity guarantees correct and handle file growth/preallocation and path handling correct on Linux/Mac/Windows.

And a neat little immutable, O(1) HTTP headers implementation that preserves wire ordering and handles all the case normalization without needing any third party libraries.

[Dreamsorcerer]

Feel free to open a PR to add to third-party section of docs.