async_downloader.py

Finally, a reliable multi-connection downloader.

Note: This project is written for justfoolingaround's project.

Features

Warning: This project only supports httpx.AsyncClient as its session class. However, migrating into aiohttp is not that difficult. You may just inherit two coroutine methods of the class (is_resumable and download_part).

• Can be used as a single connection downloader for those cases.
• Implements a highly reliable is_resumable method for validating resumable content.^{When to use this?}
• Supports pause event that immediately stops the download progress.
• Supports tqdm.tqdm to be used as its progress bar.
• Supports progress bar for individual connections.
• Supports asyncio.Future usage in the download_part function.^{When to use this?}
• Supports immediately allocating the file to the disk.

When to use asyncio.Future in the download_part function?

TL;DR: For pausing / resuming effectively.

This tool is a bare-bone downloader. The downloader cannot just guess when the previous download completed. This is majorly because it uses a single file during its IO operations. It opens a file, downloads content and puts those content in the expected positions while downloading.

So, when a pause occurs, this future is expected to get the completed position of the download progress so that the developer make implementations for resuming content that did not make it.

When to use the is_resumable method?

TL;DR: Recommended to use every time.

There are servers in the wild that are notorious for breaking standards.

The standard method is to use a HEAD request, see if Accept-Ranges and/or Content-Length are present and operate accordingly. However sometimes,

• The server may block the HEAD requests.
• The server may not oblige by the Accept-Ranges and/or Content-Length headers.

The method within the project simply sends a GET request with range: 0-0 and checks if the server sends in a standard 206 Partial Content response.