A parallel file downloader using asyncio.
parfive works by creating a downloader object, appending files to it and then running the download. parfive has a synchronous API, but uses asyncio to paralellise downloading the files.
A simple example is:
from parfive import Downloader dl = Downloader() dl.enqueue_file("http://data.sunpy.org/sample-data/predicted-sunspot-radio-flux.txt", path="./") files = dl.download()
parfive.Downloader.download returns a
parfive.Results object, which is a
list of the filenames that have been downloaded. It also tracks any files which
failed to download.
If files fail to download, the urls and the response from the server are stored
Results object returned by
parfive.Downloader. These can be used to
inform users about the errors. (Note, the progress bar will finish in an
incomplete state if a download fails, i.e. it will show
4/5 Files Downloaded).
Results object is a list with an extra attribute
errors, this property
returns a list of named tuples, where these named tuples contains the
.response, which is a
aiohttp.ClientResponse or a
parfive is available on PyPI, you can install it with pip:
pip install parfive
or if you want to use FTP downloads:
pip install parfive[ftp]
- Python 3.5+
- aioftp (for downloads over FTP)
parfive was written by Stuart Mumford.