ThrottleBuster is a Python library that accelerates file downloads by bypassing common throttling restrictions. It uses asynchronous methods to improve download speeds and is especially useful for handling large files.
- Concurrent downloading across multiple tasks
- Fully asynchronous with synchronous support
- Ready to use commandline tool
$ pip install "throttlebuster[cli]"For testing, you can set up an Nginx server with the example configuration below:.
# Test server
## Create this file in /etc/nginx/sites-enabled or add it to the default nginx.conf
server {
listen 8888;
server_name throttlebuster.test;
location / {
limit_rate 500k; # Limit rate to 500 KB/s
root /home/smartwa/y2mate; # Set your own directory
index index.html;
}
}
from throttlebuster import ThrottleBuster
async def main():
throttlebuster = ThrottleBuster(tasks=4)
downloaded_file = await throttlebuster.run(
"http://localhost:8888/test.1.opus",
)
print(
downloaded_file
)
if __name__ == "__main__":
import asyncio
asyncio.run(main())from throttlebuster import DownloadTracker, ThrottleBuster
async def callback_function(data: DownloadTracker):
percent = (data.downloaded_size / data.expected_size) * 100
print(f"> Downloading {data.saved_to.name} {percent:.2f}%", end="\r")
async def main():
throttlebuster = ThrottleBuster(tasks=1)
downloaded_file await throttlebuster.run(
"http://localhost:8888/test.1.opus", progress_hook=callback_function
)
print(
downloaded_file
)
if __name__ == "__main__":
import asyncio
asyncio.run(main())
$ python -m throttlebuster --help
$ python -m throttlebuster --helpUsage: python -m throttlebuster [OPTIONS] COMMAND [ARGS]...
Accelerate file downloads by overcoming throttling restrictions envvar-
prefix : THROTTLEBUSTER.
Options:
--version Show the version and exit.
--help Show this message and exit.
Commands:
d Download file using http/s protocol
download Download file using http/s protocol
e Estimate download duration for different task amount
estimate Estimate download duration for different task amount
$ python -m throttlebuster download http://localhost:8888/test.1.opus --tasks 14
$ python -m throttlebuster download --help
$ python -m throttlebuster download --helpUsage: python -m throttlebuster download [OPTIONS] URL
Download file using http/s protocol
Options:
-T, --tasks INTEGER RANGE Number of tasks to carry out the download
[default: 2; 1<=x<=1000]
-Z, --chunk-size INTEGER Streaming download chunk size in kilobytes
[default: 256]
-D, --dir DIRECTORY Directory for saving the downloaded file to
[default:
/home/smartwa/git/smartwa/throttlebuster]
-P, --part-dir DIRECTORY Directory for temporarily saving the
downloaded file-parts to [default:
/home/smartwa/git/smartwa/throttlebuster]
-E, --part-extension TEXT Filename extension for download parts
[default: .part]
-H, --request-headers TEXT... Httpx request header - [key value] : default
-C, --request-cookies TEXT... Httpx request cookie - [key value]: default
-B, --merge-buffer-size INTEGER RANGE
Buffer size for merging the separated files
in kilobytes [default: 256; 1<=x<=102400]
-F, --filename TEXT Filename for the downloaded content
-M, --mode [start|resume|auto] Whether to start or resume incomplete
download [default: auto]
-L, --file-size INTEGER Size of the file to be downloaded
-X, --proxy TEXT Request proxy with schema of any type
[default: system-set]
-R, --timeout-retry-attempts INTEGER
Number of times to retry download upon read
request timing out [default: 10]
-K, --colour TEXT Progress bar display color [default: cyan]
-k, --keep-parts Whether to retain the separate download
parts
-s, --simple Show percentage and bar only in progressbar
-t, --test Just test if download is possible but do not
actually download
-a, --ascii Use unicode (smooth blocks) to fill the
progress-bar meter
-l, --no-leave Do not keep traces of the progressbar
-z, --disable-progress-bar Do not show progress_bar
-i, --suppress-incompatible-error
Do no raise error when response headers lack
Etag
--follow-redirects / --no-follow-redirects
Follow url redirects by the server
[default: no-follow-redirects]
--verify / --no-verify Verify server certificate [default: verify]
-q, --quiet Do not show any interactive information
-v, --verbose Show more detailed information [default: 0]
--help Show this message and exit.
Tip
Shortcuts for $ python -m throttlebuster are $ throttlebuster & $ tbust
$ python -m throttlebuster estimate --url http://localhost:8888/miel-martin.webm 260000
$ python -m throttlebuster estimate --url http://localhost:8888/miel-martin.webm 260000
337.88 MB at 260.00 KB/s
┏━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Tasks ┃ Duration ┃ Load per task ┃
┡━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ 20 │ 1.08 Mins │ 16.89 MB │
│ 19 │ 1.14 Mins │ 17.78 MB │
│ 18 │ 1.20 Mins │ 18.77 MB │
│ 17 │ 1.27 Mins │ 19.88 MB │
│ 16 │ 1.35 Mins │ 21.12 MB │
│ 15 │ 1.44 Mins │ 22.53 MB │
│ 14 │ 1.55 Mins │ 24.13 MB │
│ 13 │ 1.67 Mins │ 25.99 MB │
│ 12 │ 1.80 Mins │ 28.16 MB │
│ 11 │ 1.97 Mins │ 30.72 MB │
│ 10 │ 2.17 Mins │ 33.79 MB │
│ 9 │ 2.41 Mins │ 37.54 MB │
│ 8 │ 2.71 Mins │ 42.24 MB │
│ 7 │ 3.09 Mins │ 48.27 MB │
│ 6 │ 3.61 Mins │ 56.31 MB │
│ 5 │ 4.33 Mins │ 67.58 MB │
│ 4 │ 5.41 Mins │ 84.47 MB │
│ 3 │ 7.22 Mins │ 112.63 MB │
│ 2 │ 10.83 Mins │ 168.94 MB │
│ 1 │ 21.66 Mins │ 337.88 MB │
└───────┴────────────┴───────────────┘
$ python -m throttlebuster estimate --help
$ python -m throttlebuster estimate --helpUsage: python -m throttlebuster estimate [OPTIONS] THROTTLE
Estimate download duration for different task amount
Options:
-U, --url TEXT Url to the target file
-S, --size INTEGER Size in bytes of the targeted file
-T, --tasks INTEGER RANGE Tasks amount to base the estimate on : Range
(2-30) [1<=x<=1000]
-j, --json Stdout estimates in json format
--help Show this message and exit.
Throttle unit is bytes.