Rust crate to GET a file with parallel GET requests
Clone or download
mmstick A large number of improvements
See the changelog for details on 0.2.0
Latest commit 5efeb2e Oct 11, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
examples Implement more features Oct 10, 2018
src A large number of improvements Oct 12, 2018
.gitignore Initial commit Oct 9, 2018
CHANGELOG.md A large number of improvements Oct 12, 2018
Cargo.toml A large number of improvements Oct 12, 2018
LICENSE Add license Oct 9, 2018
README.md A large number of improvements Oct 12, 2018

README.md

parallel-getter

When fetching a file from a web server via GET, it is possible to define a range of bytes to receive per request. This allows the possibility of using multiple GET requests on the same URL to increase the throughput of a transfer for that file. Once the parts have been fetched, they are concatenated into a single file.

Therefore, this crate will make it trivial to set up a parallel GET request, with an API that provides a configurable number of threads and an optional callback to monitor the progress of a transfer.

Features

  • Download a file with parallel GET requests to maximize bandwidth usage.
  • Specify mirrors to spread requests out to separate servers.
  • Control the number of threads to use to fetch a request.
  • Define a minimum threshold in size for storing parts in memory.
  • Define a minimum threshold for when threads should be used.
  • Specify a number of attempts to be made before giving up.
  • An optional callback for monitoring progress of a download.
  • Fall back to a single thread if the server does not support ranges.
  • Fall back to a single thread if the server does not return a content length.
  • Alternate between mirrors when retrying after a failure.

Example

extern crate reqwest;
extern crate parallel_getter;

use reqwest::Client;
use parallel_getter::ParallelGetter;
use std::fs::File;
use std::path::PathBuf;
use std::sync::Arc;

let client = Arc::new(Client::new());
let mut file = File::create("new_file").unwrap();
ParallelGetter::new("url_here", &mut file)
    // Additional mirrors that can be used.
    .mirrors(&["mirror_a", "mirror_b"])
    // Optional client to use for the request.
    .client(client)
    // Optional path to store the parts.
    .cache_path(PathBuf::from("/a/path/here"))
    // Number of theads to use.
    .threads(5)
    // threshold (length in bytes) to determine when multiple threads are required.
    .threshold_parallel(1 * 1024 * 1024)
    // threshold for defining when to store parts in memory or on disk.
    .threshold_memory(10 * 1024 * 1024)
    // Callback for monitoring progress.
    .callback(16, Box::new(|progress, total| {
        println!(
            "{} of {} KiB downloaded",
            progress / 1024,
            total / 1024
        );
    }))
    // Commit the parallel GET requests.
    .get();