Utilities to do parallel upload/download with Amazon S3
Python
Switch branches/tags
Nothing to show
Permalink
Failed to load latest commit information.
LICENSE Add license (ASL v2) Nov 16, 2012
README.md Add license (ASL v2) Nov 16, 2012
requirements.txt Adding argparse for Python<2.7 Jun 7, 2012
s3-mp-cleanup.py Cleanup Jun 18, 2013
s3-mp-copy.py Add mixed case fix to s3-mp-copy.py Dec 12, 2013
s3-mp-download.py
s3-mp-upload.py Merge branch 'master' of git://github.com/dougalb/s3-multipart into d… Dec 12, 2013

README.md

Utilities to do parallel upload/download with Amazon S3

Install with Pip

pip install -r requirements.txt

Install without Pip

easy_install -U boto

Parallel download (s3-mp-download)

Utilizes S3's support for the Range HTTP header, fetches multiple chunks of the file in parallel. See: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

$ ./s3-mp-download.py -h
usage: s3-mp-download [-h] [-np NUM_PROCESSES] [-f] src dest

Download a file from S3 in parallel

positional arguments:
  src                   The S3 key to download
  dest                  The destination file

optional arguments:
  -h, --help            show this help message and exit
  -np NUM_PROCESSES, --num-processes NUM_PROCESSES
		    Number of processors to use
  -f, --force           Overwrite an existing file

Parallel upload (s3-mp-upload)

Utilizes the Multipart Upload feature of S3. Splits up the local file into chunks and uploads them in parallel. See: http://aws.typepad.com/aws/2010/11/amazon-s3-multipart-upload.html

usage: s3-mp-upload [-h] [-n NUM_PROCESSES] [-f] [-s SPLIT] src dest

Transfer large files to S3

positional arguments:
  src                   The file to transfer
  dest                  The S3 destination object

optional arguments:
  -h, --help            show this help message and exit
  -n NUM_PROCESSES, --num-processes NUM_PROCESSES
		    Number of processors to use
  -f, --force           Overwrite an existing S3 key
  -s SPLIT, --split SPLIT
		    Split size, in Mb

Credits

As always, mad props to the Boto project and it's maintainer, Mitch https://github.com/boto/boto

License

Copyright 2012, David Arthur under Apache License, v2.0. See LICENSE