Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Parallel multipart upload and download support for s3cmd #9

Closed
wants to merge 16 commits into
from

Conversation

Projects
None yet
1 participant

t3rm1n4l commented Nov 6, 2011

I have developed parallel multipart upload and download support for s3cmd.
We can enable the parallel upload and download using the following configuration.
parallel_multipart_download = False
parallel_multipart_upload = False
parallel_multipart_download_threads = 5
parallel_multipart_upload_threads = 5
parallel_multipart_download_count = 5
parallel_multipart_upload_count = 5

Please merge my changes to the s3cmd

Looking into your review comments.

Thanks
Sarath

t3rm1n4l added some commits Oct 19, 2011

Adding changes to `recv_file` to support partial file download by spe…
…cifying start-position and end-position in bytes
Added `concat_files()` function
A function that takes destination file handle and list of source file handles, concatenate source files data and write into destination file
Added `object_multipart_get()` function
* object_multipart_get() - Download a file from s3 by parallel download of multiple split files using Worker thread pool, merge split files and perform md5 checksum verification
* Added new parameters in Config to specify worker thread numbers, file split count and to toggle parallel split download on and off
Cleanup handlers for temporary files and disk usage improvement to co…
…ncat_files() method to readily remove split files once data is read.
Added multipart upload support. Multipart upload can be enabled by a…
…dding parallel_multipart_upload = True in config file
Added exit_status for s3cmd program
s3cmd does not return valid exit status codes. Hence it is unable to identify whether the program succeeded or failed (with cause of failure)
This commit adds exit status for s3cmd sync upload, sync download, get and put operations
Exit codes are as follows : SIZE_MISMATCH=1, MD5_MISMATCH=2, RETRIES_EXCEEDED=3, UPLOAD_ABORT=4, MD5_META_NOTFOUND=5, KEYBOARD_INTERRUPT=6
Added separate config parameters for thread limit and split count wrt…
… to download and upload

added download config parameters parallel_multipart_download_threads (thread count), parallel_multipart_download_count (split count)
added upload config parameters parallel_multipart_upload_threads (thread count), parallel_multipart_upload_count (split count)

@t3rm1n4l t3rm1n4l closed this Nov 6, 2011

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment