Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallel multipart upload and download support for s3cmd #9

Closed
wants to merge 16 commits into from
Closed

Parallel multipart upload and download support for s3cmd #9

wants to merge 16 commits into from

Commits on Nov 6, 2011

  1. Adding changes to recv_file to support partial file download by spe… 

    …cifying start-position and end-position in bytes
    @t3rm1n4l
    t3rm1n4l committed Nov 6, 2011
    Configuration menu
    Copy the full SHA
    a8f80ea View commit details
    Browse the repository at this point in the history

  2. Added concat_files() function 

    A function that takes destination file handle and list of source file handles, concatenate source files data and write into destination file
    @t3rm1n4l
    t3rm1n4l committed Nov 6, 2011
    Configuration menu
    Copy the full SHA
    e60ae89 View commit details
    Browse the repository at this point in the history

  3. Added object_multipart_get() function 

    * object_multipart_get() - Download a file from s3 by parallel download of multiple split files using Worker thread pool, merge split files and perform md5 checksum verification
* Added new parameters in Config to specify worker thread numbers, file split count and to toggle parallel split download on and off
    @t3rm1n4l
    t3rm1n4l committed Nov 6, 2011
    Configuration menu
    Copy the full SHA
    8772c9a View commit details
    Browse the repository at this point in the history

  4. Added changes to s3cmd for switching parallel split download on/off b… 

    …ased on configuration file
    @t3rm1n4l
    t3rm1n4l committed Nov 6, 2011
    Configuration menu
    Copy the full SHA
    7bacce0 View commit details
    Browse the repository at this point in the history

  5. Cleanup handlers for temporary files and disk usage improvement to co… 

    …ncat_files() method to readily remove split files once data is read.
    @t3rm1n4l
    t3rm1n4l committed Nov 6, 2011
    Configuration menu
    Copy the full SHA
    e515259 View commit details
    Browse the repository at this point in the history

  6. Changed threading.active_count() to threading.activeCount() for backw… 

    …ard compatibility with python 2.4
    @t3rm1n4l
    t3rm1n4l committed Nov 6, 2011
    Configuration menu
    Copy the full SHA
    16c3db4 View commit details
    Browse the repository at this point in the history

  7. Added parameters signing support for auth signature calculation metho… 

    …d sign()
    @t3rm1n4l
    t3rm1n4l committed Nov 6, 2011
    Configuration menu
    Copy the full SHA
    abf8a6b View commit details
    Browse the repository at this point in the history

  8. Added multipart upload support. Multipart upload can be enabled by a… 

    …dding parallel_multipart_upload = True in config file
    @t3rm1n4l
    t3rm1n4l committed Nov 6, 2011
    Configuration menu
    Copy the full SHA
    14d3015 View commit details
    Browse the repository at this point in the history

  9. Replaced email.Util.formatdate() with time module - ensure compatibil… 

    …ity with python 2.4
    @t3rm1n4l
    t3rm1n4l committed Nov 6, 2011
    Configuration menu
    Copy the full SHA
    9f3003c View commit details
    Browse the repository at this point in the history

  10. FIX: Correct split numbering, unsymmetric file split issue

    @t3rm1n4l
    t3rm1n4l committed Nov 6, 2011
    Configuration menu
    Copy the full SHA
    077bf4c View commit details
    Browse the repository at this point in the history

  11. s3cmd info - fix to show correct md5sum for multipart uploaded file b… 

    …ased on custom md5 meta header
    @t3rm1n4l
    t3rm1n4l committed Nov 6, 2011
    Configuration menu
    Copy the full SHA
    bd70d2b View commit details
    Browse the repository at this point in the history

  12. Fix for python 2.4 daemon thread by using setDaemon(True) by replacin… 

    …g daemon=True
    @t3rm1n4l
    t3rm1n4l committed Nov 6, 2011
    Configuration menu
    Copy the full SHA
    942b19f View commit details
    Browse the repository at this point in the history

  13. Disable progress bar for multipart upload

    @t3rm1n4l
    t3rm1n4l committed Nov 6, 2011
    Configuration menu
    Copy the full SHA
    b5f6590 View commit details
    Browse the repository at this point in the history

  14. Added meta based md5 verfication for non-parallel downloader for file… 

    …s uploaded with multipart upload
    @t3rm1n4l
    t3rm1n4l committed Nov 6, 2011
    Configuration menu
    Copy the full SHA
    2fb607a View commit details
    Browse the repository at this point in the history

  15. Added exit_status for s3cmd program 

    s3cmd does not return valid exit status codes. Hence it is unable to identify whether the program succeeded or failed (with cause of failure)
This commit adds exit status for s3cmd sync upload, sync download, get and put operations
Exit codes are as follows : SIZE_MISMATCH=1, MD5_MISMATCH=2, RETRIES_EXCEEDED=3, UPLOAD_ABORT=4, MD5_META_NOTFOUND=5, KEYBOARD_INTERRUPT=6
    @t3rm1n4l
    t3rm1n4l committed Nov 6, 2011
    Configuration menu
    Copy the full SHA
    0d84df6 View commit details
    Browse the repository at this point in the history

  16. Added separate config parameters for thread limit and split count wrt… 

    … to download and upload

added download config parameters parallel_multipart_download_threads (thread count), parallel_multipart_download_count (split count)
added upload config parameters parallel_multipart_upload_threads (thread count), parallel_multipart_upload_count (split count)
    @t3rm1n4l
    t3rm1n4l committed Nov 6, 2011
    Configuration menu
    Copy the full SHA
    b3c47d0 View commit details
    Browse the repository at this point in the history