Fix for issue #80 #81

Closed
wants to merge 3 commits into
from

Projects

None yet

3 participants

@ksumit
ksumit commented Oct 2, 2012

fixes #80

This fix makes all the downloads happen to temporary files of type .s3mcd.XXXXX.tmp in the same folder as the target file's. Once the download is complete, the file is renamed to the actual destination. This renaming is atomic in nature; hence any parallel thread or process could work on fully downloaded data (by filtering all files matching .s3cmd.XXXXX.tmp pattern while walking the data directory).

Sumit Kumar added some commits Oct 2, 2012
Sumit Kumar Creates checkpoint files of type .s3cmd.XXXX.tmp while downloading from
S3 in the same directory as of the local destination. This can be useful
for checkpointing state of the downloaded data and then processing on a
checkpointed state itself. This is important for synchronized processing
of data downloaded from S3 synchronously (so that processing doesn't
fail on incomplete data)
9fb5900
Sumit Kumar removed possibility of creating a 0 byte file if python process is
killed abruptly
67268a2
Sumit Kumar formatting 5c6dc67
@mdomsch
Member
mdomsch commented Dec 6, 2012

I manually merged this (hand-applied each line of diff) to my 'merge' branch, given this portion of the code has changed quite a bit since the last formal release your patch was based upon. You still got credit in the git log.

Thanks!

@mludvig
Contributor
mludvig commented Mar 9, 2013

Merged through Matt's tree.

@mludvig mludvig closed this Mar 9, 2013
@ksauzz ksauzz pushed a commit to ksauzz/s3cmd that referenced this pull request Feb 14, 2014
@mdomsch Sumit Kumar + mdomsch [sync] download files to a temporary filename, then rename
This fix makes all the downloads happen to temporary files of type
.s3cmd.XXXXX.tmp in the same folder as the target file's. Once the
download is complete, the file is renamed to the actual
destination. This renaming is atomic in nature; hence any parallel
thread or process could work on fully downloaded data (by filtering
all files matching .s3cmd.XXXXX.tmp pattern while walking the data
directory).

s3tools#81

Patch manually applied by Matt Domsch because this portion of the code has
changed more than pulling or rebasing could handle.
eecbcba
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment