If getting the file list from files_from, it is going to be subset of the actual tree. We should not purge content outside of that subset as we don't know if it's valid or not. Leave it to a non-files_from run to purge.
s3cmd get to stdout: set file_exists to avoid traceback, bug #175
Fixes s3tools#175 Reported by Seth Noble. Set file_exists so we don't try to delete sys.stdout if getting the file happens to fail.
Sorry this is not correct. What will happen if we create the md5_xattr on Monday and change the file on Tuesday? Unfortunately the old md5_xattr will still be there and s3cmd will think the file hasn't changed. This is not acceptable - we need a way to disregard md5_xattr if the file has changed, that is if its size or mtime or ctime has changed. These values will have to be stored in the xattr and checked by s3cmd before using the md5 from there.
…e containing md5sums" This reverts commit 730dbcc.
This reverts commit 8f9b035.
This reverts commit f685a13.
Amazon's preferred NoSQL is now DynamoDB anyway.
Now that we can reasonably expect response['s3cmd-attrs'] to contain valid md5 data, use that in the bucket list and info commands instead of ETags if available. Furthermore, be sure to store these attributes in the put command, either when --preserve is given, or if we will be uploading a multi-chunk file. We were storing these with the sync --preserve command, be sure to do it in put also.
This is AWS-specific metadata. It should be handled in exactly one place, S3.py, rather than ad-hoc in several places.
In the recursive case, we really don't care if require_attribs is set, but we do still care if the existing MD5 value (from ETag) is a multi-part object. In that case, we definitely want to try to get it from the headers. Drop test for require_attribs. Keep test for "-" in ETag.
We were not getting the md5 value from the attribute header of each file for the recursive case, only for the non-recursive case. That was stupid. We were certainly _setting_ the md5value into the attribute header. This should eliminate the INFO: disabled md5 check for... messages on subsequent syncs when s3cmd 1.5.0-alpha+ is used to upload content in the first place.
We were looking at the wrong (post-compare) list lengths, rather than the original (discovered by walk) list lengths. And we weren't checking to see if we had any files to delete, or not, before issuing the "cowardly" message. So for a "null" (no files to transfer) sync run, we would emit the message. That's wrong. This patch fixes both problems: 1) it uses the discovered source file count. If nonzero, we will not emit the message. 2) it looks at the count of files to delete. If zero, we will not emit the message.
Otherwise options.md5_xattr does not exist.
…ning md5sums Inspired by a patch from Jay McCanta <J.McCanta@F5.com> who suggested this was possible, 2013-05-14 to the s3tools-general mailing list. This patch adds the --xattr command line and md5_xattr config file option, specifying a file system extended attribute in the USER namespace which contains the md5sum for the file. If the extended attribute is not present, the code falls back to performing local file I/O to calculate the md5sum, as before. This can be used to speed up transfers by avoiding the need to calculate md5sums for all files, assuming this has been done at least once before somehow, with the results stored in each file's xattr. In addition to Jay's work, this patch simplifies testing for xattr module import success, passes the result into the Config object, and ensures the correct namespace is used. Also added a test to the test suite, and the entry on the manpage.