… PR of petr-tichy. When using large --multipart-chunk-size-mb or --disable-multipart, the whole file was read into memory. Based on: #579
This will now return either EX_DATAERR on transfer error or EX_OSFILE if invalid file name, when stop_on_error is selected. Also add the 2 missing exit paths from the original pull request. Signed-off-by: Tim Anderson <email@example.com>
Add a configuration option that allows an error return on any transfer error of a file. This is a config option (stop_on_error) and a commandline option (--stop-on-error). The Default is the current continue behavior. With the option or config setting to True, the commands (get, put, sync remote2remote, remote2local, or local2remote) will all return an error status. The Get and Put commands return EX_PARTIAL where the typical response on sync will be EX_TEMPFAIL but could be any of: * EX_SERVERMOVED * EX_SERVERERROR * EX_ACCESSDENIED * EX_NOTFOUND * EX_CONFLICT * EX_PRECONDITION * EX_SERVICE * EX_SOFTWARE Signed-off-by: Tim Anderson <firstname.lastname@example.org>
There is a lot of different "python-magic" libraries/versions, all doing different things, so all cases need to be covered. file-5.11 has be switched with "filemagic" as it should be more commonly used. Introspection in each python magic lib has been done to ensure to handle encoding or not only when it is needed.
Right now the send_file() routine is always sending a GetBucketLocation request before actually trying to send the file, because the S3Request.region_map is empty initially. Instead, we should trust the value of Config.bucket_location until such time as we have received a 400 failure and been told the correct location. This avoids needing to have s3::GetBucketLocation IAM permission also, when we know the correct bucket_location already (either from config file or command line option).
The date header contains the remote server's current time. That's not what we want. Instead, we want the last-modified value for the file being inspected with the [info] (or equivalent) request. Thanks to Andrew Trusty for finding, diagnosing, and suggesting a fix. And thanks to DreamObjects for "forgetting" to include a date header in their response to a HEAD of an object on occasion, that led us to find this. This should also improve --no-check-md5 cases where the timestamp is being checked, as we won't think that remote files have a timestamp of "now".
This way we can catch all all GET and POST calls as https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html indicates we should do. This also lets us [ls] an object in a requester-pays bucket which we couldn't do before.
…are bytes, so unicodise them also)
…e start of the operation (reported by a status 200 but an "Error" root entry in the xml body) In addition, for the move, It is good to check for the CopyObject xml reply to see if it is a success. But, if there is a status of 200 and no "data" in response, don't fail. Other S3 server implementations use only status code and don't return any body when there is no error. References: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html http://doc.s3.amazonaws.com/proposals/copy.html
…rror expect an http response, not a text message)
…hed to "unicode" call instead of "decode" as it is supposed to be more efficient.