Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Take into account additional headers in cp and sync commands. #86

Open
wants to merge 1 commit into from

2 participants

@bladealslayer

Support for condition headers.

It is useful for the cp/sync command to support the extra headers, because conditions such as x-amz-copy-source-if-modified-since can be given.
PreconditionFailed errors need to be accounted for in responses, as these are normal when conditions are specified.

Example usage:

s3cmd cp -r -v --add-header "x-amz-copy-source-if-modified-since:`date -u -d '1 hour ago' | cut -d\ -f 1-4,6`" s3://source-bucket/ s3://dest-bucket/

@bladealslayer bladealslayer Take into account additional headers in cp and sync commands. Support…
… for condition headers.

It is useful for the cp/sync command to support the extra headers, because conditions such as x-amz-copy-source-if-modified-since can be given.
PreconditionFailed errors need to be accounted for in responses, as these are normal when conditions are specified.
62453b0
@mdomsch
Owner

I need to understand why extra_headers had been commented out in object_copy(). Same question applies to s3tools#94 which touches similar code.

What is magic response status number 412? Can we have that defined somewhere? Simply saying "skipping" doesn't tell me anything. Please be more explicit here.

Thanks,
Matt

Hi,

I have no idea why extra_headers was originally commented out. I tracked it to e0b946c, but it seems to me it was commented out for no real reason. Maybe it was temporary change that made it into the commit by mistake?

The 412 is a response code, as documented in Amazon's S3 API reference: http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectCOPY.html. It is normally returned when condition (in the extra headers) for an object copy is false.

As far as the output goes, a more explicit might be "Skipping unchanged file..." - that fits the x-amz-copy-source-if-modified-since header, but might not be very informative, if other conditions are supplied. Still x-amz-copy-source-if-modified-since is probably the most useful condition.

Cheers,
Boyan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Oct 12, 2012
  1. @bladealslayer

    Take into account additional headers in cp and sync commands. Support…

    bladealslayer authored bladealslayer committed
    … for condition headers.
    
    It is useful for the cp/sync command to support the extra headers, because conditions such as x-amz-copy-source-if-modified-since can be given.
    PreconditionFailed errors need to be accounted for in responses, as these are normal when conditions are specified.
This page is out of date. Refresh to see the latest.
Showing with 16 additions and 3 deletions.
  1. +10 −3 S3/S3.py
  2. +6 −0 s3cmd
View
13 S3/S3.py
@@ -416,10 +416,17 @@ def object_copy(self, src_uri, dst_uri, extra_headers = None):
headers["x-amz-acl"] = "public-read"
if self.config.reduced_redundancy:
headers["x-amz-storage-class"] = "REDUCED_REDUNDANCY"
- # if extra_headers:
- # headers.update(extra_headers)
+ if extra_headers:
+ headers.update(extra_headers)
request = self.create_request("OBJECT_PUT", uri = dst_uri, headers = headers)
- response = self.send_request(request)
+ try:
+ response = self.send_request(request)
+ except Exception, e:
+ if e.status == 412:
+ # PreconditionFailed response - this is ok, just skip
+ return {"status": e.status}
+ else:
+ raise e
return response
def object_move(self, src_uri, dst_uri, extra_headers = None):
View
6 s3cmd
@@ -535,6 +535,9 @@ def subcmd_cp_mv(args, process_fce, action_str, message):
extra_headers = copy(cfg.extra_headers)
response = process_fce(src_uri, dst_uri, extra_headers)
+ if response['status'] == 412:
+ info(u"Skipping file %s" % src_uri)
+ continue
output(message % { "src" : src_uri, "dst" : dst_uri })
if Config().acl_public:
info(u"Public URL is: %s" % dst_uri.public_url())
@@ -645,6 +648,9 @@ def cmd_sync_remote2remote(args):
extra_headers = copy(cfg.extra_headers)
try:
response = s3.object_copy(src_uri, dst_uri, extra_headers)
+ if response['status'] == 412:
+ output("Skipping file %s" % src_uri)
+ continue
output("File %(src)s copied to %(dst)s" % { "src" : src_uri, "dst" : dst_uri })
except S3Error, e:
error("File %(src)s could not be copied: %(e)s" % { "src" : src_uri, "e" : e })
Something went wrong with that request. Please try again.