Skip to content
This repository

Take into account additional headers in cp and sync commands. #86

Open
wants to merge 1 commit into from

2 participants

Boyan Tabakov Matt Domsch
Boyan Tabakov

Support for condition headers.

It is useful for the cp/sync command to support the extra headers, because conditions such as x-amz-copy-source-if-modified-since can be given.
PreconditionFailed errors need to be accounted for in responses, as these are normal when conditions are specified.

Example usage:

s3cmd cp -r -v --add-header "x-amz-copy-source-if-modified-since:`date -u -d '1 hour ago' | cut -d\ -f 1-4,6`" s3://source-bucket/ s3://dest-bucket/

Boyan Tabakov Take into account additional headers in cp and sync commands. Support…
… for condition headers.

It is useful for the cp/sync command to support the extra headers, because conditions such as x-amz-copy-source-if-modified-since can be given.
PreconditionFailed errors need to be accounted for in responses, as these are normal when conditions are specified.
62453b0
Matt Domsch
Collaborator

I need to understand why extra_headers had been commented out in object_copy(). Same question applies to s3tools#94 which touches similar code.

What is magic response status number 412? Can we have that defined somewhere? Simply saying "skipping" doesn't tell me anything. Please be more explicit here.

Thanks,
Matt

Hi,

I have no idea why extra_headers was originally commented out. I tracked it to e0b946c, but it seems to me it was commented out for no real reason. Maybe it was temporary change that made it into the commit by mistake?

The 412 is a response code, as documented in Amazon's S3 API reference: http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectCOPY.html. It is normally returned when condition (in the extra headers) for an object copy is false.

As far as the output goes, a more explicit might be "Skipping unchanged file..." - that fits the x-amz-copy-source-if-modified-since header, but might not be very informative, if other conditions are supplied. Still x-amz-copy-source-if-modified-since is probably the most useful condition.

Cheers,
Boyan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 1 unique commit by 2 authors.

Oct 12, 2012
Boyan Tabakov Take into account additional headers in cp and sync commands. Support…
… for condition headers.

It is useful for the cp/sync command to support the extra headers, because conditions such as x-amz-copy-source-if-modified-since can be given.
PreconditionFailed errors need to be accounted for in responses, as these are normal when conditions are specified.
62453b0
This page is out of date. Refresh to see the latest.

Showing 2 changed files with 16 additions and 3 deletions. Show diff stats Hide diff stats

  1. 13  S3/S3.py
  2. 6  s3cmd
13  S3/S3.py
@@ -416,10 +416,17 @@ def object_copy(self, src_uri, dst_uri, extra_headers = None):
416 416
             headers["x-amz-acl"] = "public-read"
417 417
         if self.config.reduced_redundancy:
418 418
             headers["x-amz-storage-class"] = "REDUCED_REDUNDANCY"
419  
-        # if extra_headers:
420  
-        #   headers.update(extra_headers)
  419
+        if extra_headers:
  420
+          headers.update(extra_headers)
421 421
         request = self.create_request("OBJECT_PUT", uri = dst_uri, headers = headers)
422  
-        response = self.send_request(request)
  422
+        try:
  423
+            response = self.send_request(request)
  424
+        except Exception, e:
  425
+            if e.status == 412:
  426
+                # PreconditionFailed response - this is ok, just skip
  427
+                return {"status": e.status}
  428
+            else:
  429
+                raise e
423 430
         return response
424 431
 
425 432
     def object_move(self, src_uri, dst_uri, extra_headers = None):
6  s3cmd
@@ -535,6 +535,9 @@ def subcmd_cp_mv(args, process_fce, action_str, message):
535 535
 
536 536
         extra_headers = copy(cfg.extra_headers)
537 537
         response = process_fce(src_uri, dst_uri, extra_headers)
  538
+        if response['status'] == 412:
  539
+            info(u"Skipping file %s" % src_uri)
  540
+            continue
538 541
         output(message % { "src" : src_uri, "dst" : dst_uri })
539 542
         if Config().acl_public:
540 543
             info(u"Public URL is: %s" % dst_uri.public_url())
@@ -645,6 +648,9 @@ def cmd_sync_remote2remote(args):
645 648
         extra_headers = copy(cfg.extra_headers)
646 649
         try:
647 650
             response = s3.object_copy(src_uri, dst_uri, extra_headers)
  651
+            if response['status'] == 412:
  652
+                output("Skipping file %s" % src_uri)
  653
+                continue
648 654
             output("File %(src)s copied to %(dst)s" % { "src" : src_uri, "dst" : dst_uri })
649 655
         except S3Error, e:
650 656
             error("File %(src)s could not be copied: %(e)s" % { "src" : src_uri, "e" : e })
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.