How to create copy of object version into another bucket

[vlcinsky]

I have versioned bucket and need to copy set of object versions into another (non versioned) bucket.

There are some SomeResource.copy_from and Client.copy_object methods.

However, I was unable to find how to specify source object version_id. The only place, where I have found CopySourceVersionId was response, but no place to put it into request.

Is there any way to copy object version into another bucket without downloading it?

With boto it was possible (but I my source bucket is having dots in names and this is not working with old boto any more.

[vlcinsky]

I have found the solution.

Searching for similar problems I have found issue boto/botocore#509 I have found undocumented method to specify version id in botocore.

Testing it on boto3 I have found, it works there.

To name specific object version as SourceObject, one has to use following pattern: "/{bucket}/{key}?versionId={id}".

Now I am able to copy from specific version using boto3.

I still consider this an issue, as current version of documentation does not explain it well (it omits the version and does not explain the need to use the leading slash).

Existing specification for Object.copy_from reads for the SourceCopy parameter:

CopySource (string) -- [REQUIRED] The name of the source bucket and key name of the source object, separated by a slash (/). Must be URL-encoded.

I would propose to modify the text to clarify:

• pattern for the CopySource (either "/{bucket}/{key}" or "/{bucket}/{key}?versionId={id}"
• explain the url-encoded requirement - it is not clear, if it requires whole string to be url-encoded, or only the versionId or somehow differently. In my case, I did not url-encode anything.

[rayluo]

Thanks very much for finding the solution and sharing it! We will definitely take a look into this and see how we can improve the documentation.

[vlcinsky]

@rayluo Looking forward to that.
Yesterday I was trying to make copy of an object, having strange characters in the path (bucket-name/test/&?:/feed.txt) and did not succeed. It is either due to my misunderstanding or there is a bug.

If your updated doc explains this, it would be great. Otherwise we shall create separate issue for this particular corner case.

[kyleknap]

With the release of boto3 1.4.0, we introduced a managed copy() method that handles the copying of multipart and nonmultipart objects: https://boto3.readthedocs.io/en/latest/guide/s3.html#copies. Each of these methods make use of CopySource being a dictionary consisting of a bucket, key, and optional version id. Using this dictionary, we will create the proper CopySource key for you so you do not worry how to format or url encode the string anymore. Therefore, resolving the issue.

[rajeget]

@rayluo Looking forward to that.
Yesterday I was trying to make copy of an object, having strange characters in the path (bucket-name/test/&?:/feed.txt) and did not succeed. It is either due to my misunderstanding or there is a bug.

If your updated doc explains this, it would be great. Otherwise we shall create separate issue for this particular corner case.

@vlcinsky the above problem is because of escape char , please see aws/aws-sdk-ruby#969 and draw parallel