copy_to and move_to on Aws::S3::Object don't work with access point ARNs

[roccoblues]

• I've gone though Developer Guide for v3 and API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug
copy_to and move_to on Aws::S3::Object don't work with access point ARNs.

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version
aws-sdk-s3

Version of Ruby, OS environment
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]

To Reproduce (observed behavior)

client = Aws::S3::Client.new(
  region:            ENV['AWS_REGION'],
  access_key_id:     ENV['AWS_ACCESS_KEY_ID'],
  secret_access_key: ENV['AWS_SECRET_ACCESS_KEY']
)

bucket_arn = "arn:aws:s3:#{ENV['AWS_REGION']}:#{ENV['AWS_ACCOUNT_ID']}:accesspoint/#{ENV['AWS_ACCESS_POINT']}"
key = 'foo'
new_key = 'bar'

object = Aws::S3::Object.new(bucket_arn, key, client: client)
object.exists?
object.move_to("#{bucket_arn}/#{new_key}")

irb(main):040:0> object = Aws::S3::Object.new(bucket_arn, key, client: client)
=> #<Aws::S3::Object:0x00000000090191f0 @bucket_name="arn:aws:s3:eu-central-1:0000000:accesspoint/images", @key="preview_PIWRiV1BwPEq-zmn1g...
irb(main):041:0> object.exists?
=> true
irb(main):042:0> object.move_to("#{bucket_arn}/#{new_key}")
Traceback (most recent call last):
        1: from (irb):42
ArgumentError (Missing ARN accesspoint name.)

Expected behavior
No ArgumentError should be raised and the copy_to / move_to operation should succeed.

[mullermp]

Thanks for opening up an issue. I'm taking a look. As a general FYI, Resource models are unmaintained because they're buggy and code generated around hand-maintained definition file that in itself has limitations.

Have you tried with the client API copy_object method?

[alextwoods]

Additionally - you can pass bucket and key to the copy_to method and those should work. The underying issue is that we don't have support for parsing the arbitrary / in both bucket and key in our Object Copier.

This should work:

object.move_to(bucket: bucket_arn, key: new_key)

[mullermp]

Please try the workarounds (client API with no resource model usage or passing bucket to move_to). This is unfortunately marked as "wontfix" because fixing resource models involves extensive monkey patching and other potentially breaking hacks.

[roccoblues]

Thanks for the fast response!

@alextwoods I tried your suggestion but it then fails with:

Aws::S3::Errors::InvalidArgument (Invalid resource in copy source ARN)

I guess the Aws::S3::Object.new also can't handle a bucket ARN.

Unfortunately I can't easily switch to the client API because the higher level library we use in a legacy project (https://github.com/sorentwo/carrierwave-aws) is build around the resource model.
But I'll see if I can come up with a PR for them or if I can convince our ops people to go without an accesspoint.

Again, thanks for you fast help. Feel free to close this issue.

[mullermp]

Actually, I think you can still use resource models for this, provided you initialize Bucket and not Object. Here is the code for Bucket initialize. Are you able to create an instance of bucket with the bucket name as the ARN, and then fetch the object from there? Bucket names are used for the host name, and object is just a path. Try Aws::S3::Bucket.new(arn, client: client). Example usage in a test is here.

If this doesn't work, I would agree, either not use accesspoint with resource models or somehow find a way to use the official client API.

Just a bit of background, S3 was insistent on making this a client side feature. Our Client APIs handle this behavior through our plugin system but the hand written resource API models do not handle these kinds of abstractions. You can see bucket.rb above as a monkey patch.

[roccoblues]

Hi @mullermp, the bucket version also doesn't work:

bucket = Aws::S3::Bucket.new(bucket_arn, client: client)
object = bucket.object(key)
irb(main):014:0> object.exists?
=> true
irb(main):015:0> object.move_to(bucket: bucket_arn, key: new_key)
Traceback (most recent call last):
        1: from (irb):15
Aws::S3::Errors::InvalidArgument (Invalid resource in copy source ARN)

Anyway, we have switched to a version without accesspoint now and everthing is working as expected.

[alextwoods]

@roccoblues - Sorry that was still not working and that you had to switch to a version without accesspoint.

When I test locally I the code that I suggested was working:

bucket_arn = 'arn:aws:s3:us-west-2:123456789012:accesspoint/myendpoint'
key = 'foo'
new_key = 'bar'

object = Aws::S3::Object.new(bucket_arn, key, client: client)
object.exists?
object.move_to(bucket: bucket_arn, key: new_key)

This results in the following SDK calls:

[Aws::S3::Client 200 0.009667 0 retries] copy_object(bucket:"arn:aws:s3:us-west-2:123456789012:accesspoint/myendpoint",key:"bar",copy_source:"arn:aws:s3:us-west-2:123456789012:accesspoint/myendpoint/foo")  
[Aws::S3::Client 200 0.00053 0 retries] delete_object(bucket:"arn:aws:s3:us-west-2:123456789012:accesspoint/myendpoint",key:"foo")  

Do you have any more sample code or any other differences?