CopyObject says NoSuchKey despite GetObject showing that there is such a key

[meirf]

@xibz , here is the more focused version of #1431

Please fill out the sections below to help us address your issue.

Version of AWS SDK for Go?

1.8.11

Version of Go (go version)?

1.8.3

What issue did you see?

CopyObject is returning an error "NoSuchKey: The specified key does not exist." even though GetObject on that exact key is returning that object's data showing that it does exist. This is happening on a tiny fraction of the objects that I'm copying.

My goal is to copy data in the following scheme:

(original)
BUCKET_BAZ

• key_foo
• key_bar

(goal)
BUCKET_BAZ

• key_foo
• key_bar
copy_folder
  • key_foo
  • key_bar

Almost all objects are copied into copy_folder, but certain files always give the NoSuchKey error when being Copy-ed. It's always the same files that are problematic. I don't know what's different about these files.

Steps to reproduce

Unfortunately, I am only able to reproduce this on my own data. I don't know how to generate these problematic files, but once I know which file is problematic, here is how I show the problem in code:

getResp, err := s3Client.GetObject(&s3.GetObjectInput{
	Bucket: aws.String(bucket),
	Key:    aws.String(key),
})
glog.Info(getResp) // prints all details about the file
glog.Info(err) // <nil>

copyResp, err := s3Client.CopyObject(&s3.CopyObjectInput{
	Bucket:     aws.String(bucket),
	CopySource: aws.String(path.Join(bucket, key)),
	Key:        aws.String(path.Join("copy_folder", key)),
})
glog.Info(copyResp) // empty
glog.Info(err) // NoSuchKey error

The CLI aws s3 cp does work on the same key.

All the problematic files I've seen so far have WebsiteRedirectLocation set, but I don't think that's the issue because some of the files that are copied successfully also have that field set.

[jasdel]

Hi @meirf thanks for contacting us. Could you provide an example of the keys that you're running into an issue with? This would greatly help is investigating the issue if you're able to provide an example of object key's that seem to be problematic.

The only thing I think could be causing an issue with the example code above is the path.Join. This function will strip off trailing / characters from the joined value. If any of the objects in your bucket have a trailing / this would be stripped by path.Join and could be the cause of the NoSuchKey error. In addition to cleaning duplicate side-by-side / characters.

[jasdel]

In addition I suggest checking how the CopySource value is built. From the CopyObjectInput docs the CopySource value must be URL-encoded. I don't think the SDK will do this automatically for this field. I'd need to investigate to verify this though.

[meirf]

Here are example keys being rejected:

• a.b.c/foo/bar/baz/foo/bar/baz����.txt
• a.b.c/foo/bar/baz/foo/bar/baz��ퟑ�.txt
• a.b.c/foo/bar/baz/foo/bar/baz/foo+bar
• a.b.c/foo#bar%5Bbaz%5D/0
• a.b.c/foo/bays/foo-%23foo.html
• a.b.c/

Tha path.Join issue you mentioned definitely explains a.b.c/

[jasdel]

Thanks for the additional information. I think part of the issue for those is that the CopySource value must be URL-encoded.

[meirf]

Okay. I now know both categories of errors I was seeing: trailing / being stripped off by path.Join and not url encoding CopySource. I am only url-encoding CopySource - not the Key field and things seem to work. I am closing this with that assumption (should not encode Key).

Thank you.

[jasdel]

Glad to help, let us know if you have any additional feedback, issues, or questions with the SDK.

[open-cipher]

although this issue is referenced for aws-sdk-go, the same thing helped me for aws-sdk-js. Thank you !!