s3 bucket existence - unexpected behavior

[Techbrunch]

This bug is similar to #1161:

require 'aws-sdk'

resource = Aws::S3::Resource.new(region: 'us-east-1')

bucket = resource.bucket('healix')
if bucket.exists?
  puts "test"
end

/home/ubuntu/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/aws-sdk-resources-2.5.0/lib/aws-sdk-resources/resource.rb:134:in `rescue in exists?': Aws::S3::Errors::BadRequest
    from /home/ubuntu/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/aws-sdk-resources-2.5.0/lib/aws-sdk-resources/resource.rb:131:in `exists?'
    from bug.rb:6:in `<main>'

If I try with aws-shell:

A client error (NoSuchBucket) occurred when calling the ListObjects operation: The specified bucket does not exist

[Techbrunch]

Hum interesting, I just tried now and it works ? Did you do anything ?

[awood45]

No changes on our end. I can't reproduce this, so it's hard to say why you're seeing this without a wire trace of the call. The bucket also doesn't exist (as of the time of writing), so I don't think it's the same issue we saw before.

[Techbrunch]

Closing it as I cannot reproduce it for now. @awood45 how can I check what request is failing if this happen again in the future ? because there was clearly something going wrong at the time.

[awood45]

If it's consistent over a period of time, run a wire trace of your call. For example:

require 'aws-sdk'

resource = Aws::S3::Resource.new(region: 'us-east-1', http_wire_trace: true)

bucket = resource.bucket('healix')
bucket.exists?

This will output a wire trace of the client calls made, and this would allow you to verify if we are making a well-formed request. If we are, then the issue is with the service. If we aren't making a valid request, that's a bug.

[Techbrunch]

Go it again:

opening connection to bucket.s3.amazonaws.com:443...
opened
starting SSL for bucket.s3.amazonaws.com:443...
SSL established
<- "HEAD / HTTP/1.1\r\nContent-Type: \r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby2/2.5.0 ruby/2.2.3 x86_64-linux resources\r\nX-Amz-Date: 20160816T193602Z\r\nHost: bucket.s3.amazonaws.com\r\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\nAuthorization: AWS4-HMAC-SHA256 Credential=***/20160816/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=72c1f54e43a99e11ae7ec806b083acede862b34b09840bf81b6434fda30515ff\r\nContent-Length: 0\r\nAccept: */*\r\n\r\n"
-> "HTTP/1.1 400 Bad Request\r\n"
-> "x-amz-request-id: 1BBCA6D05881677F\r\n"
-> "x-amz-id-2: ftR5jxUqVEu/sxq8uGJCj1B6jSnRCgWFRZnahhFsGrbbSjv+zgGL9wPlYTlk1K/AdNehp6SP8bE=\r\n"
-> "Content-Type: application/xml\r\n"
-> "Transfer-Encoding: chunked\r\n"
-> "Date: Tue, 16 Aug 2016 19:35:50 GMT\r\n"
-> "Connection: close\r\n"
-> "Server: AmazonS3\r\n"
-> "\r\n"
Conn close
/home/ubuntu/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/aws-sdk-resources-2.5.0/lib/aws-sdk-resources/resource.rb:134:in `rescue in exists?': Aws::S3::Errors::BadRequest
    from /home/ubuntu/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/aws-sdk-resources-2.5.0/lib/aws-sdk-resources/resource.rb:131:in `exists?'
    from reproduce.rb:6:in `<main>'

[awood45]

Trimmed parts of that - interesting that you got exactly that error, I don't see anything off-hand that's broken. However, I can take this to the service team to take a look.

[trevorrowe]

I went HEAD the bucket. I was curious to know what region the bucket is in. The bucket seems to no longer exist. My guess is the bucket is not in us-east-1, but it is being signed as such. Normally the SDK would detect this region-missmatch, but S3 does not return sufficient information. Normally the error would be returned in the response body, but HEAD responses do not have a body.

Also, a HEAD bucket request would normally return an "x-amz-bucket-region" response header. Did you remove this from the error response when copy-pasting it?

[awood45]

After a discussion with Trevor - are you getting this error on a freshly created bucket?

[awood45]

It looks like you're getting some edge case behavior that has to do with recent creation/deletion of the bucket. If the bucket is recently deleted, but did exist in a different region, you'll potentially see the 400 instead of a 404.

This does raise an interesting question about if we can handle this edge case in the #exists? implementation (treat 400 + no x-amz-bucket-region), but it should be VERY rare, and may not always be semantically correct.

I'm going to take this as a feature request (looking for a clean way to handle this edge case), but we may not be able to fix this particular issue. Let me know if you have any other questions or concerns.

[Techbrunch]

@awood45 I have no way to check if the bucket was freshly created but this appear to be the issue. This edge case seems to be handled by aws-shell, maybe you can have a look at how they handle this issue.

[awood45]

Will take a look at that, thanks. Added to feature request backlog.

[BalabhadraVenkat]

When I used s3 endpoint as 'https://us-east-1.s3.amazonaws.com' and upload the content to 'bucket', it used to fail with saying bucket not found. But if I create a dummy bucket 'bucket.us-east-1' it used to work.

After a while, realized that using the endpoint 's3.amazonaws.com' without any region in it, fixed the error.