[S3] Bucket name gets duplicated in case of redirect from AWS

[otzy007]

If you set the wrong region for the bucket AWS will redirect you to the right location. The problem is that during the reconnect the bucket name gets duplicated into the host link and the operation will fail.

Example code:

require 'fog'

storage = Fog::Storage.new({
  :provider => 'AWS',
  :aws_secret_access_key => 'secret',
  :aws_access_key_id => 'key',
  region: 'us-west-2'
  })

storage.get_bucket('bucketx')

The output:

fog][WARNING] fog: followed redirect to bucketx.s3-us-west-2.amazonaws.com, connecting to the matching region will be more performant
Excon::Errors::SocketError: hostname "bucketx. bucketx.s3.eu-central-1.amazonaws.com" does not match the server certificate (OpenSSL::SSL::SSLError)
    from /Users/andrei/.rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/openssl/ssl.rb:240:in `post_connection_check'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/ssl_socket.rb:139:in `initialize'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:387:in `new'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:387:in `socket'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:106:in `request_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/mock.rb:47:in `request_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/instrumentor.rb:22:in `request_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/base.rb:15:in `request_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/base.rb:15:in `request_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/base.rb:15:in `request_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:233:in `request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/idempotent.rb:26:in `error_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/base.rb:10:in `error_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/base.rb:10:in `error_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:256:in `rescue in request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:204:in `request'
... 4 levels...
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:204:in `request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/idempotent.rb:26:in `error_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/base.rb:10:in `error_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/base.rb:10:in `error_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:256:in `rescue in request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:204:in `request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/fog-xml-0.1.1/lib/fog/xml/sax_parser_connection.rb:37:in `request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/fog-xml-0.1.1/lib/fog/xml/connection.rb:7:in `request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/fog-aws-0.1.1/lib/fog/aws/storage.rb:587:in `_request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/fog-aws-0.1.1/lib/fog/aws/storage.rb:582:in `request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/fog-aws-0.1.1/lib/fog/aws/storage.rb:613:in `rescue in _request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/fog-aws-0.1.1/lib/fog/aws/storage.rb:587:in `_request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/fog-aws-0.1.1/lib/fog/aws/storage.rb:582:in `request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/fog-aws-0.1.1/lib/fog/aws/requests/storage/get_bucket.rb:43:in `get_bucket'
    from (irb):11
    from /Users/andrei/.rvm/rubies/ruby-2.2.4/bin/irb:11:in `<main>'

[geemus]

It seems quite strange that it appears twice (and that there is a whitespace), I haven't had any other reports of this behavior though. If you want to follow redirects, you should avoid specifying a region in the connection (the default region has some special behaviors that help here I think). Otherwise if you specify the final region explicitly it should also work. Hope that helps, but let me know if you have further question or concerns. Thanks!

[cristianbica]

Issue seems to be here. The redirect response from Amazon looks like:

<Message>
The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
</Message>
<Bucket>BUCKET_NAME</Bucket>
<Endpoint>BUCKET_NAME.s3-us-west-2.amazonaws.com</Endpoint>

So in the code liked above it will set new_params[:bucket_name] to the bucket name and the new_params[:host] to the url to the bucket with it's correct region.

[geemus]

Ah. I think specifying the correct region up front should avoid this (and is probably the ideal solution). Otherwise we can try to fix the redirect logic, but it has been notoriously tricky to get to cooperate historically (there are a lot of edge cases).