Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[S3] Bucket name gets duplicated in case of redirect from AWS #242

Closed
otzy007 opened this Issue Apr 13, 2016 · 3 comments

Comments

Projects
None yet
3 participants
@otzy007
Copy link

otzy007 commented Apr 13, 2016

If you set the wrong region for the bucket AWS will redirect you to the right location. The problem is that during the reconnect the bucket name gets duplicated into the host link and the operation will fail.

Example code:

require 'fog'

storage = Fog::Storage.new({
  :provider => 'AWS',
  :aws_secret_access_key => 'secret',
  :aws_access_key_id => 'key',
  region: 'us-west-2'
  })

storage.get_bucket('bucketx')

The output:

fog][WARNING] fog: followed redirect to bucketx.s3-us-west-2.amazonaws.com, connecting to the matching region will be more performant
Excon::Errors::SocketError: hostname "bucketx. bucketx.s3.eu-central-1.amazonaws.com" does not match the server certificate (OpenSSL::SSL::SSLError)
    from /Users/andrei/.rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/openssl/ssl.rb:240:in `post_connection_check'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/ssl_socket.rb:139:in `initialize'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:387:in `new'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:387:in `socket'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:106:in `request_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/mock.rb:47:in `request_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/instrumentor.rb:22:in `request_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/base.rb:15:in `request_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/base.rb:15:in `request_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/base.rb:15:in `request_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:233:in `request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/idempotent.rb:26:in `error_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/base.rb:10:in `error_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/base.rb:10:in `error_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:256:in `rescue in request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:204:in `request'
... 4 levels...
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:204:in `request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/idempotent.rb:26:in `error_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/base.rb:10:in `error_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/middlewares/base.rb:10:in `error_call'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:256:in `rescue in request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/excon-0.44.4/lib/excon/connection.rb:204:in `request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/fog-xml-0.1.1/lib/fog/xml/sax_parser_connection.rb:37:in `request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/fog-xml-0.1.1/lib/fog/xml/connection.rb:7:in `request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/fog-aws-0.1.1/lib/fog/aws/storage.rb:587:in `_request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/fog-aws-0.1.1/lib/fog/aws/storage.rb:582:in `request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/fog-aws-0.1.1/lib/fog/aws/storage.rb:613:in `rescue in _request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/fog-aws-0.1.1/lib/fog/aws/storage.rb:587:in `_request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/fog-aws-0.1.1/lib/fog/aws/storage.rb:582:in `request'
    from /Users/andrei/.rvm/gems/ruby-2.2.4/gems/fog-aws-0.1.1/lib/fog/aws/requests/storage/get_bucket.rb:43:in `get_bucket'
    from (irb):11
    from /Users/andrei/.rvm/rubies/ruby-2.2.4/bin/irb:11:in `<main>'
@geemus

This comment has been minimized.

Copy link
Member

geemus commented Apr 18, 2016

It seems quite strange that it appears twice (and that there is a whitespace), I haven't had any other reports of this behavior though. If you want to follow redirects, you should avoid specifying a region in the connection (the default region has some special behaviors that help here I think). Otherwise if you specify the final region explicitly it should also work. Hope that helps, but let me know if you have further question or concerns. Thanks!

@geemus geemus closed this Apr 18, 2016

@cristianbica

This comment has been minimized.

Copy link

cristianbica commented Dec 14, 2016

Issue seems to be here. The redirect response from Amazon looks like:

<Message>
The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
</Message>
<Bucket>BUCKET_NAME</Bucket>
<Endpoint>BUCKET_NAME.s3-us-west-2.amazonaws.com</Endpoint>

So in the code liked above it will set new_params[:bucket_name] to the bucket name and the new_params[:host] to the url to the bucket with it's correct region.

@geemus

This comment has been minimized.

Copy link
Member

geemus commented Dec 14, 2016

Ah. I think specifying the correct region up front should avoid this (and is probably the ideal solution). Otherwise we can try to fix the redirect logic, but it has been notoriously tricky to get to cooperate historically (there are a lot of edge cases).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.