New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

repeated bucket name in the URL on AWS and issue with :path_style #228

Open
valerioleo opened this Issue Feb 17, 2016 · 12 comments

Comments

Projects
None yet
7 participants
@valerioleo

valerioleo commented Feb 17, 2016

I am trying to set fog for aws for my rails application, together with CarrierWave.
I created a CarrierWave initializer with al the settings needed.
Files are uploaded without problem in my S3 bucket but there are still two things to fix:

  1. the url saved in my model is different from the one saved in the S3 bucket.
    For example:

The real link of the image (retrieved from the bucket control panel) is:
https://s3.eu-central-1.amazonaws.com/my_bucket_name/my_bucket_name/my_bucket_name/uploads/restaurant/gallery/1/image.jpeg

While in my model the image is saved as:
https://my_APP_name.s3.amazonaws.com/uploads/restaurant/gallery/1/image.jpg
Which is the non styled url (but I explicitly declared :path_style => true)

I tried to change :path_style to false but in that case amazon retrieves me this error:

hostname "my_bucket_name.my_bucket_name.s3.eu-central-1.amazonaws.com" does not match the server certificate (OpenSSL::SSL::SSLError).
I sincerely have no clue about how to fix it.

  1. As you can see, I don't know for which reason but the images are uploaded in several subdirectories (/my_bucket_name/my_bucket_name/my_bucket_name/restaurant/...) and this is driving me crazy.

I attach here the configuration file which I believe is fine. Thanks everybody for the support.

CarrierWave.configure do |config|
    config.fog_provider = 'fog/aws' 
    config.fog_credentials = {
        :provider => 'AWS',
        :aws_access_key_id => 'KEY',
        :aws_secret_access_key => 'SECRET_KEY',
        :region => 'eu-west-1',
        :path_style => true
    }
    config.fog_directory = 'my_bucket_name'    
    config.fog_public = true

end

PS: "my_bucket_name" and "my_app_name" are fantasy names just to help explaining the issue and they are not the real names. Anyway, real names are composed exclusively of lowercase letter.

@valerioleo valerioleo changed the title from repeated bucket name in the URL on AWS and wrong url to repeated bucket name in the URL on AWS and issue with :path_style Feb 17, 2016

@geemus

This comment has been minimized.

Show comment
Hide comment
@geemus

geemus Feb 17, 2016

Member

The error with path-style false may have to do with the bucket name including underscores, as I think this might not match with the wildcard SSL on s3. I'm uncertain about the repeated domain names though. Someone within the carrierwave project may better be able to help there as I don't think it has to do with fog itself, so much as the integration.

Member

geemus commented Feb 17, 2016

The error with path-style false may have to do with the bucket name including underscores, as I think this might not match with the wildcard SSL on s3. I'm uncertain about the repeated domain names though. Someone within the carrierwave project may better be able to help there as I don't think it has to do with fog itself, so much as the integration.

@valerioleo

This comment has been minimized.

Show comment
Hide comment
@valerioleo

valerioleo Feb 17, 2016

The error with path-style false may have to do with the bucket name including underscores

Hi Geemus! Thanks for the reply.
In this issue post I changed both bucket and app name just to make easier to explain the problem.
The original bucket name (it's called "restaurantplates") has neither underscores nor periods, it is just lowercase letters. And the same is for the app name.

valerioleo commented Feb 17, 2016

The error with path-style false may have to do with the bucket name including underscores

Hi Geemus! Thanks for the reply.
In this issue post I changed both bucket and app name just to make easier to explain the problem.
The original bucket name (it's called "restaurantplates") has neither underscores nor periods, it is just lowercase letters. And the same is for the app name.

@geemus

This comment has been minimized.

Show comment
Hide comment
@geemus

geemus Feb 19, 2016

Member

Ah, thanks for that clarification. I knew that sometimes particular names conflicted with the wildcard cert S3 uses, but it doesn't sound like that is the case here after all. Were you able to get in touch with anybody from carrierwave to offer advice? I haven't worked with it in such a long time I worry I may be missing something obvious. Thanks!

Member

geemus commented Feb 19, 2016

Ah, thanks for that clarification. I knew that sometimes particular names conflicted with the wildcard cert S3 uses, but it doesn't sound like that is the case here after all. Were you able to get in touch with anybody from carrierwave to offer advice? I haven't worked with it in such a long time I worry I may be missing something obvious. Thanks!

@karwank

This comment has been minimized.

Show comment
Hide comment
@karwank

karwank Dec 27, 2016

Any progress with this?

karwank commented Dec 27, 2016

Any progress with this?

@RomanKisilenko

This comment has been minimized.

Show comment
Hide comment
@RomanKisilenko

RomanKisilenko Jan 4, 2017

Same issue here, is there any fix known?

RomanKisilenko commented Jan 4, 2017

Same issue here, is there any fix known?

@karwank

This comment has been minimized.

Show comment
Hide comment
@karwank

karwank Jan 4, 2017

It turned out that I had wrong region in configuration. Changing ragion fixed the issue.

karwank commented Jan 4, 2017

It turned out that I had wrong region in configuration. Changing ragion fixed the issue.

@geemus

This comment has been minimized.

Show comment
Hide comment
@geemus

geemus Jan 4, 2017

Member

Yeah, when this issue occurs it usually relates to a failed attempt to follow redirects when the region is set incorrectly for the connection. Unfortunately, they are not true redirects, just a series of inconsistent error messages that we have done our best to turn into working redirects (sometimes it just works, but we have had a number of cases where it ends up doing things like this, unfortunately). Setting the region to match the file is a good way to avoid this issue altogether, otherwise I would welcome help improving the redirect code (but know that I've spent a lot of time struggling with it and not had much luck improving the situation due to various inconsistencies and the difficulty to test all the possibilities).

Member

geemus commented Jan 4, 2017

Yeah, when this issue occurs it usually relates to a failed attempt to follow redirects when the region is set incorrectly for the connection. Unfortunately, they are not true redirects, just a series of inconsistent error messages that we have done our best to turn into working redirects (sometimes it just works, but we have had a number of cases where it ends up doing things like this, unfortunately). Setting the region to match the file is a good way to avoid this issue altogether, otherwise I would welcome help improving the redirect code (but know that I've spent a lot of time struggling with it and not had much luck improving the situation due to various inconsistencies and the difficulty to test all the possibilities).

@scsmith

This comment has been minimized.

Show comment
Hide comment
@scsmith

scsmith Jan 5, 2017

I'm not sure if this is 100% the OP's issue but from what I can see S3 has changed how it's region redirects occur.

Previously if you used a path_style request it would go from s3.amazonaws.com -> eu-west-1.s3.amazonaws.com and make the request again.

Instead now it seems to redirect from s3.amazonaws.com -> bucketname.s3.amazonaws.com.

Because of this Fog can't recognise the region and will continue to sign with the wrong region (us-east-1) by default. It seems like this differs from the logic in the redirect code, as stated, or at least there needs to be a new detection method for signing with the wrong region in addition to what is already present.

scsmith commented Jan 5, 2017

I'm not sure if this is 100% the OP's issue but from what I can see S3 has changed how it's region redirects occur.

Previously if you used a path_style request it would go from s3.amazonaws.com -> eu-west-1.s3.amazonaws.com and make the request again.

Instead now it seems to redirect from s3.amazonaws.com -> bucketname.s3.amazonaws.com.

Because of this Fog can't recognise the region and will continue to sign with the wrong region (us-east-1) by default. It seems like this differs from the logic in the redirect code, as stated, or at least there needs to be a new detection method for signing with the wrong region in addition to what is already present.

@geemus

This comment has been minimized.

Show comment
Hide comment
@geemus

geemus Jan 5, 2017

Member

@scsmith that could very well be. The behavior I saw in the past was that different requests redirected in different ways (which made it really hard to debug or workaround), perhaps that is no longer the case though.

Member

geemus commented Jan 5, 2017

@scsmith that could very well be. The behavior I saw in the past was that different requests redirected in different ways (which made it really hard to debug or workaround), perhaps that is no longer the case though.

@Tranxuanvu

This comment has been minimized.

Show comment
Hide comment
@Tranxuanvu

Tranxuanvu Jun 9, 2017

@scsmith Thanks bro 😃

Tranxuanvu commented Jun 9, 2017

@scsmith Thanks bro 😃

@lustan3216

This comment has been minimized.

Show comment
Hide comment
@lustan3216

lustan3216 Nov 9, 2017

I'm have the same issue.
And i found out its might be AWS redirection problem.
How i solved is change region, i move bucket from Tokyo to Singapore.
It jsut working.

lustan3216 commented Nov 9, 2017

I'm have the same issue.
And i found out its might be AWS redirection problem.
How i solved is change region, i move bucket from Tokyo to Singapore.
It jsut working.

@geemus

This comment has been minimized.

Show comment
Hide comment
@geemus

geemus Nov 9, 2017

Member

@lustan3216 yeah, there have definitely been some weird edge cases relating to redirection. Glad you found a fix and thanks for taking time to share details.

Member

geemus commented Nov 9, 2017

@lustan3216 yeah, there have definitely been some weird edge cases relating to redirection. Glad you found a fix and thanks for taking time to share details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment