Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

URL for S3 Ireland #1840

Closed
luisramalho opened this Issue Aug 7, 2012 · 34 comments

Comments

Projects
None yet

When Ireland is chosen as the region on S3 the URL is the following:

s3-eu-west-1.amazonaws.com/bucket/[]...

Spree however doesn't seem to have an option to change that... and all the product have the standard URL that does not work.

s3.amazonaws.com/bucket/[]...

Thank you!

Contributor

argami commented Aug 7, 2012

That is directly related with paperclip and the S3 option, not spree.

Look in here s3 about s3_host_alias and this maybe helps you.

Member

radar commented Aug 7, 2012

The latest versions of Spree do have a s3_host_alias config option which is designed with a situation like this in mind.

@radar radar closed this Aug 7, 2012

Specifiying

Spree.config do |config|
    config.s3_host_alias = "s3-eu-west-1.amazonaws.com"
end

in config/initializers/spree.rb does'nt seem to affect image urls.

Did I miss something in S3 configuration ?

@radar radar reopened this Sep 18, 2012

Member

radar commented Sep 18, 2012

@bobmaerten Just to confirm: what version of Spree are you using, and where are you seeing these incorrect URLs? I'll investigate the problem this week most likely.

I'm using gem'ed version (1.2.0), and image links aren't correct both on admin and front side (s3.amazonaws.com instead s3-eu-west-1.amazonaws.com).
Changed Gemfile to use master, but no effects. I'll try to setup a fresh master one today.

Retried on a fresh install (spree install --edge), pictures are nicely uploaded to my EU bucket, but links are not correctly displayed, neither on admin or front.

Same issue here, but I'm trying to use the host is Sao Paulo.

Same issue here, but I'm trying to use Sao Paulo (too).

Another thing I see is the image urls. The s3's endpoint has changed:

Generated Image URL: http://s3.amazonaws.com/sexyone-products/app/public/spree/products/2/product/581454_535324936483775_1481621561_n.jpg?1348953443

<Error>
  <Code>PermanentRedirect</Code>
  <Message>
    The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
  </Message>
  <RequestId>58BCC73524477FFE</RequestId>
  <Bucket>sexyone-products</Bucket>
  <HostId>
    qWBrvcug/45zeERwvK1rAKbwi7IMKa391VCBlGLfLSnL46JVVrTQ2ym7JWrNTMiF
  </HostId>
  <Endpoint>sexyone-products.s3.amazonaws.com</Endpoint>
</Error>

@jnettome jnettome referenced this issue in joneslee85/spree-heroku Sep 30, 2012

Merged

URL issues in S3 #1

I wrote a temporarily fix for that problem using spree-heroku gem.
My pull request is here, an the fixed repository (temp) is jnettome/spree-heroku.

I'll try to do the same here. Can I implement something like this? (reference: https://github.com/thoughtbot/paperclip/blob/master/lib/paperclip/storage/s3.rb)

Member

radar commented Oct 3, 2012

Ok, I spent all of my morning and about half an hour this afternoon figuring this out. It was fun finally getting to to work and I can understand your frustrations with it. It most certainly wasn't easy.

Here's what I had to do to get this to work:

Spree.config do |config|
  config.use_s3 = true
  config.s3_bucket = '<bucket>'
  config.s3_access_key = "<key>"
  config.s3_secret = "</secret>"
  config.attachment_url = ":s3_eu_url"
  config.s3_host_alias = "s3-eu-west-1.amazonaws.com"
end

Paperclip.interpolates(:s3_eu_url) do |attachment, style|
  "#{attachment.s3_protocol}://#{Spree::Config[:s3_host_alias]}/#{attachment.bucket_name}/#{attachment.path(style).gsub(%r{^/}, "")}"
end

The secret sauce is the attachment_url and s3_host_alias configuration options. Those combined tell Paperclip to interpolate the attachment's information into the correct format for the URL. This will work for any S3 host that isn't the default.

I hope this helps you guys.

@radar radar closed this Oct 3, 2012

Thanks so much Ryan!

Thank you Radar

@radar this does not work for me. Where do you put this code? I cannot upload images - the error is

"AWS::S3::Errors::PermanentRedirect (The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.):"

I copied the code you provided in the spree.rb initializer and put requests do not work.

Please advise.
Any help would be much appreciated.

@luisramalho, @juancabrito - were you able to fix this issue with the solution provided by @radar? It does not work for me. I copied:

Spree.config do |config|
  config.use_s3 = true
  config.s3_bucket = ''
  config.s3_access_key = ""
  config.s3_secret = ""
  config.attachment_url = ":s3_eu_url"
  config.s3_host_alias = "s3-eu-west-1.amazonaws.com"
end
Paperclip.interpolates(:s3_eu_url) do |attachment, style|
  "#{attachment.s3_protocol}://#{Spree::Config[:s3_host_alias]}/#{attachment.bucket_name}/#{attachment.path(style).gsub(%r{^/}, "")}"
end

into config/initializers/spree.rb and still nothing. I'm suing spree 1_2_stable and have been trying to work around this issue for few days to no avail. I keep getting:

"AWS::S3::Errors::PermanentRedirect (The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.):"

Any ideas?

Of course I populated my bucket, key and secret - I get this error when I try to upload a new product image.

Member

radar commented Nov 20, 2012

@pavelnikolov Please show the stacktrace for your issue.

@radar, thanks for looking into it

[paperclip] Saving attachments.
[paperclip] saving /spree/products/20/original/Solaris_fig2.jpg
   (0.8ms)  rollback transaction
Completed 500 Internal Server Error in 3798ms
AWS::S3::Errors::PermanentRedirect (The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.):
  aws-sdk (1.3.9) lib/aws/core/client.rb:277:in `return_or_raise'
  aws-sdk (1.3.9) lib/aws/core/client.rb:362:in `client_request'
  (eval):3:in `put_object'
  aws-sdk (1.3.9) lib/aws/s3/s3_object.rb:315:in `write'
  paperclip (2.8.0) lib/paperclip/storage/s3.rb:295:in `block in flush_writes'
  paperclip (2.8.0) lib/paperclip/storage/s3.rb:281:in `each'
  paperclip (2.8.0) lib/paperclip/storage/s3.rb:281:in `flush_writes'
  paperclip (2.8.0) lib/paperclip/attachment.rb:214:in `save'
  paperclip (2.8.0) lib/paperclip.rb:472:in `block in save_attached_files'
  paperclip (2.8.0) lib/paperclip.rb:465:in `block in each_attachment'
  paperclip (2.8.0) lib/paperclip.rb:464:in `each'
  paperclip (2.8.0) lib/paperclip.rb:464:in `each_attachment'
  paperclip (2.8.0) lib/paperclip.rb:471:in `save_attached_files'
  activesupport (3.2.9) lib/active_support/callbacks.rb:416:in `_run__1209570969874130156__save__2670435773145347909__callbacks'
  activesupport (3.2.9) lib/active_support/callbacks.rb:405:in `__run_callback'
  activesupport (3.2.9) lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
  activesupport (3.2.9) lib/active_support/callbacks.rb:81:in `run_callbacks'
  activerecord (3.2.9) lib/active_record/callbacks.rb:264:in `create_or_update'
  activerecord (3.2.9) lib/active_record/persistence.rb:84:in `save'
  activerecord (3.2.9) lib/active_record/validations.rb:50:in `save'
  activerecord (3.2.9) lib/active_record/attribute_methods/dirty.rb:22:in `save'
  activerecord (3.2.9) lib/active_record/transactions.rb:259:in `block (2 levels) in save'
  activerecord (3.2.9) lib/active_record/transactions.rb:313:in `block in with_transaction_returning_status'
  activerecord (3.2.9) lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
  activerecord (3.2.9) lib/active_record/transactions.rb:208:in `transaction'
  activerecord (3.2.9) lib/active_record/transactions.rb:311:in `with_transaction_returning_status'
  activerecord (3.2.9) lib/active_record/transactions.rb:259:in `block in save'
  activerecord (3.2.9) lib/active_record/transactions.rb:270:in `rollback_active_record_state!'
  activerecord (3.2.9) lib/active_record/transactions.rb:258:in `save'
  /Users/nikolov/.rvm/gems/ruby-1.9.3-p286/bundler/gems/spree-0d81ee0a2265/core/app/controllers/spree/admin/resource_controller.rb:43:in `create'
  actionpack (3.2.9) lib/action_controller/metal/implicit_render.rb:4:in `send_action'
  actionpack (3.2.9) lib/abstract_controller/base.rb:167:in `process_action'
  actionpack (3.2.9) lib/action_controller/metal/rendering.rb:10:in `process_action'
  actionpack (3.2.9) lib/abstract_controller/callbacks.rb:18:in `block in process_action'
  activesupport (3.2.9) lib/active_support/callbacks.rb:491:in `_run__561421351246939326__process_action__3672537410684143120__callbacks'
  activesupport (3.2.9) lib/active_support/callbacks.rb:405:in `__run_callback'
  activesupport (3.2.9) lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
  activesupport (3.2.9) lib/active_support/callbacks.rb:81:in `run_callbacks'
  actionpack (3.2.9) lib/abstract_controller/callbacks.rb:17:in `process_action'
  actionpack (3.2.9) lib/action_controller/metal/rescue.rb:29:in `process_action'
  actionpack (3.2.9) lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
  activesupport (3.2.9) lib/active_support/notifications.rb:123:in `block in instrument'
  activesupport (3.2.9) lib/active_support/notifications/instrumenter.rb:20:in `instrument'
  activesupport (3.2.9) lib/active_support/notifications.rb:123:in `instrument'
  actionpack (3.2.9) lib/action_controller/metal/instrumentation.rb:29:in `process_action'
  actionpack (3.2.9) lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
  activerecord (3.2.9) lib/active_record/railties/controller_runtime.rb:18:in `process_action'
  actionpack (3.2.9) lib/abstract_controller/base.rb:121:in `process'
  actionpack (3.2.9) lib/abstract_controller/rendering.rb:45:in `process'
  actionpack (3.2.9) lib/action_controller/metal.rb:203:in `dispatch'
  actionpack (3.2.9) lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
  actionpack (3.2.9) lib/action_controller/metal.rb:246:in `block in action'
  actionpack (3.2.9) lib/action_dispatch/routing/route_set.rb:73:in `call'
  actionpack (3.2.9) lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
  actionpack (3.2.9) lib/action_dispatch/routing/route_set.rb:36:in `call'
  journey (1.0.4) lib/journey/router.rb:68:in `block in call'
  journey (1.0.4) lib/journey/router.rb:56:in `each'
  journey (1.0.4) lib/journey/router.rb:56:in `call'
  actionpack (3.2.9) lib/action_dispatch/routing/route_set.rb:601:in `call'
  /Users/nikolov/.rvm/gems/ruby-1.9.3-p286/bundler/gems/spree-0d81ee0a2265/core/lib/spree/core/middleware/redirect_legacy_product_url.rb:13:in `call'
  /Users/nikolov/.rvm/gems/ruby-1.9.3-p286/bundler/gems/spree-0d81ee0a2265/core/lib/spree/core/middleware/seo_assist.rb:27:in `call'
  railties (3.2.9) lib/rails/engine.rb:479:in `call'
  railties (3.2.9) lib/rails/railtie/configurable.rb:30:in `method_missing'
  journey (1.0.4) lib/journey/router.rb:68:in `block in call'
  journey (1.0.4) lib/journey/router.rb:56:in `each'
  journey (1.0.4) lib/journey/router.rb:56:in `call'
  actionpack (3.2.9) lib/action_dispatch/routing/route_set.rb:601:in `call'
  warden (1.1.1) lib/warden/manager.rb:35:in `block in call'
  warden (1.1.1) lib/warden/manager.rb:34:in `catch'
  warden (1.1.1) lib/warden/manager.rb:34:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
  rack (1.4.1) lib/rack/etag.rb:23:in `call'
  rack (1.4.1) lib/rack/conditionalget.rb:35:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/head.rb:14:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/params_parser.rb:21:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/flash.rb:242:in `call'
  rack (1.4.1) lib/rack/session/abstract/id.rb:205:in `context'
  rack (1.4.1) lib/rack/session/abstract/id.rb:200:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/cookies.rb:341:in `call'
  activerecord (3.2.9) lib/active_record/query_cache.rb:64:in `call'
  activerecord (3.2.9) lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
  activesupport (3.2.9) lib/active_support/callbacks.rb:405:in `_run__2122742935584408543__call__2670435773145347909__callbacks'
  activesupport (3.2.9) lib/active_support/callbacks.rb:405:in `__run_callback'
  activesupport (3.2.9) lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
  activesupport (3.2.9) lib/active_support/callbacks.rb:81:in `run_callbacks'
  actionpack (3.2.9) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/reloader.rb:65:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
  railties (3.2.9) lib/rails/rack/logger.rb:32:in `call_app'
  railties (3.2.9) lib/rails/rack/logger.rb:16:in `block in call'
  activesupport (3.2.9) lib/active_support/tagged_logging.rb:22:in `tagged'
  railties (3.2.9) lib/rails/rack/logger.rb:16:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/request_id.rb:22:in `call'
  rack (1.4.1) lib/rack/methodoverride.rb:21:in `call'
  rack (1.4.1) lib/rack/runtime.rb:17:in `call'
  activesupport (3.2.9) lib/active_support/cache/strategy/local_cache.rb:72:in `call'
  rack (1.4.1) lib/rack/lock.rb:15:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/static.rb:62:in `call'
  railties (3.2.9) lib/rails/engine.rb:479:in `call'
  railties (3.2.9) lib/rails/application.rb:223:in `call'
  rack (1.4.1) lib/rack/content_length.rb:14:in `call'
  railties (3.2.9) lib/rails/rack/log_tailer.rb:17:in `call'
  rack (1.4.1) lib/rack/handler/webrick.rb:59:in `service'

also the images in the public part are not shown - spree is looking in https://s3.amaz.... instead of https://s3-eu-west-1.amazo....

I just tried with latest version of Spree and it worked

Member

radar commented Dec 10, 2012

Great, thanks for letting us know!

On Mon, Dec 10, 2012 at 1:45 AM, pavelnikolov notifications@github.comwrote:

I just tried with latest version of Spree and it worked


Reply to this email directly or view it on GitHubhttps://github.com/spree/spree/issues/1840#issuecomment-11171617.

jwall149 commented Jan 1, 2013

I had the same problem with Spree 1.3.0 and Singapore AWS, after half a day of debug, problem solved.

The problem:

  • Paperclip uses default s3_host_name options since we don't give it at:

https://github.com/thoughtbot/paperclip/blob/v2.8.0/lib/paperclip/storage/s3.rb
line 152

  • aws_sdk takes s3_end_point option to connect, and that option refers to s3_host_name

My solution is a bit ugly, but it works for mee:

lib/extend/s3_region.rb

module Extend
  module S3Region
    extend ActiveSupport::Concern
    included do
      def self.adapt_s3_region(field, region = nil)
        return if region.nil?
        # Load user defined paperclip settings
        config = Spree::Config
        if config[:use_s3]
          case region
          when :singapore
            self.attachment_definitions[field][:s3_host_name] = "s3-ap-southeast-1.amazonaws.com"
          when :tokyo
            self.attachment_definitions[field][:s3_host_name] = "s3-ap-northeast-1.amazonaws.com"
          when :sydney
            self.attachment_definitions[field][:s3_host_name] = "s3-ap-southeast-2.amazonaws.com"
          end
        end
      end
    end
  end
end

app/models/image_decorator.rb

require 'extend/s3_region'
Spree::Image.class_eval do
  include Extend::S3Region
  adapt_s3_region(:attachment,:singapore)
end
Member

radar commented Jan 3, 2013

@jwall149 Are you sure these hacks are required? I thought you could just set s3_host_alias as shown in the "Deploying to Heroku" guide and apply the other tweaks and it would work?

jwall149 commented Jan 3, 2013

@radar Yes, I'm sure that is necessary for me to work. I'm using spree 1.3.0, paperclip 2.8.0 with aws-sdk 1.3.9

I tried with s3_host_alias but it did not work and have to use that hack.

In addition, I'm not using heroku, I'm with Appfog.

Member

radar commented Jan 4, 2013

Ok then. That's unfortunate.

On Thu, Jan 3, 2013 at 11:45 PM, Tonny notifications@github.com wrote:

@radar https://github.com/radar Yes, I'm sure that is necessary for me
to work. I'm using spree 1.3.0, paperclip 2.8.0 with aws-sdk 1.3.9

I tried with s3_host_alias but it did not work and have to use that hack.


Reply to this email directly or view it on GitHubhttps://github.com/spree/spree/issues/1840#issuecomment-11842531.

Only thing I had to do was set a s3_host_alias and then set attachment_url to ":s3_alias_url". No Paperclip.interpolates required. Nothing else seemed to work. I'm using CloudFront for the host so I don't have to worry about regions.

# config/initializers/spree.rb
Spree.config do |config|
  # ...

  # AWS S3
  config.use_s3          = true
  config.attachment_url  = ":s3_alias_url"
  config.s3_access_key   = ENV['AWS_ACCESS_KEY_ID']
  config.s3_secret       = ENV['AWS_SECRET_ACCESS_KEY']
  config.s3_bucket       = ENV['FOG_DIRECTORY']
  config.s3_host_alias   = ENV['CF_HOST']
end

Also it looks like s3_host_alias can be a Proc so you could have it change the endpoint to say assets[0-3].example.com for non-blocking downloads. I'm using SSL on CloudFront so I'm not using custom subdomains. Maybe someone else can figure out how to write the Proc.

Member

radar commented Mar 27, 2013

Interesting... maybe Paperclip fixed their code?

On Wed, Mar 27, 2013 at 10:57 AM, Eric Boehs notifications@github.comwrote:

Only thing I had to do was set a s3_host_alias and then set attachment_urlto
":s3_alias_url". No Paperclip.interpolates required. Nothing else seemed
to work. I'm using CloudFront for the host so I don't have to worry about
regions.

Spree.config do |config|

...

AWS S3

config.use_s3 = true
config.attachment_url = ":s3_alias_url"
config.s3_access_key = ENV['AWS_ACCESS_KEY_ID']
config.s3_secret = ENV['AWS_SECRET_ACCESS_KEY']
config.s3_bucket = ENV['FOG_DIRECTORY']
config.s3_host_alias = ENV['CF_HOST']end


Reply to this email directly or view it on GitHubhttps://github.com/spree/spree/issues/1840#issuecomment-15496631
.

Ryan Bigg
Community Manager
Spree Commerce, Inc.

Register now for SpreeConf
May 20-21 in Washington, D.C.
http://spreeconf.com

Well the current version of Paperclip bundled is 6 months old. And the s3 storage file hasn't been touched in a year. And the logic for s3_alias_url hasn't changed since 2010.

I guess I just found a glitch in the matrix. :)

Found it! thoughtbot/paperclip@bb2d3b0#commitcomment-156147

Just have to know how to source code dive through git history, and then find the relevant comments on github.

Simple. Just Works.™

ericboehs, when you said found it, do you roll back to that paperclip commit?

This is what I have in my spree.rb from 3 months ago:

  # AWS S3
  config.use_s3          = true
  config.attachment_url  = ":s3_alias_url"
  config.attachment_path = 'products/:id/:style/:basename.:extension'
  config.s3_access_key   = ENV['AWS_ACCESS_KEY_ID']
  config.s3_secret       = ENV['AWS_SECRET_ACCESS_KEY']
  config.s3_bucket       = ENV['FOG_DIRECTORY']
  config.s3_host_alias   = ENV['CF_HOST'] || "#{ENV['FOG_DIRECTORY']}.s3.amazonaws.com"

CF_HOST is for cloud front else it falls back to FOG_DIR.s3.amazonaws.com.

I tried it without Heroku it doesn't work for me work (Spree 2.1) ....
As a workaround I changed Paperclip defaults (config/application.rb)

    # Paperclip S3 defaults for EU
    config.paperclip_defaults = {
      :storage => :s3,
      :s3_host_name => 's3-eu-west-1.amazonaws.com'
    }

It's not perfect solution but solves the problem for now.

eljago commented Nov 26, 2013

I created a bucket in the Sao Paulo region and lzbikowski-opensource's solution worked for me.

It's the only proposed solution that I tried tho.

In backend set "preferences attachment url = :s3_domain_url" and in application.rb

# Paperclip
config.paperclip_defaults = {
  s3_domain_url: "s3-ap-southeast-1.amazonaws.com",
}

Do not forget to restart server.

Contributor

westonplatter commented Aug 28, 2014

+1 @ericboehs' solution for US S3 buckets,

Spree.config do |config|
  config.use_s3           = true
  config.attachment_url   = ":s3_alias_url"
  config.s3_bucket        = ENV["s3_bucket"]
  config.s3_access_key    = ENV["s3_access_key"]
  config.s3_secret        = ENV["s3_secret"]
  config.s3_host_alias    = "#{ENV['s3_bucket']}.s3.amazonaws.com"
  config.attachment_path  = '/spree/products/:id_:basename_:style.:extension'
end

This works currently: #7252 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment