Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

URL for S3 Ireland #1840

Closed
luisramalho opened this Issue · 33 comments
@luisramalho

When Ireland is chosen as the region on S3 the URL is the following:

s3-eu-west-1.amazonaws.com/bucket/[]...

Spree however doesn't seem to have an option to change that... and all the product have the standard URL that does not work.

s3.amazonaws.com/bucket/[]...

Thank you!

@argami

That is directly related with paperclip and the S3 option, not spree.

Look in here s3 about s3_host_alias and this maybe helps you.

@radar
Collaborator

The latest versions of Spree do have a s3_host_alias config option which is designed with a situation like this in mind.

@radar radar closed this
@bobmaerten

Specifiying

Spree.config do |config|
    config.s3_host_alias = "s3-eu-west-1.amazonaws.com"
end

in config/initializers/spree.rb does'nt seem to affect image urls.

Did I miss something in S3 configuration ?

@radar radar reopened this
@radar
Collaborator

@bobmaerten Just to confirm: what version of Spree are you using, and where are you seeing these incorrect URLs? I'll investigate the problem this week most likely.

@bobmaerten

I'm using gem'ed version (1.2.0), and image links aren't correct both on admin and front side (s3.amazonaws.com instead s3-eu-west-1.amazonaws.com).
Changed Gemfile to use master, but no effects. I'll try to setup a fresh master one today.

@bobmaerten

Retried on a fresh install (spree install --edge), pictures are nicely uploaded to my EU bucket, but links are not correctly displayed, neither on admin or front.

@juancabrito

Same issue here, but I'm trying to use the host is Sao Paulo.

@jnettome

Same issue here, but I'm trying to use Sao Paulo (too).

Another thing I see is the image urls. The s3's endpoint has changed:

Generated Image URL: http://s3.amazonaws.com/sexyone-products/app/public/spree/products/2/product/581454_535324936483775_1481621561_n.jpg?1348953443

<Error>
  <Code>PermanentRedirect</Code>
  <Message>
    The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
  </Message>
  <RequestId>58BCC73524477FFE</RequestId>
  <Bucket>sexyone-products</Bucket>
  <HostId>
    qWBrvcug/45zeERwvK1rAKbwi7IMKa391VCBlGLfLSnL46JVVrTQ2ym7JWrNTMiF
  </HostId>
  <Endpoint>sexyone-products.s3.amazonaws.com</Endpoint>
</Error>
@jnettome jnettome referenced this issue in joneslee85/spree-heroku
Merged

URL issues in S3 #1

@jnettome

I wrote a temporarily fix for that problem using spree-heroku gem.
My pull request is here, an the fixed repository (temp) is jnettome/spree-heroku.

I'll try to do the same here. Can I implement something like this? (reference: https://github.com/thoughtbot/paperclip/blob/master/lib/paperclip/storage/s3.rb)

@radar
Collaborator

Ok, I spent all of my morning and about half an hour this afternoon figuring this out. It was fun finally getting to to work and I can understand your frustrations with it. It most certainly wasn't easy.

Here's what I had to do to get this to work:

Spree.config do |config|
  config.use_s3 = true
  config.s3_bucket = '<bucket>'
  config.s3_access_key = "<key>"
  config.s3_secret = "</secret>"
  config.attachment_url = ":s3_eu_url"
  config.s3_host_alias = "s3-eu-west-1.amazonaws.com"
end

Paperclip.interpolates(:s3_eu_url) do |attachment, style|
  "#{attachment.s3_protocol}://#{Spree::Config[:s3_host_alias]}/#{attachment.bucket_name}/#{attachment.path(style).gsub(%r{^/}, "")}"
end

The secret sauce is the attachment_url and s3_host_alias configuration options. Those combined tell Paperclip to interpolate the attachment's information into the correct format for the URL. This will work for any S3 host that isn't the default.

I hope this helps you guys.

@radar radar closed this
@luisramalho

Thanks so much Ryan!

@juancabrito

Thank you Radar

@pavelnikolov

@radar this does not work for me. Where do you put this code? I cannot upload images - the error is

"AWS::S3::Errors::PermanentRedirect (The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.):"

I copied the code you provided in the spree.rb initializer and put requests do not work.

Please advise.
Any help would be much appreciated.

@pavelnikolov

@luisramalho, @juancabrito - were you able to fix this issue with the solution provided by @radar? It does not work for me. I copied:

Spree.config do |config|
  config.use_s3 = true
  config.s3_bucket = ''
  config.s3_access_key = ""
  config.s3_secret = ""
  config.attachment_url = ":s3_eu_url"
  config.s3_host_alias = "s3-eu-west-1.amazonaws.com"
end

Paperclip.interpolates(:s3_eu_url) do |attachment, style|
  "#{attachment.s3_protocol}://#{Spree::Config[:s3_host_alias]}/#{attachment.bucket_name}/#{attachment.path(style).gsub(%r{^/}, "")}"
end

into config/initializers/spree.rb and still nothing. I'm suing spree 1_2_stable and have been trying to work around this issue for few days to no avail. I keep getting:

"AWS::S3::Errors::PermanentRedirect (The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.):"

Any ideas?

@pavelnikolov

Of course I populated my bucket, key and secret - I get this error when I try to upload a new product image.

@radar
Collaborator

@pavelnikolov Please show the stacktrace for your issue.

@pavelnikolov

@radar, thanks for looking into it

[paperclip] Saving attachments.
[paperclip] saving /spree/products/20/original/Solaris_fig2.jpg
   (0.8ms)  rollback transaction
Completed 500 Internal Server Error in 3798ms

AWS::S3::Errors::PermanentRedirect (The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.):
  aws-sdk (1.3.9) lib/aws/core/client.rb:277:in `return_or_raise'
  aws-sdk (1.3.9) lib/aws/core/client.rb:362:in `client_request'
  (eval):3:in `put_object'
  aws-sdk (1.3.9) lib/aws/s3/s3_object.rb:315:in `write'
  paperclip (2.8.0) lib/paperclip/storage/s3.rb:295:in `block in flush_writes'
  paperclip (2.8.0) lib/paperclip/storage/s3.rb:281:in `each'
  paperclip (2.8.0) lib/paperclip/storage/s3.rb:281:in `flush_writes'
  paperclip (2.8.0) lib/paperclip/attachment.rb:214:in `save'
  paperclip (2.8.0) lib/paperclip.rb:472:in `block in save_attached_files'
  paperclip (2.8.0) lib/paperclip.rb:465:in `block in each_attachment'
  paperclip (2.8.0) lib/paperclip.rb:464:in `each'
  paperclip (2.8.0) lib/paperclip.rb:464:in `each_attachment'
  paperclip (2.8.0) lib/paperclip.rb:471:in `save_attached_files'
  activesupport (3.2.9) lib/active_support/callbacks.rb:416:in `_run__1209570969874130156__save__2670435773145347909__callbacks'
  activesupport (3.2.9) lib/active_support/callbacks.rb:405:in `__run_callback'
  activesupport (3.2.9) lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
  activesupport (3.2.9) lib/active_support/callbacks.rb:81:in `run_callbacks'
  activerecord (3.2.9) lib/active_record/callbacks.rb:264:in `create_or_update'
  activerecord (3.2.9) lib/active_record/persistence.rb:84:in `save'
  activerecord (3.2.9) lib/active_record/validations.rb:50:in `save'
  activerecord (3.2.9) lib/active_record/attribute_methods/dirty.rb:22:in `save'
  activerecord (3.2.9) lib/active_record/transactions.rb:259:in `block (2 levels) in save'
  activerecord (3.2.9) lib/active_record/transactions.rb:313:in `block in with_transaction_returning_status'
  activerecord (3.2.9) lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
  activerecord (3.2.9) lib/active_record/transactions.rb:208:in `transaction'
  activerecord (3.2.9) lib/active_record/transactions.rb:311:in `with_transaction_returning_status'
  activerecord (3.2.9) lib/active_record/transactions.rb:259:in `block in save'
  activerecord (3.2.9) lib/active_record/transactions.rb:270:in `rollback_active_record_state!'
  activerecord (3.2.9) lib/active_record/transactions.rb:258:in `save'
  /Users/nikolov/.rvm/gems/ruby-1.9.3-p286/bundler/gems/spree-0d81ee0a2265/core/app/controllers/spree/admin/resource_controller.rb:43:in `create'
  actionpack (3.2.9) lib/action_controller/metal/implicit_render.rb:4:in `send_action'
  actionpack (3.2.9) lib/abstract_controller/base.rb:167:in `process_action'
  actionpack (3.2.9) lib/action_controller/metal/rendering.rb:10:in `process_action'
  actionpack (3.2.9) lib/abstract_controller/callbacks.rb:18:in `block in process_action'
  activesupport (3.2.9) lib/active_support/callbacks.rb:491:in `_run__561421351246939326__process_action__3672537410684143120__callbacks'
  activesupport (3.2.9) lib/active_support/callbacks.rb:405:in `__run_callback'
  activesupport (3.2.9) lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
  activesupport (3.2.9) lib/active_support/callbacks.rb:81:in `run_callbacks'
  actionpack (3.2.9) lib/abstract_controller/callbacks.rb:17:in `process_action'
  actionpack (3.2.9) lib/action_controller/metal/rescue.rb:29:in `process_action'
  actionpack (3.2.9) lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
  activesupport (3.2.9) lib/active_support/notifications.rb:123:in `block in instrument'
  activesupport (3.2.9) lib/active_support/notifications/instrumenter.rb:20:in `instrument'
  activesupport (3.2.9) lib/active_support/notifications.rb:123:in `instrument'
  actionpack (3.2.9) lib/action_controller/metal/instrumentation.rb:29:in `process_action'
  actionpack (3.2.9) lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
  activerecord (3.2.9) lib/active_record/railties/controller_runtime.rb:18:in `process_action'
  actionpack (3.2.9) lib/abstract_controller/base.rb:121:in `process'
  actionpack (3.2.9) lib/abstract_controller/rendering.rb:45:in `process'
  actionpack (3.2.9) lib/action_controller/metal.rb:203:in `dispatch'
  actionpack (3.2.9) lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
  actionpack (3.2.9) lib/action_controller/metal.rb:246:in `block in action'
  actionpack (3.2.9) lib/action_dispatch/routing/route_set.rb:73:in `call'
  actionpack (3.2.9) lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
  actionpack (3.2.9) lib/action_dispatch/routing/route_set.rb:36:in `call'
  journey (1.0.4) lib/journey/router.rb:68:in `block in call'
  journey (1.0.4) lib/journey/router.rb:56:in `each'
  journey (1.0.4) lib/journey/router.rb:56:in `call'
  actionpack (3.2.9) lib/action_dispatch/routing/route_set.rb:601:in `call'
  /Users/nikolov/.rvm/gems/ruby-1.9.3-p286/bundler/gems/spree-0d81ee0a2265/core/lib/spree/core/middleware/redirect_legacy_product_url.rb:13:in `call'
  /Users/nikolov/.rvm/gems/ruby-1.9.3-p286/bundler/gems/spree-0d81ee0a2265/core/lib/spree/core/middleware/seo_assist.rb:27:in `call'
  railties (3.2.9) lib/rails/engine.rb:479:in `call'
  railties (3.2.9) lib/rails/railtie/configurable.rb:30:in `method_missing'
  journey (1.0.4) lib/journey/router.rb:68:in `block in call'
  journey (1.0.4) lib/journey/router.rb:56:in `each'
  journey (1.0.4) lib/journey/router.rb:56:in `call'
  actionpack (3.2.9) lib/action_dispatch/routing/route_set.rb:601:in `call'
  warden (1.1.1) lib/warden/manager.rb:35:in `block in call'
  warden (1.1.1) lib/warden/manager.rb:34:in `catch'
  warden (1.1.1) lib/warden/manager.rb:34:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
  rack (1.4.1) lib/rack/etag.rb:23:in `call'
  rack (1.4.1) lib/rack/conditionalget.rb:35:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/head.rb:14:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/params_parser.rb:21:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/flash.rb:242:in `call'
  rack (1.4.1) lib/rack/session/abstract/id.rb:205:in `context'
  rack (1.4.1) lib/rack/session/abstract/id.rb:200:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/cookies.rb:341:in `call'
  activerecord (3.2.9) lib/active_record/query_cache.rb:64:in `call'
  activerecord (3.2.9) lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
  activesupport (3.2.9) lib/active_support/callbacks.rb:405:in `_run__2122742935584408543__call__2670435773145347909__callbacks'
  activesupport (3.2.9) lib/active_support/callbacks.rb:405:in `__run_callback'
  activesupport (3.2.9) lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
  activesupport (3.2.9) lib/active_support/callbacks.rb:81:in `run_callbacks'
  actionpack (3.2.9) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/reloader.rb:65:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
  railties (3.2.9) lib/rails/rack/logger.rb:32:in `call_app'
  railties (3.2.9) lib/rails/rack/logger.rb:16:in `block in call'
  activesupport (3.2.9) lib/active_support/tagged_logging.rb:22:in `tagged'
  railties (3.2.9) lib/rails/rack/logger.rb:16:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/request_id.rb:22:in `call'
  rack (1.4.1) lib/rack/methodoverride.rb:21:in `call'
  rack (1.4.1) lib/rack/runtime.rb:17:in `call'
  activesupport (3.2.9) lib/active_support/cache/strategy/local_cache.rb:72:in `call'
  rack (1.4.1) lib/rack/lock.rb:15:in `call'
  actionpack (3.2.9) lib/action_dispatch/middleware/static.rb:62:in `call'
  railties (3.2.9) lib/rails/engine.rb:479:in `call'
  railties (3.2.9) lib/rails/application.rb:223:in `call'
  rack (1.4.1) lib/rack/content_length.rb:14:in `call'
  railties (3.2.9) lib/rails/rack/log_tailer.rb:17:in `call'
  rack (1.4.1) lib/rack/handler/webrick.rb:59:in `service'

also the images in the public part are not shown - spree is looking in https://s3.amaz.... instead of https://s3-eu-west-1.amazo....

@pavelnikolov

I just tried with latest version of Spree and it worked

@radar
Collaborator
@jwall149

I had the same problem with Spree 1.3.0 and Singapore AWS, after half a day of debug, problem solved.

The problem:

  • Paperclip uses default s3_host_name options since we don't give it at:

https://github.com/thoughtbot/paperclip/blob/v2.8.0/lib/paperclip/storage/s3.rb
line 152

  • aws_sdk takes s3_end_point option to connect, and that option refers to s3_host_name

My solution is a bit ugly, but it works for mee:

lib/extend/s3_region.rb

module Extend
  module S3Region
    extend ActiveSupport::Concern
    included do
      def self.adapt_s3_region(field, region = nil)
        return if region.nil?
        # Load user defined paperclip settings
        config = Spree::Config
        if config[:use_s3]
          case region
          when :singapore
            self.attachment_definitions[field][:s3_host_name] = "s3-ap-southeast-1.amazonaws.com"
          when :tokyo
            self.attachment_definitions[field][:s3_host_name] = "s3-ap-northeast-1.amazonaws.com"
          when :sydney
            self.attachment_definitions[field][:s3_host_name] = "s3-ap-southeast-2.amazonaws.com"
          end
        end
      end
    end
  end
end

app/models/image_decorator.rb

require 'extend/s3_region'
Spree::Image.class_eval do
  include Extend::S3Region
  adapt_s3_region(:attachment,:singapore)
end
@radar
Collaborator

@jwall149 Are you sure these hacks are required? I thought you could just set s3_host_alias as shown in the "Deploying to Heroku" guide and apply the other tweaks and it would work?

@jwall149

@radar Yes, I'm sure that is necessary for me to work. I'm using spree 1.3.0, paperclip 2.8.0 with aws-sdk 1.3.9

I tried with s3_host_alias but it did not work and have to use that hack.

In addition, I'm not using heroku, I'm with Appfog.

@radar
Collaborator
@ericboehs

Only thing I had to do was set a s3_host_alias and then set attachment_url to ":s3_alias_url". No Paperclip.interpolates required. Nothing else seemed to work. I'm using CloudFront for the host so I don't have to worry about regions.

# config/initializers/spree.rb
Spree.config do |config|
  # ...

  # AWS S3
  config.use_s3          = true
  config.attachment_url  = ":s3_alias_url"
  config.s3_access_key   = ENV['AWS_ACCESS_KEY_ID']
  config.s3_secret       = ENV['AWS_SECRET_ACCESS_KEY']
  config.s3_bucket       = ENV['FOG_DIRECTORY']
  config.s3_host_alias   = ENV['CF_HOST']
end

Also it looks like s3_host_alias can be a Proc so you could have it change the endpoint to say assets[0-3].example.com for non-blocking downloads. I'm using SSL on CloudFront so I'm not using custom subdomains. Maybe someone else can figure out how to write the Proc.

@radar
Collaborator
@ericboehs

Well the current version of Paperclip bundled is 6 months old. And the s3 storage file hasn't been touched in a year. And the logic for s3_alias_url hasn't changed since 2010.

I guess I just found a glitch in the matrix. :)

@ericboehs

Found it! thoughtbot/paperclip@bb2d3b0#commitcomment-156147

Just have to know how to source code dive through git history, and then find the relevant comments on github.

Simple. Just Works.™

@steve21124

ericboehs, when you said found it, do you roll back to that paperclip commit?

@ericboehs

This is what I have in my spree.rb from 3 months ago:

  # AWS S3
  config.use_s3          = true
  config.attachment_url  = ":s3_alias_url"
  config.attachment_path = 'products/:id/:style/:basename.:extension'
  config.s3_access_key   = ENV['AWS_ACCESS_KEY_ID']
  config.s3_secret       = ENV['AWS_SECRET_ACCESS_KEY']
  config.s3_bucket       = ENV['FOG_DIRECTORY']
  config.s3_host_alias   = ENV['CF_HOST'] || "#{ENV['FOG_DIRECTORY']}.s3.amazonaws.com"

CF_HOST is for cloud front else it falls back to FOG_DIR.s3.amazonaws.com.

@lzbikowski-opensource

I tried it without Heroku it doesn't work for me work (Spree 2.1) ....
As a workaround I changed Paperclip defaults (config/application.rb)

    # Paperclip S3 defaults for EU
    config.paperclip_defaults = {
      :storage => :s3,
      :s3_host_name => 's3-eu-west-1.amazonaws.com'
    }

It's not perfect solution but solves the problem for now.

@eljago

I created a bucket in the Sao Paulo region and lzbikowski-opensource's solution worked for me.

It's the only proposed solution that I tried tho.

@artemaminov

In backend set "preferences attachment url = :s3_domain_url" and in application.rb

# Paperclip
config.paperclip_defaults = {
  s3_domain_url: "s3-ap-southeast-1.amazonaws.com",
}

Do not forget to restart server.

@westonplatter

+1 @ericboehs' solution for US S3 buckets,

Spree.config do |config|
  config.use_s3           = true
  config.attachment_url   = ":s3_alias_url"
  config.s3_bucket        = ENV["s3_bucket"]
  config.s3_access_key    = ENV["s3_access_key"]
  config.s3_secret        = ENV["s3_secret"]
  config.s3_host_alias    = "#{ENV['s3_bucket']}.s3.amazonaws.com"
  config.attachment_path  = '/spree/products/:id_:basename_:style.:extension'
end
@rafaqz rafaqz referenced this issue in openfoodfoundation/openfoodnetwork
Closed

Amazon S3 uses only US standard url to retreive images #516

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.