Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

AWS::S3::Errors::SignatureDoesNotMatch #817

Closed
dbachet opened this Issue · 27 comments

10 participants

@dbachet

I'm migrating my app from Rails 3.0.9 to 3.2.3 / ruby 1.9.3. I was using paperclip(2.3.11) with aws-s3 gem to store my pictures.

Now I want to use the last version of paperclip(3.0.1) with aws-sdk gem.

When trying to upload a file, I get the error:

AWS::S3::Errors::SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your key and signing method.

It's creating the original style image on the S3 bucket, but not the other styles.

Here is also what I get from console:

    [paperclip] Saving attachments.
    [paperclip] saving /presentation_pictures/43/original/Image_1.png
    [AWS S3 200 0.813272 0 retries] put_object(:acl=>:public_read,:bucket_name=>"*******_dev",:content_type=>"image/png",:data=>#<Paperclip::UploadedFileAdapter:0x2e144b4 @target=#<ActionDispatch::Http::UploadedFile:0x2dbb1fc @original_filename="Image 1.png", @content_type="image/png", @headers="Content-Disposition: form-data; name=\"presentation_picture[picture]\"; filename=\"Image 1.png\"\r\nContent-Type: image/png\r\n", @tempfile=#<File:/var/folders/tb/tbmv4LE9EwGNPr1QqU2S0E+++TI/-Tmp-/RackMultipart20120407-33502-1gajfe9>>, @tempfile=#<File:/var/folders/tb/tbmv4LE9EwGNPr1QqU2S0E+++TI/-Tmp-/Image 1.png20120407-33502-an4dcy>>,:key=>"presentation_pictures/43/original/Image_1.png")

A downgrade to paperclip 3.0.0 solved the problem. That made me think that it can be a paperclip issue ? :s

So I write here :)

I've set my aws.yml file as:

    development: &development
      bucket: bucket_name_for_dev
      access_key_id: *****
      secret_access_key: *******
    staging:
      <<: *development
      bucket: bucket_name_for_staging

    production:
      <<: *development
      bucket: bucket_name_for_prod

my model contains :

    has_attached_file :picture, :styles => { :medium => "300x300>", :thumb => "50x50>" }, :default_url => 'picture_missing.png',
         :storage => :s3,
         :bucket => "bucket_name_for_dev",
         :s3_credentials => Rails.root.join("config/aws.yml"),
         :path => "/presentation_pictures/:id/:style/:filename",
         :url  => ":s3_eu_url"

    attr_accessible :picture
    attr_accessor :picture_url

I'll stick to 3.0.0, then it's not a prior problem to me. But if I can help you, don't hesitate to ask me information if you think it's a paperclip issue.

Today, I'm not able to launch tests as... I don't know how :/ But I could try!

Issue happened with:

Ruby 1.9.3p125
Rails 3.2.3
ImageMagick 6.6.9-4
paperclip 3.0.1
aws-sdk 1.3.9
nokogiri 1.4.7

Thx a lot for your job!

@jonhyman

I had this same issue this afternoon after upgrading. Downgrading to 3.0.0 worked for me.

@sikachu
Owner

Hi guys,

Did Paperclip 3.0.2 solve this problem for you? I think this is related to the binary file open.

@jonhyman

I was on master 6 days ago when it was failing (had to be on master because I fell victim to the Rack::Test::UploadedFile problem). Do you think a commit in the last few days has fixed it? I can give it a shot today/tomorrow.

@sikachu
Owner

Do you mind giving me the ref you're pointing to? (in your Gemfile.lock)

@jonhyman

Good call. I reverted from a2a4c7a

@dbachet

Hi,

By my side, the 3.0.2 version didn't fix the problem. :(

@sikachu
Owner

Hey guys,

I've been trying to replicate your problems on my side for a while, using the newly generated app with Paperclip from current master. I can't really replicate the problem, and not sure what caused the issue.

So, looking from the configuration, are you guys both using the EU bucket? Did you guys set the host value in your configuration?

I really would like to resolve this problem for you guys. Do you mind creating a app that replicate the problem? Also, the new version of aws-sdk was released as well, so you might want to try it to see if it fixes your problem.

Thank you for your patience.

@jonhyman

My configuration is simply this. My buckets are all in the US Standard region (US-EAST-1, I believe). I am on aws-sdk 1.3.9.

{:storage => :s3,
                                    :bucket => "mybucketname",
                                    :s3_credentials => ...,
                                    :path => ":class/:attachment/:hash/:style.:extension",
                                    :hash_secret => ...}

What host value should I try configuring?

@sikachu
Owner
@garyharan

The same issue started popping up for me today. I get validation errors if I fall back to 3.0.0. Anyone have a solution for this?

@garyharan

The following fails for me unless I remove the line marked with OFFENDING LINE in the comment:

VALID_CONTENT_TYPES = %w(image/jpg image/jpeg image/png image/gif)
has_attached_file :logo,
:styles => {:medium => '300x300>'}, # OFFENDING LINE
:storage => :s3,
:s3_credentials => Rails.root.join("config","s3.yml"),
:path => "/company_logos/:id/:filename"
validates_attachment_content_type :logo, :content_type => VALID_CONTENT_TYPES, :message => "should be \"jpg, jpeg, png or gif\""

I hope this helps find the error.

@granoff

Fwiw, I am experience this problem as well. I tried v3.0.0 but it ddn't work locally (my Mac). When I went to 3.0.2, the problem went away locally, but when I pushed to Heroku, the error came back. I cannot remove my :styles directive as it is rather complex and important to how we use S3. I am at a loss as to what to try next. Hopefully, some solution will become apparent! :-)

@granoff

In a final act of desperation (it was late :-) ) I heeded a warning in the readme that using the master branch as the gem source, rather than a released version, might resolve issues.... so I tried it, and it worked!

That doesn't leave me with a really warm feeling because for a production system I'd rather not rely on a bleeding edge repo, but that's the nature of this ecosystem, isn't it? :-)

The issue noted in the above link (a fork of paperclip) addresses something which may or may not be part of this repo... I have not yet culled the most recent commits for it.

@NicoArbogast

Hello, I think I encountered the same issue (as far as I understand) and managed to get everything to work perfectly fixing the following line (NicoArbogast@66441fc). I found the explanation/answer in here igor-alexandrov/paperclip-aws#10.

@jyurek
Owner

@granoff, we try to release regularly (bi-weekly, actually), so if master works for you now it should be in a released version soon. @garyharan, does master solve the problem for you as well?

@NicoArbogast, that's great that that solves the problem. Do you happen to have a test for that change, though? I'd have to assume not since we had such a hard time even reproducing the problem.

@NicoArbogast

Hi, I am sorry I don't have any test for that change. It was an issue I encountered only on my prod env. I tried the patch following the discussion here igor-alexandrov/paperclip-aws#10 and it worked. I'm sorry I did not have time to dig deeper into this issue.

@sikachu
Owner

Oh, I think I can see this problem on *nix system. Let me check if we have covered it.

So, the reason is that using file command in Linux would return this:

> `file -b --mime-type paperclip.xls`
 => "application/vnd.ms-office\n" 

We need to strip that \n.

@garyharan

The only place I had the issue was on my staging server and right now I have users testing it so I won't be able to check until tomorrow.

@sikachu
Owner
@sikachu
Owner

I've add a test to prevent regression in 3d814fb. 3.0.3 will go out soon.

@sikachu sikachu closed this
@jpadvo

The patch @NicoArbogast recommended works for me, but it doesn't look like it is included in 3.03 or 3.04, I had to use his repo or download the gem and apply the fix myself.

@sikachu sikachu reopened this
@sikachu
Owner

Reopen this. I'm going to have a look at this one.

@sikachu
Owner

@jpadvo I've tried to reproduce this and still unable to. I've also added a test case to make sure that we have stripped newline character from file content type.

Can I have more information from you?

  • What's the OS are you using?
  • What version of file command (file --version)
  • Can you test this command against a file in your directory, and post me the result?
    ruby -e "puts \`file -b --mime test.png\`.inspect"

Thank you.

@jpadvo

Sure, thanks for looking into it!

I'm using OS X Lion.

Output of the first command:

$ file --version
file-5.04
magic file from /usr/share/file/magic

...and the second, on two different formats of image file:

$ ruby -e "puts \`file -b --mime uZXKQ.jpeg\`.inspect"
"image/jpeg; charset=binary\n"

$ ruby -e "puts \`file -b --mime dZQMk.png\`.inspect"
"image/png; charset=binary\n"
@fjg fjg referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@fjg fjg referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@nzifnab

Hrm... did this ever get fixed? I've upgraded one of our apps to Rails 3.2 and have been running into this error with S3 on both heroku, and locally on my Mac. I followed the stacktrace to this line in paperclip:

https://github.com/thoughtbot/paperclip/blob/master/lib/paperclip/storage/s3.rb#L306

Where I logged the value of write_options and see that it is:

write_options = {:content_type=>"image/jpeg\r", :acl=>:public_read}

Changing the linked line from paperclip's gem to:

write_options = {
  content_type => file.content_type.to_s.chomp,
  :acl => acl
}

Fixes the problem for me. Looking above at this issue it seems like people had already pinpointed the source of the problem (content_type was getting a newline character), is there a specific reason it was never fixed in paperclip's code?

@nzifnab

After further exploration I've narrowed the root of the problem to this:

https://github.com/thoughtbot/paperclip/blob/master/lib/paperclip/io_adapters/uploaded_file_adapter.rb#L18

Where that @target variable is:

#<ActionDispatch::Http::UploadedFile:0x0000010341a218 @original_filename="myimg.png", @content_type="image/png\r", @headers=...<SNIP>
@nzifnab nzifnab referenced this issue from a commit in nzifnab/paperclip
@nzifnab nzifnab Rack gets a newline at the end of the content_type in certain instanc…
…es: 'image/jpeg\r' which causes S3 to be unable to accept the file. Fixes #817
92007c3
@nzifnab

Hmm. We have two versions of file upload: One of them sends the file to another one of our apps via Net::HTTP::Multipart (something like that), and then that core app handles the upload with paperclip, and that is the one that was getting the carriage return appended to content_type. This was fixed by pointing the paperclip gem to my fork (with the fix above). We upgraded another of our apps which also does uploads, but does them right in the app (not sending them to the core service like the first app). I forgot to upgrade the gem to point to my fork but it appears to be working just fine. They both send to S3, and they both use the same S3 credentials and such.

Maybe it's the Net::HTTP version of the file that's getting a newline added at some time during the process. Curious.

@mike-burns mike-burns closed this issue from a commit
@nzifnab nzifnab Rack gets a newline at the end of the content_type in certain instanc…
…es: 'image/jpeg\r' which causes S3 to be unable to accept the file. Fixes #817
eb92834
@mike-burns mike-burns closed this in eb92834
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.