Issues with Spoof File Checks #1429

Closed
ericchernuka opened this Issue Jan 31, 2014 · 59 comments

Comments

Projects
None yet

Hi,
I just upgraded to Paperclip 4.0 and now I'm getting an error about spoofed_media_type. I found the helper for:

do_not_validate_attachment_file_type :push_certificate

But I still receive error the error message. Does Paperclip::MediaTypeSpoofDetector.using(adapter, value.original_filename).spoofed? not take into account the do_not_validate?

Member

jyurek commented Feb 1, 2014

No, the spoof detector is not affected by that helper, only the enforced check for content type/filename validations.

What is the filename and type of file that you're uploading? What is the output of file --mime-type for this file?

The filename is "ios_push_notification_certificate.pem" and the type that set on the form uploader is "application/x-x509-ca-cert". When I run file --mime-type it returns ios_push_notification_certificate.pem: text/plain

Member

jyurek commented Feb 1, 2014

Well, that's annoying. The file command completely fails us here. I think letting do_not_validate_attachment_file_type bypass this might be the thing to do. I'll let you know when I add that in. Sorry about the inconvenience.

No problem. I thought I could get around it by registering a new mime-type for the lookup but wasn't able to have that work either.

Thanks for the quickly reply and looking into this. Thought I was going nuts.

schristm commented Feb 2, 2014

I've got the same problem here. I'm wondering if there's a workaround for the time being until there's an update. For now, I'll just roll back to the previous version. Overall though, this looks like a nice release. Looking forward to testing it out more. Thanks for producing this gem.

glebtv commented Feb 7, 2014

This also messes with files assigned with uri-open for me (same error)

glebtv commented Feb 7, 2014

Workaround:

require 'paperclip/media_type_spoof_detector'
module Paperclip
  class MediaTypeSpoofDetector
    def spoofed?
      false
    end
  end
end

@glebtv is right.
I just encountered the same issue ( Paperclip 4.1.0 ). ( used the exact same workaround )
Passing a URI as paperclip image creates a temp file with a filename e.g. 'open-uri20140210-81840-d9erdb'. Note that it doesn't have an extenstion - guess paperclip doesn't like that anymore.

Member

jyurek commented Feb 10, 2014

Nope, it doesn't. But that sounds like something we can fix. I assume the URL will have an extension, so we can tack one on to the Tempfile name.

I have come across the same problem attempting to upload a word .doc file which is resulting in the spoofed error message as the file utility returns null for the mime-type. This is on an Ubuntu 12.04.4 LTS (GNU/Linux 3.8.0-29-generic x86_64) server with version is 5.09 of the file utility and its magic.mgc file from 2011.

These are the relevant rails application log entries:

I, [2014-02-11T13:54:01.212664 #2215]  INFO -- :   Parameters: {"utf8"=>"✓", "authenticity_token"=>"*****", "attachment"=>{"attachable_id"=>"133", "attachable_type"=>"Member", "name"=>"doc", "description"=>"", "attachment"=>#<ActionDispatch::Http::UploadedFile:0x007fb27d422840 @tempfile=#<Tempfile:/tmp/RackMultipart20140211-2215-12kq7vg>, @original_filename="CRM Reports.doc", @content_type="application/msword", @headers="Content-Disposition: form-data; name=\"attachment[attachment]\"; filename=\"CRM Reports.doc\"\r\nContent-Type: application/msword\r\n">}, "commit"=>"Create Attachment"}
I, [2014-02-11T13:54:01.251359 #2215]  INFO -- : Command :: file -b --mime-type '/tmp/965e167d5e49be81f12458cd73908c8720140211-2215-qg25m9'
I, [2014-02-11T13:54:01.261300 #2215]  INFO -- : [paperclip] Content Type Spoof: Filename CRM_Reports.doc (["application/msword", "application/word", "application/x-msword", "application/x-word", "text/plain"]), content type discovered from file command: . See documentation to allow this combination.

For reference, I can upload the same file in development on my mac running mavericks with version is 5.04 of the file utility.

I also have issue with LinkedIn avatars (probably other networks have the same)
i try to set user avatar from LinkedIn auth data.
for example link to my avatar, there is no correct file name, so console output is

Command :: file -b --mime-type '/var/folders/36/x84hhxs12p976ft_2zh9d7fh0000gn/T/5bee0e2ae651d1f03d8279bf1325360920140213-78899-5f5vji'
[paperclip] Content Type Spoof: Filename 0_R84qR_HfdTSmRmE3RkdIRhVfIFU7UDE3JG2wRhdKqT7xkWgTBQH4v8gYLoRK4owDV_VoqbUB66UF ([]), content type discovered from file command: image/jpeg.

and model is not saved

The issue also affects Facebook avatars captured from OAuth authentication. The image comes back as part of the JSON response as an URL of the form http://graph.facebook.com/10000000/picture?type=large (note no extension), which then fails validation.

Contributor

djcp commented Feb 14, 2014

@chrismhilton - can you send me an example of a file that gives a null mime-type? I cannot replicate the condition where file would return nothing. I'm on debian testing, BTW. djcp at thoughtbot dot com

Contributor

teeparham commented Feb 18, 2014

Here's an attachment that was previously saved from gravatar. It originally had no extension. When I reprocess it, it fails spoof detection:

(note: the trailing dot is part of the url)
https://dcluasumhcbm6.cloudfront.net/images/6635/original139a7916496355bb264ef5c0a15944db.

[paperclip] copying images/6635/original359f43a6098ab57d0627650dc0265bf9. to local file /var/folders/gg/76wnz1s57gl_whn0xqb4hz5m0000gn/T/596a5011c51bd5107971bf1b98dad9eb20140218-4537-bxfcel
[AWS S3 200 0.675437 0 retries] get_object(:bucket_name=>"xxx",:key=>"images/6635/original359f43a6098ab57d0627650dc0265bf9.")  
[paperclip] saving images/6635/medium359f43a6098ab57d0627650dc0265bf9.
[AWS S3 200 1.063706 0 retries] put_object(:acl=>:public_read,:bucket_name=>"xxx",:cache_control=>"public,max-age=31536000",:content_length=>14778,:content_type=>"image/jpeg",:data=>Paperclip::FileAdapter: 7eafe68436d86220a2f116db76bcb31e20140218-4537-17g095b20140218-4537-4dm8jo,:expires=>"Wed, 18 Feb 2015 16:54:32 GMT",:key=>"images/6635/medium359f43a6098ab57d0627650dc0265bf9.")  
Command :: file -b --mime-type '/var/folders/gg/76wnz1s57gl_whn0xqb4hz5m0000gn/T/596a5011c51bd5107971bf1b98dad9eb20140218-4537-bxfcel'
[paperclip] Content Type Spoof: Filename open-uri20121207-25901-1brx8vl ([]), content type discovered from file command: image/jpeg. See documentation to allow this combination.

agius commented Feb 18, 2014

Also hits when there's a difference between production / dev dbs... I have an image attachment for companies which uses different s3 buckets for production vs dev. When I copy the production db to dev, paperclip tries to download the image from the production s3 bucket and doesn't have access. So it fails with the same error message. Frustrating.

An additional note: on CentOS 5.10 (at least) it seems the file command doesn't have a --mime-type parameter, just --mime, which causes spoof detection to fail.
A workaround is:

module Paperclip
  class MediaTypeSpoofDetector
    def type_from_file_command
      begin
        Paperclip.run("file", "-b --mime :file", :file => @file.path)
      rescue Cocaine::CommandLineError
        ""
      end
    end
  end
end
Contributor

will-r commented Feb 20, 2014

I can confirm (after @chrismhilton) that file version 5.09 will not give a mime type for word files. I too am using Ubuntu 12.04 so it could be local to that distribution, but for now I have disabled the checks.

It is also a nuisance that the anti-spoof validation runs on every save, as it means an unchanged file has to be pulled down from S3. Could it be applied only when the file has changed? Happy to submit a pr.

Member

jyurek commented Feb 21, 2014

There is a change on master (8b5289b) that only checks for spoofing when the file changes. That should go out relatively soon. @browntiger Could you open another ticket about that, so we can keep separate causes for the same problem contained? Thanks.

@jyurek, Thanks, I opened ticket #1451 for the problem.

guyisra commented Feb 24, 2014

I confirm spoofed problems with facebook image URI

Where's the documentation to re-enable setting and changing attachments to an object?

I'm having issues as well with the latest version of Paperclip as far as normal pngs being seen as invalid png files on Windows.

However, it seems more controlled and has to do more of not being able to remove the restriction of assigning an attachment to an object via the command line (for testing purposes, as well as for mocking,stubbing, and extending factories).

image

im getting an error: "Validation failed: Datafile has an extension that does not match its contents" when saving an xml file through a rake task fetching from FTP server. the same file can be attached to the model through active admin. running file sc.xml --mime-type gives application/xml. Is this the same issue? im using ruby 2.0.0 and rails 4.0.2 .

@glebtv 's workaround got me rolling for now. (thanks!) and probably confirms its the same issue i guess.

ays0110 commented Mar 12, 2014

Sorry for the noobish question, but where do I put this workaround? In my model that has the paperclip attachment? As a helper somewhere?

@ays0110 i put it at the top of the rake file that was having problems. that might not be best practice, but there it is. worked for me. might help you. i'd try it a few places and see if it works. then think about where is best for it. for me its a bit hacky, so as near to the need as possible. hth

@asecondwill /config/initializers/paperclip.rb is the natural place for it.

@rossshannon ah ok, yeah that makes sense. thank you.

The content not match.. :(
" Avatar has an extension that does not match its contents"
What's up?

Contributor

djcp commented Mar 28, 2014

@lozandier - Your problem is probably that you don't have the file command available on $PATH, or it doesn't work properly. This should work OK on a POSIX-compatible OS like linux or OSX. Please open up a separate ticket if you can't get this to work after installing file and confirming it works and is correctly on your path.

Contributor

djcp commented Mar 28, 2014

Closing this ticket, it's veered from its original subject and we have fixes (via overrides) in place.

@djcp djcp closed this Mar 28, 2014

I'm a total noob - what exactly is the fix for this? Overwriting the spoofed method from @glebtv ? Or something else? Is there a new update to the gem I should install? @djcp

Contributor

richpeck commented Apr 11, 2014

Getting this error too

kenn commented Apr 11, 2014

FWIW, if you're getting photos from an external URL (e.g. Facebook API), and doing this:

picture.data = open(url)

Change it to:

picture.data = url

and it fixed the problem for me. Just pass a URL-like string, instead of a tempfile object.

If you use the patch above to force spoofed? to false, you might be missing an extension on the file, like picture. instead of picture.jpg - it occurred to me and I didn't like it.

@kenn Your method doesn't work when getting user avatar from Facebook OAuth authentication. I got redirection forbidden error.

kenn commented Apr 14, 2014

@linhmtran168 That's exactly what I'm doing, using koala gem. Works for me.

> api = Koala::Facebook::API.new(access_token)
> url = api.get_object('10153283921325045')['images'][0]['source']
=> "https://scontent-b.xx.fbcdn.net/hphotos-prn1/t1.0-9/1238245_10153283921325045_976150929_n.jpg"
> picture.data = url
Command :: file -b --mime-type ...
...
> picture.save
=> true

@linhmtran168 If you use last paperclip's master, this should be fixed. I think this recent commit fixed it: 263a498

Salute!

thanks @deivid-rodriguez for pointing that out!

@kunalksm No worries! :)

I still have this 'Content Type Spoof' error with the lastest paperclip patch!

@kenn's tip above works for me!

I get it working now. I did exactly the opposite of @kenn's tip. It's like this:

user.avatar = open(avatar_url)

@dkonayuki that worked for me as well for omniauth with paperclip 4.1.1

url = auth.info.image
user.avatar = open(url)

note: adding :image_size => 'SIZE' as an option for omniauth breaks both methods :/

Contributor

justinko commented Jul 22, 2014

@chrismhilton I'm on ubuntu 12.04 LTS as well. You need to set the path to the file binary in Cocaine. I added this to config/initializers/cocaine.rb: Cocaine::CommandLine.path = '/usr/bin'

Contributor

justinko commented Jul 23, 2014

@chrismhilton actually, you should use Paperclip.options[:command_path] = '/usr/bin'

Hi
I am getting a similar error
"Attachment translation missing:
en.activerecord.errors.models.submission_detail.attributes.attachment.spoofed_media_type"

When I upload files (word, excel) with Japanese content in it

I tried adding 'do_not_validate_attachment_file_type' but no use.

I do not want to override the paperclip module

Please help me to fix.

If you are using ruby on windows (like me, sadly) then note that there is no "file" command on windows for paperclip to check spoofing. As such, you always get a message like this in your rails server:

[paperclip] Content Type Spoof: Filename upload_test.csv (["text/csv", "text/comma-separated-values"]), content type discovered from file command: . See documentation to allow this combination.

Thankfully, you can add the file command using the correct Gnuwin32 open source package. Make sure you set your environment variables "path" to include the installed bin directory and close all windows terminals to pick up the change (this includes any IDE's, such as RubyMine) then restart your rails server.

mehulkar commented Sep 5, 2014

Getting the same issue with .plist files. file --mime-type returns text/plain and it fails spoof detection

Is there a workaround to allow you to use something like:

url = auth.info.image
user.avatar = open(url)

... and specify image_size?

@ozvillafan I recommend using Ubuntu or another well-received version of
Linux, unless you cannot reliably backup or recover your particular Windows
Parition with a Windows X disc (or if you want to boot by default to
Windows using Window's bootloader). Ubuntu auto-partitioning command often
messes up and it's usually a better idea to do it yourself for the /,
swapfile, and the optional /home partition

On Tue, Sep 2, 2014 at 3:49 AM, ozvillafan notifications@github.com wrote:

If you are using ruby on windows (like me, sadly) then note that there is
no "file" command on windows for paperclip to check spoofing. As such, you
always get a message like this in your rails server:

[paperclip] Content Type Spoof: Filename upload_test.csv (["text/csv",
"text/comma-separated-values"]), content type discovered from file command:
. See documentation to allow this combination.

Thankfully, you can add the file command using the correct Gnuwin32 open
source package. Make sure you set your environment variables "path" to
include the installed bin directory and close all windows terminals to pick
up the change (this includes any IDE's, such as RubyMine) then restart your
rails server.


Reply to this email directly or view it on GitHub
#1429 (comment)
.

Kevin Lozandier
lozandier@gmail.com lozandier@gmail.com

@mehulkar were you able to pass the spoof detection? I'm getting errors when uploading images via api as base64 string and then decoding and saving to paperclip. File is image/jpg and detection is application/octet-stream

@scanales we basically did a mass override and turned off the feature using @glebtv's workaround since we can trust our users:

require 'paperclip/media_type_spoof_detector'
module Paperclip
  class MediaTypeSpoofDetector
    def spoofed?
      false
    end
  end
end

Same error, using latest paperclip gem relese, on Ubuntu 14.04. Confirming that workaround from @glebtv does work.

@mehulkar I've tried that but it's not working for me, I'm on CentOS. I get the following error:

Command :: file -b --mime '/tmp/53d9d5a46c76ace548ff62a0385485a420141016-13087-gmpqhm.jpeg'
Command :: identify -format '%wx%h,%[exif:orientation]' '/tmp/53d9d5a46c76ace548ff62a0385485a420141016-13087-gmpqhm.jpeg[0]' 2>/dev/null
[paperclip] An error was received while processing: #<Paperclip::Errors::NotIdentifiedByImageMagickError: Paperclip::Errors::NotIdentifiedByImageMagickError>

This is the code I added to the initializer:

require 'paperclip/media_type_spoof_detector'         

Paperclip::MediaTypeSpoofDetector.class_eval do
    def spoofed?
      false
    end
    def type_from_file_command      
        begin       
          Paperclip.run("file", "-b --mime :file", :file => @file.path)
        rescue Cocaine::CommandLineError
          ""
        end       
    end      
end

Marthyn commented Feb 9, 2015

require 'paperclip/media_type_spoof_detector'

module Paperclip
  class MediaTypeSpoofDetector
    def type_from_file_command
      begin
        Paperclip.run("file", "-b --mime :file", :file => @file.path)
      rescue

      end
    end
  end
end

seems to work when file is uploaded on windows

mjodko commented Mar 1, 2015

As an improvement to @glebtv's workaround you might want to try this:

# config/initializers/paperclip.rb
require 'paperclip/media_type_spoof_detector'

module Paperclip
  class MediaTypeSpoofDetector
    old_spoofed = instance_method(:spoofed?)

    define_method(:spoofed?) do
      if supplied_file_content_types.count > 0
        old_spoofed.bind(self).()
      else
        false
      end
    end
  end
end

@glebtv @mjodko thank you for this. Giggling that this is in production on our servers right now.

zdavis added a commit to ManifoldScholar/manifold that referenced this issue Feb 28, 2016

[B] Address Paperclip spoofing false positives
On the staging host, Paperclip repeatedly reports non-spoofed
attachments as spoofed, which breaks ingestion. We're quick-fixing it
with a monkey patch for now, and in doing so will incur some technical
debt. We'll circle back later and attempt to come up with a more robust
fix, perhaps by avioding paperclip alltogether.

See thoughtbot/paperclip#1429 for more on
this issue.

Hopefully this replaces the monkey patching: #2378

No need for monkey patch.

Paperclip has content-type checking and media-type checking. Both seem to do the same thing using file -b --mime.

Media-type checking uses the global config of content_type_mappings. However, sometimes you want per-model exceptions. Paperclip provides the option for it but not well documented. I got it from the source here:
https://github.com/thoughtbot/paperclip/blob/v4.3/lib/paperclip/has_attached_file.rb#L83

You can do the following to control media-type and content-type checking:

class PaperclipModel < ActiveRecord::Base

   has_attached_file :attachment, { validate_media_type: false }

  validates_attachment :attachment, {
     # tweak as desired
     content_type: { content_type: ["text/csv", "text/plain", Paperclip::ContentTypeDetector::SENSIBLE_DEFAULT] }
  }
end

tomfast commented Apr 3, 2017

Using the latest version of Paperclip (5.1.0) resolved the issue for me. I'm suspecting

  • Add default content_type_detector to UploadedFileAdapter (#2270)

did the trick for me.

Source: https://github.com/thoughtbot/paperclip/blob/master/NEWS for 5.1.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment