Skip to content
This repository

fix invalid characters in multipart uploads #515

Merged
merged 1 commit into from 12 months ago

6 participants

Michael Raidel Konstantin Shabanov Konstantin Haase Jeremy Evans Santiago Pastorino Oscar Del Ben
Michael Raidel

Fixes invalid characters in multipart uploads by removing all characters which are not valid. Otherwise uploads with invalid characters raise an exception (ArgumentError: invalid byte sequence in UTF-8)

Michael Raidel

travis build fails because of missing bacon gem for rubinius so it has nothing to do with the pull request

lib/rack/multipart/parser.rb
... ...
@@ -137,6 +137,9 @@ def get_filename(head)
137 137
         if filename && filename.scan(/%.?.?/).all? { |s| s =~ /%[0-9a-fA-F]{2}/ }
138 138
           filename = Utils.unescape(filename)
139 139
         end
  140
+        if filename && String.instance_methods.include?(:valid_encoding?)
4

You can use respond_to, but can you comment on what happens on versions of Ruby prior to 1.9?

Michael Raidel
mraidel added a note February 12, 2013

on ruby 1.8.7 this does nothing which is fine because the problem only exists since multibyte support in ruby 1.9

Konstantin Haase Collaborator
rkh added a note February 12, 2013

You can use respond_to

or method_defined?

Michael Raidel
mraidel added a note February 12, 2013

yes, method_defined? sounds good. respond_to? would work here too but it doesn't feel right to me because we would test on another instance of string, not the one we are using the method on (as we are using chars which returns new instances of String). Have pushed a fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Konstantin Shabanov

What is the problem with merging this PR?

Konstantin Haase
Collaborator
rkh commented April 03, 2013

No one reviewed it yet.

Jeremy Evans

I had the same problem on one of my sites. I tested this commit and it fixes the error. However, I think the implementation is suboptimal from a performance standpoint. It should only do the work of filtering the string if it is not already in a valid encoding, since checking a string for a valid encoding is very fast, rebuilding is slow, and the vast majority of filenames are already in a valid encoding. To the new if statement, you should append && !filename.valid_encoding?. This should be be about 2.5x faster for a single byte filename, 7x faster for 10 byte filename, 35x faster for a 100 byte filename, and close to 300x faster for a 1KB filename.

With that change I think this should go in.

Santiago Pastorino
Collaborator

:+1: to what @jeremyevans said. So @mraidel can you do that modification and squash your commits?

Santiago Pastorino
Collaborator

And also your test https://travis-ci.org/rack/rack/jobs/4745776 fail in Ruby 2.0

Santiago Pastorino
Collaborator

@mraidel You will need to properly encode the String in the test

Michael Raidel

@jeremyevans @spastorino I have included both recommended changes and squashed the commits!

Santiago Pastorino
Collaborator

@raggi @rkh :+1: good to merge

Konstantin Haase rkh commented on the diff April 20, 2013
lib/rack/multipart/parser.rb
@@ -137,6 +137,9 @@ def get_filename(head)
137 137
         if filename && filename.scan(/%.?.?/).all? { |s| s =~ /%[0-9a-fA-F]{2}/ }
138 138
           filename = Utils.unescape(filename)
139 139
         end
  140
+        if filename && String.method_defined?(:valid_encoding?) && !filename.valid_encoding?
  141
+          filename = filename.chars.select { |char| char.valid_encoding? }.join
  142
+        end
1
Konstantin Haase Collaborator
rkh added a note April 20, 2013

On 2.1, we could use String#scrub, btw.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Konstantin Haase rkh merged commit a811f12 into from April 20, 2013
Konstantin Haase rkh closed this April 20, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 1 unique commit by 1 author.

Apr 20, 2013
Michael Raidel fix invalid characters in multipart uploads b61d33c
This page is out of date. Refresh to see the latest.
3  lib/rack/multipart/parser.rb
@@ -137,6 +137,9 @@ def get_filename(head)
137 137
         if filename && filename.scan(/%.?.?/).all? { |s| s =~ /%[0-9a-fA-F]{2}/ }
138 138
           filename = Utils.unescape(filename)
139 139
         end
  140
+        if filename && String.method_defined?(:valid_encoding?) && !filename.valid_encoding?
  141
+          filename = filename.chars.select { |char| char.valid_encoding? }.join
  142
+        end
140 143
         if filename && filename !~ /\\[^\\"]/
141 144
           filename = filename.gsub(/\\(.)/, '\1')
142 145
         end
6  test/multipart/invalid_character
... ...
@@ -0,0 +1,6 @@
  1
+--AaB03x
  2
+Content-Disposition: form-data; name="files"; filename="invalidÃ.txt"
  3
+Content-Type: text/plain
  4
+
  5
+contents
  6
+--AaB03x--
14  test/spec_multipart.rb
@@ -166,6 +166,20 @@ def rd.length
166 166
     params["files"][:tempfile].read.should.equal "contents"
167 167
   end
168 168
 
  169
+  should "parse multipart upload with filename with invalid characters" do
  170
+    env = Rack::MockRequest.env_for("/", multipart_fixture(:invalid_character))
  171
+    params = Rack::Multipart.parse_multipart(env)
  172
+    params["files"][:type].should.equal "text/plain"
  173
+    params["files"][:filename].should.match(/invalid/)
  174
+    head = "Content-Disposition: form-data; " +
  175
+      "name=\"files\"; filename=\"invalid\xC3.txt\"\r\n" +
  176
+      "Content-Type: text/plain\r\n"
  177
+    head = head.force_encoding("ASCII-8BIT") if head.respond_to?(:force_encoding)
  178
+    params["files"][:head].should.equal head
  179
+    params["files"][:name].should.equal "files"
  180
+    params["files"][:tempfile].read.should.equal "contents"
  181
+  end
  182
+
169 183
   should "not include file params if no file was selected" do
170 184
     env = Rack::MockRequest.env_for("/", multipart_fixture(:none))
171 185
     params = Rack::Multipart.parse_multipart(env)
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.