problem with async uploading #177

Closed
macbury opened this Issue May 3, 2012 · 8 comments

Comments

Projects
None yet
4 participants

macbury commented May 3, 2012

When i run this code:
require "rubygems"
require "bundler/setup"

Bundler.require :default

class UploadForm < Goliath::API
def response(env)
[200, { "Content-Type" => "text/html" }, '

'+''+''+'']
end
end

class AsyncUpload < Goliath::API
use Goliath::Rack::Params # parse & merge query and body parameters
use Goliath::Rack::DefaultMimeType # cleanup accepted media types
use Goliath::Rack::Render, 'json' # auto-negotiate response format

def on_headers(env, headers)
env.logger.info 'received headers: ' + headers.inspect
env['async-headers'] = headers
end

def on_body(env, data)
env.logger.info 'received data: ' + data
(env['async-body'] ||= '') << data
end

def on_close(env)
env.logger.info 'closing connection'
end

def response(env)
[200, {}, {body: env['async-body'], head: env['async-headers']}]
end
end

class Router < Goliath::API
get '/', UploadForm
post '/upload', AsyncUpload
end

runner = Goliath::Runner.new(ARGV, nil)
runner.api = Router.new
runner.app = Goliath::Rack::Builder.build(Router, runner.api)
runner.run

i got this error:

root@h162:/home/projects/twist# ruby app.rb --stdout -p 5005
[10012:INFO] 2012-05-03 04:27:29 :: Starting server on 0.0.0.0:5005 in development mode. Watch out for stones.
[10012:INFO] 2012-05-03 04:27:30 :: received headers: {"Host"=>"macbury.pl:5005", "Connection"=>"keep-alive", "Content-Length"=>"1351", "Cache-Control"=>"max-age=0", "Origin"=>"http://macbury.pl:5005", "User-Agent"=>"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.19 (KHTML, like Gecko) Ubuntu/12.04 Chromium/18.0.1025.168 Chrome/18.0.1025.168 Safari/535.19", "Content-Type"=>"multipart/form-data; boundary=----WebKitFormBoundaryPEtlZT0SINkzfztC", "Accept"=>"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Referer"=>"http://macbury.pl:5005/", "Accept-Encoding"=>"gzip,deflate,sdch", "Accept-Language"=>"pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4", "Accept-Charset"=>"ISO-8859-2,utf-8;q=0.7,*;q=0.3", "Cookie"=>"__cfduid=d7f0d2d827f9507bde6e1fed77ce16f691332514155; connect.sid=XogBudUiwOKnBTAROLcya5bZ.CYAuS5BqLyz6nPFh6ouuMucvBOMaVT%2By1fOB9hXlHVE"}
[10012:INFO] 2012-05-03 04:27:30 :: received data: ------WebKitFormBoundaryPEtlZT0SINkzfztC
Content-Disposition: form-data; name="file-data"; filename="SDK Readme.txt"
Content-Type: text/plain

Welcome to the Android SDK!

The Android SDK archive initially contains only the basic SDK tools. It does
not contain an Android platform or any third-party libraries. In fact, it
doesn't even have all the tools you need to develop an application.

In order to start developing applications, you must install the Platform-tools
and at least one version of the Android platform, using the SDK Manager.

Platform-tools contains build tools that are periodically updated to support new
features in the Android platform (which is why they are separate from basic
SDK tools), including adb, dexdump, and others.

To install Platform-tools, Android platforms and other add-ons, you must
have an Internet connection, so if you plan to use the SDK while
offline, please make sure to download the necessary components while online.

To start the SDK Manager, please execute the program "android".

From the command-line you can also directly trigger an update by
executing:
  tools/android update sdk --no-ui

Tip: use --help to see the various command-line options.


For more information, please consult the Android web site at
  http://developer.android.com/sdk/


------WebKitFormBoundaryPEtlZT0SINkzfztC--

[10012:ERROR] 2012-05-03 04:27:30 :: bad content body
/usr/local/rvm/gems/ruby-1.9.2-head@twist/gems/rack-1.4.1/lib/rack/multipart/parser.rb:73:in `block in fast_forward_to_first_boundary'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/gems/rack-1.4.1/lib/rack/multipart/parser.rb:70:in `loop'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/gems/rack-1.4.1/lib/rack/multipart/parser.rb:70:in `fast_forward_to_first_boundary'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/gems/rack-1.4.1/lib/rack/multipart/parser.rb:15:in `parse'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/gems/rack-1.4.1/lib/rack/multipart.rb:25:in `parse_multipart'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/bundler/gems/goliath-a9fbe078a81d/lib/goliath/rack/params.rb:23:in `retrieve_params'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/bundler/gems/goliath-a9fbe078a81d/lib/goliath/rack/builder.rb:43:in `block (3 levels) in build'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/gems/http_router-0.9.7/lib/http_router.rb:165:in `call'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/gems/http_router-0.9.7/lib/http_router.rb:165:in `process_destination_path'
(eval):16:in `block (2 levels) in inject_root_methods'
(eval):6:in `catch'
(eval):6:in `block in inject_root_methods'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/gems/http_router-0.9.7/lib/http_router/node/root.rb:68:in `[]'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/gems/http_router-0.9.7/lib/http_router.rb:118:in `block in call'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/gems/http_router-0.9.7/lib/http_router.rb:118:in `catch'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/gems/http_router-0.9.7/lib/http_router.rb:118:in `call'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/bundler/gems/goliath-a9fbe078a81d/lib/goliath/rack/async_middleware.rb:73:in `call'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/gems/rack-1.4.1/lib/rack/content_length.rb:14:in `call'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/gems/async-rack-0.5.1/lib/async_rack/async_callback.rb:114:in `call'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/gems/async-rack-0.5.1/lib/async_rack/async_callback.rb:91:in `block in new'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/gems/rack-1.4.1/lib/rack/reloader.rb:44:in `call'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/gems/rack-1.4.1/lib/rack/reloader.rb:44:in `call'
/usr/local/rvm/gems/ruby-1.9.2-head@twist/bundler/gems/goliath-a9fbe078a81d/lib/goliath/request.rb:163:in `block in process'
[10012:INFO] 2012-05-03 04:27:30 :: Status: 500, Content-Length: 16, Response Time: 32.20ms
[10012:INFO] 2012-05-03 04:27:30 :: closing connection
[10012:ERROR] 2012-05-03 04:27:30 :: You need to implement response
[10012:INFO] 2012-05-03 04:27:30 :: Status: 500, Content-Length: 35, Response Time: 52.73ms

igrigorik closed this May 3, 2012

I'm having the same problem. I can see this is closed but without comments. Any advice?

Owner

igrigorik commented Sep 25, 2012

If you have an isolated test case which reproduces the problem.. that will help! :)

ok, steps to reproduce

  1. checkout master
  2. run ruby async_upload.rb -sv from examples folder
  3. upload a file with curl like this curl -F file=@list.txt localhost:9000

My findings so far, it seems to be related to Goliath::Rack::Params.

EDIT: For clarity my error is the following:


------------------------------79e32baaf9f2--

[34204:ERROR] 2012-09-25 23:19:10 :: bad content body
[34204:ERROR] 2012-09-25 23:19:10 :: /Users/fregini/.rvm/gems/ruby-1.9.3-p194/gems/rack-1.4.1/lib/rack/multipart/parser.rb:73:in `block in fast_forward_to_first_boundary'
/Users/fregini/.rvm/gems/ruby-1.9.3-p194/gems/rack-1.4.1/lib/rack/multipart/parser.rb:70:in `loop'
/Users/fregini/.rvm/gems/ruby-1.9.3-p194/gems/rack-1.4.1/lib/rack/multipart/parser.rb:70:in `fast_forward_to_first_boundary'
/Users/fregini/.rvm/gems/ruby-1.9.3-p194/gems/rack-1.4.1/lib/rack/multipart/parser.rb:15:in `parse'
/Users/fregini/.rvm/gems/ruby-1.9.3-p194/gems/rack-1.4.1/lib/rack/multipart.rb:25:in `parse_multipart'
/Users/fregini/Work/goliath/lib/goliath/rack/params.rb:23:in `retrieve_params'
/Users/fregini/Work/goliath/lib/goliath/rack/params.rb:60:in `block in call'
/Users/fregini/Work/goliath/lib/goliath/rack/validator.rb:40:in `safely'
/Users/fregini/Work/goliath/lib/goliath/rack/params.rb:59:in `call'
/Users/fregini/Work/goliath/lib/goliath/rack/async_middleware.rb:73:in `call'
/Users/fregini/.rvm/gems/ruby-1.9.3-p194/gems/rack-1.4.1/lib/rack/content_length.rb:14:in `call'
/Users/fregini/.rvm/gems/ruby-1.9.3-p194/gems/async-rack-0.5.1/lib/async_rack/async_callback.rb:114:in `call'
/Users/fregini/.rvm/gems/ruby-1.9.3-p194/gems/async-rack-0.5.1/lib/async_rack/async_callback.rb:91:in `block in new'
/Users/fregini/.rvm/gems/ruby-1.9.3-p194/gems/rack-1.4.1/lib/rack/reloader.rb:44:in `call'
/Users/fregini/.rvm/gems/ruby-1.9.3-p194/gems/rack-1.4.1/lib/rack/reloader.rb:44:in `call'
/Users/fregini/Work/goliath/lib/goliath/request.rb:163:in `block in process'
[34204:INFO] 2012-09-25 23:19:10 :: Status: 500, Content-Length: 28, Response Time: 2025.69ms
[34204:INFO] 2012-09-25 23:19:10 :: closing connection

Forcing to populate rack.input in on_body seems to avoid the exception, but I'm not sure it's the right way to go.

  def on_body(env, data)
    env['rack.input'] << data 
  end
Owner

igrigorik commented Sep 27, 2012

Ah, hmm.. Yeah, that would do it. That's an annoying gotcha. Params middleware is not smart enough to handle the multi-part upload. Your solution works.. except for cases where the upload may be massive, and you definitely don't want to be buffering the data twice. In fact, stashing the entire response inside of any variable (like the demo script shows), is probably an anti-pattern.

Anyway I think there's a problem somewhere. The example is broken and we shuold fix it somehow, it may be misleading for newcomers.
Trying to upload a file with curl was the first thought I had, which sends multipart. What are the other ways one would use to upload a file?

Was just bit by this problem.

Seeing as the typical use case for streaming uploads is very large files and file uploads typically happen through multipart requests it would make sense to update the example.

Owner

igrigorik commented Oct 25, 2012

Well, the fix is simple: remove Rack::Params from the middleware list.

@igrigorik igrigorik added a commit that referenced this issue Oct 25, 2012

@igrigorik igrigorik remove params middleware to fix multipart upload
See discussion in #177.
d10e135
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment