POST with non-urlencoded data fails when request does not specify a content-type

[hbf]

I am doing a POST to a Sinatra application:

curl -v --data-binary @sample-request -H "Content-Type:" http://localhost:7500/rpc/v1/sample

The effect of the -H "Content-Type:" is that my request does not contain a Content-Type header (see output below). The data I post is a binary Protobuf message (and thus not of type application/x-www-form-urlencoded).

I get the following error:

$ curl -v --data-binary @sample-request -H "Content-Type:" http://localhost:7500/rpc/v1/sample
* About to connect() to localhost port 7500 (#0)
*   Trying ::1...
* Connection refused
*   Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 7500 (#0)
> POST /rpc/v1/sample HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5
> Host: localhost:7500
> Accept: */*
> Content-Length: 71
> 
* upload completely sent off: 71 out of 71 bytes
< HTTP/1.1 500 Internal Server Error 
< Content-Type: text/plain
< Content-Length: 3164
< Server: WEBrick/1.3.1 (Ruby/1.9.3/2013-02-22)
< Date: Fri, 25 Oct 2013 04:06:05 GMT
< Connection: Keep-Alive
< 
ArgumentError: invalid byte sequence in UTF-8
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/utils.rb:104:in `normalize_params'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/utils.rb:96:in `block in parse_nested_query'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/utils.rb:93:in `each'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/utils.rb:93:in `parse_nested_query'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/request.rb:373:in `parse_query'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/request.rb:211:in `POST'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/request.rb:225:in `params'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:877:in `call!'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:870:in `call'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/xss_header.rb:18:in `call'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/path_traversal.rb:16:in `call'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/json_csrf.rb:18:in `call'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/base.rb:49:in `call'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/base.rb:49:in `call'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/frame_options.rb:31:in `call'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/nulllogger.rb:9:in `call'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/head.rb:11:in `call'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sinatra-1.4.3/lib/sinatra/showexceptions.rb:21:in `call'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:175:in `call'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1949:in `call'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1449:in `block in call'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1726:in `synchronize'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1449:in `call'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/handler/webrick.rb:60:in `service'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/1.9.1/webrick/httpserver.rb:138:in `service'
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/1.9.1/webrick/httpserver.rb:94:in `run'
* Connection #0 to host localhost left intact
    /usr/local/Cellar/ruby193/1.9.3-p392/lib/ruby/1.9.1/webrick/server.rb:191:in `block in start_thread'* Closing connection #0

It seems that Sinatra is trying to urldecode the request body. According to the HTTP Spec, a server should assume application/octet-stream if no content-type is given.

The request works fine when I add to curl the option -H "Content-Type: application/octet-stream".

[kgrz]

Related: #514

[lukateras]

Investigating.

[lukateras]

Sorry for taking such a long time for this.

This is the very reason: https://github.com/rack/rack/blob/master/lib/rack/request.rb#L33
Rack::Request#content_type returns nil if you have Content-Type: header, and it auto-chooses UTF-8 encoding for your binary sent data which gives you the error.

To work properly, #content_type method should be rewritten like this:

def content_type
  content_type = @env['CONTENT_TYPE']
  content_type.nil? || content_type.empty? ? 'application/octet-stream' : content_type
end

I suppose I should send a PR to rack/rack. Comments are much appreciated, since I’m not sure it is a bug.

[rkh]

Closing this here, as it's a Rack bug.

[rtyler]

@hbf I also hit this bug and participated on the pull request for Rack. I don't think it's realistic to get this changed in Rack so long as they're not breaking major version compatibility.

The workaround that I've validated works, and I would suggest is to use the following middleware in your Protobuf HTTP servers:

class DefaultContentTypeMiddleware
  def initialize(app)
    @app = app
  end

  def call(env)
    # Default to the 'application/octet-stream' content type if one is not set.
    if env['CONTENT_TYPE'].nil? || env['CONTENT_TYPE'].empty?
      env['CONTENT_TYPE'] = 'application/octet-stream'
    end
    @app.call(env)
  end
end

You could even go so far as to default to application/x-protobuf but I'll leave that decision up to you :)