message is empty when encoding = chuncked

[sander-su]

I have some cases where messages I send to the plugin are outputted empty.
The difference I found with the messages that have data is that they are not send in chunks.

Issue seems to be puma related: puma/puma#620

I might have a workaround to force my java app to http 1 and disable chunking but would be nice to have this fixed without a workaround

[dsolsona]

I can reproduce this with logstash 1.4.4 and 1.5.3

/opt/logstash/bin/logstash -e "input { http { codec => json {} } } output { stdout { codec => rubydebug} }"

If I send

curl -H "content-type: application/json" -XPUT 'http://127.0.0.1:8080/' -d '{
    "user" : "me",
    "message" : "trying out the http input"
}'

I get

{
          "user" => "me",
       "message" => "trying out the http input",
      "@version" => "1",
    "@timestamp" => "2015-07-22T08:19:13.826Z",
       "headers" => {
           "content_type" => "application/json",
         "request_method" => "PUT",
           "request_path" => "/",
            "request_uri" => "/",
           "http_version" => "HTTP/1.1",
        "http_user_agent" => "curl/7.26.0",
              "http_host" => "127.0.0.1:8080",
            "http_accept" => "*/*",
         "content_length" => "66"
    }
}

But If instead I send

curl -H "Transfer-Encoding: chunked" -H "content-type: application/json" -XPUT 'http://127.0.0.1:8080/' -d '{
    "user" : "me",
    "message" : "trying out the http input"
}'

what I get is an empty message and the error HTTP parse error, malformed request

2015-07-22 08:20:02 +0000: HTTP parse error, malformed request (): #<Puma::HttpParserError: Invalid HTTP format, parsing fails.>
2015-07-22 08:20:02 +0000: ENV: {"rack.version"=>[1, 3], "rack.errors"=>#<IO:fd 2>, "rack.multithread"=>true, "rack.multiprocess"=>false, "rack.run_once"=>false, "SCRIPT_NAME"=>"", "CONTENT_TYPE"=>"text/plain", "QUERY_STRING"=>"", "SERVER_PROTOCOL"=>"HTTP/1.1", "SERVER_SOFTWARE"=>"2.11.3", "GATEWAY_INTERFACE"=>"CGI/1.2"}
---
{
       "message" => "",
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
    "@timestamp" => "2015-07-22T08:20:02.348Z",
       "headers" => {
                  "content_type" => "application/json",
                "request_method" => "PUT",
                  "request_path" => "/",
                   "request_uri" => "/",
                  "http_version" => "HTTP/1.1",
               "http_user_agent" => "curl/7.26.0",
                     "http_host" => "127.0.0.1:8080",
                   "http_accept" => "*/*",
        "http_transfer_encoding" => "chunked"
    }
}

[jsvd]

This is still an issue in puma 2.15.3:

Here's a test that confirms the problem:

  context "with chunked headers" do
    subject { LogStash::Inputs::Http.new("port" => port) }
    it "should process the request normally" do
      subject.register
      Thread.new { subject.run(queue) }
      agent.post!("http://localhost:#{port}/meh.json",
                  :headers => {
                     "content-type" => "text/plain",
                     "Transfer-Encoding" => "chunked" },
                  :body => { "message_body" => "Hello" }.to_json)
      event = queue.pop
      expect(event["message_body"]).to eq("Hello")
    end
  end

Failures:

  1) LogStash::Inputs::Http with chunked headers should process the request normally
     Failure/Error: expect(event["message_body"]).to eq("Hello")

       expected: "Hello"
            got: nil

       (compared using ==)

[jsvd]

this is no longer an issue since 3.1.0 has replaced puma with netty