Bad request when you don't put a slash at the end of the URL

[Pigueiras]

Hello,

the library fails when you don't specify a path. For example this ruby code works:

require 'ftw'
agent = FTW::Agent.new
request = agent.post('http://127.0.0.1/')
puts agent.execute(request)

Output:

16:39  $ ruby test-ftw.rb 
HTTP/1.1 404 Not Found
date: Thu, 13 Nov 2014 15:40:03 GMT
server: Apache/2.4.10 (Fedora)
content-length: 198
keep-alive: timeout=5, max=100
connection: Keep-Alive
content-type: text/html; charset=iso-8859-1

Without the / at the end of the URL:

require 'ftw'
agent = FTW::Agent.new
request = agent.post('http://127.0.0.1')
puts agent.execute(request)

Output:

16:41  $ ruby test-ftw.rb 
/.../.gem/ruby/gems/ftw-0.0.41/lib/ftw/protocol.rb:34:in `<<': Could not parse data entirely (HTTP::Parser::Error)
    from /..../.gem/ruby/gems/ftw-0.0.41/lib/ftw/protocol.rb:34:in `read_http_message'
    from /..../.gem/ruby/gems/ftw-0.0.41/lib/ftw/request.rb:93:in `execute'
    from /..../.gem/ruby/gems/ftw-0.0.41/lib/ftw/agent.rb:289:in `execute'
    from test-ftw.rb:4:in `<main>'

I am trying with a new httpd installation, in case you want to reproduce the same error. What I get in the access logs is the following.

XXXXXX - - [13/Nov/2014:15:48:01 +0100] "POST  HTTP/1.1" 400 317 "-" "-"
XXXXXX - - [13/Nov/2014:15:51:41 +0100] "POST / HTTP/1.1" 200 86 "-" "logstash"

This was resulting in this error in logstash, when I was trying to use a Flume source to get messages.

Unhandled exception {:request=><FTW::Request(@4246) @protocol="http" @body="{\"message\":\"...",\"@version\":\"1\",\"@timestamp\":\"2014-11-12T14:15:04.867Z\",\"host\":\"....ch\",\"path\":..."}" @method="POST" @headers=FTW::HTTP::Headers <{"host"=>"....", "connection"=>"keep-alive", "content-type"=>"json", "content-length"=>271}> @version=1.1 @request_uri="" @logger=#<Cabin::Channel:0x60938447 @subscriber_lock=#<Mutex:0x74d182d9>, @data={}, @metrics=#<Cabin::Metrics:0x66e5b079 @channel=#<Cabin::Channel:0x60938447 ...>, @metrics={}, @metrics_lock=#<Mutex:0x71a50cb5>>, @subscribers={}, @level=:info> @port=10002 >, :response=>nil, :exception=>#<HTTP::Parser::Error: invalid method>, :stacktrace=>["org/ruby_http_parser/RubyHttpParser.java:341:in `<<'", "/root/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/protocol.rb:34:in `read_http_message'", "/root/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/request.rb:93:in `execute'", "/root/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:325:in `execute'", "/root/logstash-1.4.2/lib/logstash/outputs/http.rb:126:in `receive'", "/root/logstash-1.4.2/lib/logstash/outputs/base.rb:86:in `handle'", "(eval):26:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "/root/logstash-1.4.2/lib/logstash/pipeline.rb:266:in `output'", "/root/logstash-1.4.2/lib/logstash/pipeline.rb:225:in `outputworker'", "/root/logstash-1.4.2/lib/logstash/pipeline.rb:152:in `start_outputs'"], :level=>:warn}

[jordansissel]

What's the expected behavior? Should we raise a specific exception due to invalid URL?

[Pigueiras]

Something like this would probably solve the issue.

diff --git a/lib/ftw/request.rb b/lib/ftw/request.rb
index be75334..14323a5 100644
--- a/lib/ftw/request.rb
+++ b/lib/ftw/request.rb
@@ -111,7 +111,11 @@ class FTW::Request
     end
     # uri.password
     # uri.user
-    @request_uri = uri.path
+    if !uri.path.empty?
+       @request_uri = uri.path
+    else
+       @request_uri = "/"
+    end
     # Include the query string, too.
     @request_uri += "?#{uri.query}" if !uri.query.nil?

Or if you consider the URL as an invalid one, throwing an exception with a proper message when the uri.path is empty would also work.

The thing is that if I do curl without the / at the end, it works. That's why I wasn't expecting a fail because of this and the error that I was obtaining from the library wasn't an obvious one at all.

[magnusbaeck]

What's the expected behavior? Should we raise a specific exception due to invalid URL?

Path-less URLs aren't invalid (see §3 of RFC 3986) so this library should happily accept them and refrain from making malformed HTTP requests like GET HTTP/1.1.

There's at least one issue reported in https://github.com/logstash-plugins/logstash-input-websocket that stems from this bug.