Processing of malformed HTTP header names

[nnposter]

Local function parse_header() in library http.lua performs processing of HTTP response headers. It currently returns with an error (and no other results) if a header name is malformed, resulting in a completely failed response. This can be caused by an invalid character in the name or just superfluous whitespace between the name and the colon. This rigidity is causing problems with testing not-so-compliant targets.

A real-world example on which NSE scripts fail:

HTTP/1.1 200 OK
Server: Netgear
Content-Type: text/html
Pragma: no-cache
Last Modified: Fri, 16 July 2001 01:01:01 GMT
Connection: close

In contrast, browsers and other tools tend to handle this situation more gracefully:

• Firefox ignores the particular header, continuing with the next header.
• Internet Explorer tries to salvage the name if possible. As an example, it will treat line aaa bbb: ccc as header aaa bbb, despite the illegal space in the middle.
• Wireshark will treat the remainder of the response as the body.

I would like to propose a change in parse_header() to mimic the Firefox behavior. Namely, a header with a malformed name will be skipped, i.e. it will not show up in the header table, but its original textual representation is still preserved in rawheader.

A corresponding patch is pretty straightforward. For the sake of clarity the patch presented here ignores whitespace changes (option -b). A real patch would include proper indentation.

--- ../../svn/nmap/nselib/http.lua  2016-09-27 12:26:44.928032942 -0600
+++ nselib/http.lua 2016-10-27 07:22:15.613209959 -0600
@@ -661,10 +661,15 @@
   while pos <= #header do
     -- Get the field name.
     e, name = get_token(header, pos)
-    if not name or e > #header or string.sub(header, e, e) ~= ":" then
-      return nil, string.format("Can't get header field name at %q", string.sub(header, pos, pos + 30))
+    -- Do not bail out if the header is malformed. Consume the header line
+    -- anyway, getting to the next header, but do not create a new entry in
+    -- the "header" table.
+    if e then
+      if header:sub(e, e) ~= ":" then
+        name = nil
     end
     pos = e + 1
+    end

     -- Skip initial space.
     pos = skip_lws(header, pos)
@@ -680,6 +685,7 @@
       pos = skip_lws(header, pos)
     end

+    if name then
     -- Set it in our table.
     name = string.lower(name)
     local value = table.concat(words, " ")
@@ -699,6 +705,7 @@
         -- Ignore any cookie parsing error
       end
     end
+    end

     -- Next field, or end of string. (If not it's an error.)
     s, e = string.find(header, "^\r?\n", pos)

Please provide feedback on whether this change would cause any issues.