This repository has been archived by the owner. It is now read-only.

HTTP parser fails if server returns no headers #1711

Closed
davglass opened this Issue Sep 15, 2011 · 11 comments

Comments

Projects
None yet
9 participants
@davglass

I was doing some low-level scraping on a few of my devices (Cable Modem, X10 system, ADT alarm system) and noticed that all of these embedded devices return no headers in their http responses. It returns a status code, but no headers. With no headers being sent, the Node HTTP parser fails (silently in 0.4.11 and with a parse error in 0.5.x)

Here is a repro case:
https://gist.github.com/1217143

I couldn't find an entry in the spec that "required" headers to be sent from the server, but I feel that the parser shouldn't choke if no headers are returned, since curl/wget/links all work as expected.

@ry

This comment has been minimized.

Show comment
Hide comment
@ry

ry Sep 15, 2011

Responding from a server with

HTTP/1.1 200 OK\r\n\r\n<html><head><title></title></head><body></body></html>

Is invalid HTTP.

When the parser reaches the end of the CRLFCRLF it has two ways of interpreting this message: first that it has no content and the next character begins the next message or that the user agent should consider the rest of the connection to be the body terminated by EOF.

HTTP/1.1 servers default to Connection: keep-alive and therefore the user agent considers this to be a zero length message to be soon followed by another HTTP response.

The relevant branch in http-parser is https://github.com/ry/http-parser/blob/c0ecab0516147401b5fd02a2272ebfb5dce8deb4/http_parser.c#L1564-1571 and here https://gist.github.com/8d854170ac16c39fdc74 is a test I was using to determine this.

@mnot am I correct or should this be interpreted as a zero-length message?

ry commented Sep 15, 2011

Responding from a server with

HTTP/1.1 200 OK\r\n\r\n<html><head><title></title></head><body></body></html>

Is invalid HTTP.

When the parser reaches the end of the CRLFCRLF it has two ways of interpreting this message: first that it has no content and the next character begins the next message or that the user agent should consider the rest of the connection to be the body terminated by EOF.

HTTP/1.1 servers default to Connection: keep-alive and therefore the user agent considers this to be a zero length message to be soon followed by another HTTP response.

The relevant branch in http-parser is https://github.com/ry/http-parser/blob/c0ecab0516147401b5fd02a2272ebfb5dce8deb4/http_parser.c#L1564-1571 and here https://gist.github.com/8d854170ac16c39fdc74 is a test I was using to determine this.

@mnot am I correct or should this be interpreted as a zero-length message?

@davglass

This comment has been minimized.

Show comment
Hide comment
@davglass

davglass Sep 15, 2011

@ry I Thought that too, but reading this:

http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4

"Both types of message consist of a start-line, zero or more header fields (also known as "headers"), an empty line (i.e., a line with nothing preceding the CRLF) indicating the end of the header fields, and possibly a message-body"

That made me think that headers are optional since it's worded as "zero or more".

@ry I Thought that too, but reading this:

http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4

"Both types of message consist of a start-line, zero or more header fields (also known as "headers"), an empty line (i.e., a line with nothing preceding the CRLF) indicating the end of the header fields, and possibly a message-body"

That made me think that headers are optional since it's worded as "zero or more".

@ry

This comment has been minimized.

Show comment
Hide comment
@ry

ry Sep 15, 2011

@davglass headers are indeed optional but you will need either a Transfer-Encoding or a Content-Length or a Connection: close in order to have a response with a body using HTTP/1.1. The response

HTTP/1.1 200 OK\r\n\r\n

is valid

ry commented Sep 15, 2011

@davglass headers are indeed optional but you will need either a Transfer-Encoding or a Content-Length or a Connection: close in order to have a response with a body using HTTP/1.1. The response

HTTP/1.1 200 OK\r\n\r\n

is valid

@mnot

This comment has been minimized.

Show comment
Hide comment
@mnot

mnot Sep 16, 2011

See:
http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-16#section-3.3

The relevant rule is the catch-all;

  1. Otherwise, this is a response message without a declared message-
    body length, so the message-body length is determined by the
    number of octets received prior to the server closing the
    connection.

I.e., it's all body.

mnot commented Sep 16, 2011

See:
http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-16#section-3.3

The relevant rule is the catch-all;

  1. Otherwise, this is a response message without a declared message-
    body length, so the message-body length is determined by the
    number of octets received prior to the server closing the
    connection.

I.e., it's all body.

@c4milo

This comment has been minimized.

Show comment
Hide comment
@c4milo

c4milo Sep 16, 2011

See also: http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4 point 5.
That's what curl actually does: https://github.com/bagder/curl/blob/master/lib/http.c#L2831

camilo@imacsita:~> curl -vvvv http://localhost:8500

  • About to connect() to localhost port 8500 (#0)
  • Trying 127.0.0.1... connected
  • Connected to localhost (127.0.0.1) port 8500 (#0)

    GET / HTTP/1.1
    User-Agent: curl/7.21.6 (x86_64-apple-darwin10.7.0) libcurl/7.21.6 OpenSSL/1.0.0d zlib/1.2.5 libidn/1.22
    Host: localhost:8500
    Accept: /

    < HTTP/1.1 200 OK
  • no chunk, no close, no size. Assume close to signal end
    <
  • Closing connection #0 <title></title>camilo@imacsita:~>

c4milo commented Sep 16, 2011

See also: http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4 point 5.
That's what curl actually does: https://github.com/bagder/curl/blob/master/lib/http.c#L2831

camilo@imacsita:~> curl -vvvv http://localhost:8500

  • About to connect() to localhost port 8500 (#0)
  • Trying 127.0.0.1... connected
  • Connected to localhost (127.0.0.1) port 8500 (#0)

    GET / HTTP/1.1
    User-Agent: curl/7.21.6 (x86_64-apple-darwin10.7.0) libcurl/7.21.6 OpenSSL/1.0.0d zlib/1.2.5 libidn/1.22
    Host: localhost:8500
    Accept: /

    < HTTP/1.1 200 OK
  • no chunk, no close, no size. Assume close to signal end
    <
  • Closing connection #0 <title></title>camilo@imacsita:~>
@pstadler

This comment has been minimized.

Show comment
Hide comment
@pstadler

pstadler Oct 3, 2011

I've got the same problem. Receiving HTTP/0.9 200 OK without headers. curl output:

curl -vvvv http://domain.tld:5222
* About to connect() to domain.tld port 5222 (#0)
*   Trying x.x.x.x... connected
* Connected to domain.tld (x.x.x.x) port 5222 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: domain.tld:5222
> Accept: */*
> 
* Connection #0 to host domain.tld left intact
* Closing connection #0
<?xml version='1.0'?><stream:stream xmlns='jabber:client' xmlns:stream='http://etherx.jabber.org/streams' id='0123456789' from='domain.tld' version='1.0'><stream:error><xml-not-well-formed xmlns='urn:ietf:params:xml:ns:xmpp-streams'/></stream:error></stream:stream>

Node.js:

Error: Parse Error
    at Socket.ondata (http.js:1231:22)
    at Socket._onReadable (net.js:677:27)
    at IOWatcher.onReadable [as callback] (net.js:177:10)

pstadler commented Oct 3, 2011

I've got the same problem. Receiving HTTP/0.9 200 OK without headers. curl output:

curl -vvvv http://domain.tld:5222
* About to connect() to domain.tld port 5222 (#0)
*   Trying x.x.x.x... connected
* Connected to domain.tld (x.x.x.x) port 5222 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: domain.tld:5222
> Accept: */*
> 
* Connection #0 to host domain.tld left intact
* Closing connection #0
<?xml version='1.0'?><stream:stream xmlns='jabber:client' xmlns:stream='http://etherx.jabber.org/streams' id='0123456789' from='domain.tld' version='1.0'><stream:error><xml-not-well-formed xmlns='urn:ietf:params:xml:ns:xmpp-streams'/></stream:error></stream:stream>

Node.js:

Error: Parse Error
    at Socket.ondata (http.js:1231:22)
    at Socket._onReadable (net.js:677:27)
    at IOWatcher.onReadable [as callback] (net.js:177:10)
@dazagrohovaz

This comment has been minimized.

Show comment
Hide comment
@dazagrohovaz

dazagrohovaz Nov 17, 2011

hi everybody,
response "HTTP/1.0 200 Connection established\r\n\r\n" could not be parsed either...
other issue like this:
joyent#1956

regards

hi everybody,
response "HTTP/1.0 200 Connection established\r\n\r\n" could not be parsed either...
other issue like this:
joyent#1956

regards

@c4milo

This comment has been minimized.

Show comment
Hide comment
@c4milo

c4milo Nov 17, 2011

is this still happening with nodejs 0.6.1, I haven't try it out.

c4milo commented Nov 17, 2011

is this still happening with nodejs 0.6.1, I haven't try it out.

@bnoordhuis

This comment has been minimized.

Show comment
Hide comment
@bnoordhuis

bnoordhuis Nov 17, 2011

Member

Confirmed. A HTTP/1.0 response body without headers works, a HTTP/0.9 response does not.

Member

bnoordhuis commented Nov 17, 2011

Confirmed. A HTTP/1.0 response body without headers works, a HTTP/0.9 response does not.

bnoordhuis added a commit that referenced this issue Nov 17, 2011

test: add 'response body with no headers' http test
HTTP/0.9 - fails with a parse error
HTTP/1.0 - works
HTTP/1.1 - fails with an empty response body

See #1711.
@bnoordhuis

This comment has been minimized.

Show comment
Hide comment
@bnoordhuis

bnoordhuis Nov 17, 2011

Member

(Failing) test added in 4f38c5e.

Member

bnoordhuis commented Nov 17, 2011

(Failing) test added in 4f38c5e.

ry added a commit to nodejs/http-parser that referenced this issue Nov 22, 2011

felixge added a commit to felixge/node that referenced this issue Nov 22, 2011

test: add 'response body with no headers' http test
HTTP/0.9 - fails with a parse error
HTTP/1.0 - works
HTTP/1.1 - fails with an empty response body

See #1711.

@felixge felixge closed this in 6e1e9e2 Nov 22, 2011

alexkwolfe added a commit to alexkwolfe/node that referenced this issue Dec 23, 2011

test: add 'response body with no headers' http test
HTTP/0.9 - fails with a parse error
HTTP/1.0 - works
HTTP/1.1 - fails with an empty response body

See #1711.

alexkwolfe added a commit to alexkwolfe/node that referenced this issue Dec 23, 2011

Get test-http-response-no-headers.js to pass
Main fix was in 3abebf which added HTTP/0.9 support to http parser.

Changed test because HTTP 1.1 mandates keep-alive when no headers are
given.

Fixes #1711

@vokal-isaac vokal-isaac referenced this issue in vokal/VOKMockUrlProtocol Oct 27, 2014

Merged

Update README with response file format examples #6

@shauket

This comment has been minimized.

Show comment
Hide comment
@shauket

shauket Nov 9, 2015

kindly guide me how can i parse header response using c++

shauket commented Nov 9, 2015

kindly guide me how can i parse header response using c++

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.