Host header should be capitalized #346

Closed
davidcornu opened this Issue Oct 19, 2012 · 12 comments

8 participants

@davidcornu

Ran into an issue today making requests against an external web service because request was sending a lowercase host header.

http://en.wikipedia.org/wiki/List_of_HTTP_header_fields
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.23

@davidcornu

Further research by @cdnbacon has shown that this is also the case for several other headers.

@cdnbacon

It seems those passed by callers are not case-transformed (which is good). But I'm curious why certain headers handled by request explicitly are lowercase. Any reason for doing this? Host and Content-Length are some of them.

request.get({
 url:'http://www.example.com', 
 body:'Test', 
 headers:{MiXed: true, UPPER: 'ValUE', lower: true}
});

Observed raw HTTP request

Note Host and Content-Length are lowercased.

GET http://www.example.com/ HTTP/1.1
MiXed: true
UPPER: ValUE
lower: true
host: www.example.com
content-length: 4
Connection: keep-alive

Test

Expected raw HTTP request

Note Host and Content-Length have spec-compliant casing.

GET http://www.example.com/ HTTP/1.1
MiXed: true
UPPER: ValUE
lower: true
Host: www.example.com
Content-Length: 4
Connection: keep-alive

Test
@cdnbacon

Problem seems to be with service, headers should be considered in a case-insensitive manner according to official spec.

http://www.ietf.org/rfc/rfc2616.txt

4.2 Message Headers

   HTTP header fields, which include general-header (section 4.5),
   request-header (section 5.3), response-header (section 6.2), and
   entity-header (section 7.1) fields, follow the same generic format as
   that given in Section 3.1 of RFC 822 [9]. Each header field consists
   of a name followed by a colon (":") and the field value. Field names
   are case-insensitive. 
@davidcornu davidcornu closed this Oct 24, 2012
@chuyeow

Even though the spec says headers are case-insensitive, there are certain servers that misbehave when they get a lowercase Host header. One example is Hacker News - try this:

curl -H 'host: news.ycombinator.com' -is http://news.ycombinator.com/

HTTP/1.0 301 Moved Permanently
Location: http://news.ycombinator.com/
Connection: close

If you use a capitalized Host header, it works fine.

curl -H 'Host: news.ycombinator.com' -is http://news.ycombinator.com/

I'd say it's worthwhile being "nice" and either:

  1. always send a capitalized Host header, or
  2. allow developers to set a capitalized Host header like so: request({ headers: { Host: 'www.example.com' } }) - right now this results in 2 Host header entries like so: headers: { host: 'www.example.com', Host: 'www.example.com' }
@zweifisch

Host as default +1

@mikeal
request member

the problem with changing it is that node programs are used to checking for lowercase headers.

a change landed in request that make the host header check caseless, so you can fix this against broken servers by setting "Host":"hostname" yourself in the headers object.

@vohof

@cdnbacon how did you dump the raw HTTP request?

@ajb

This is giving me errors, specifically with the 'Cookie' header.

@mikeal
request member

then the server is broken :)

@ajb

I agree -- unfortunately I don't control the servers we're scraping government RFP data from.

Would it make sense to enable capitalized headers with a simple setting?

@mikeal
request member

if you set the headers yourself we don't mess with the casing. there's actually a lot of code to make sure we check if headers exist in a caseless way and don't mess with them if you set them.

@dandv

@vohof: if you set NODE_DEBUG=request, the request module will dump to STDERR the headers and other debugging information.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment