net/textproto: allows multi-line header field names

[katiehockman]

Reported by @empijei

Go supports multiline headers with the obsolete leading whitespace syntax as described in RPC 7230 3.2.3 and 3.2.4. This is acceptable for header field values as long as servers "replace each received obs-fold with one or more SP octets prior to interpreting the field value or forwarding the message downstream"

However, this behavior also occurs with header field names, which isn't specified in any RFC and is likely an over-extension of HTTP parsing rules. In other words, the following is allowed, and resolves to Gopher-New- Line:

GET /path HTTP/1.1
Host: golang.org
Gopher-New-
 Line: This is a header on multiple lines
Content-Type: application/goroutines

Servers should reject requests with a field-name containing whitespace.

/cc @bradfitz @FiloSottile

[bradfitz]

Does net/textproto do this intentionally for some reason? Do other protocol implementations using net/textproto require this? net/mail? net/smtp?

[bradfitz]

Looking at older RFCs (e.g. https://www.ietf.org/rfc/rfc2822.txt section 2.2) I'm not seeing any support for folding mid field name.

So this can probably be fixed in net/textproto without anything in net/http.

[bradfitz]

Yeah, even the classic RFC 822 makes it clear we're just doing it wrong:

https://www.ietf.org/rfc/rfc822.txt

     3.1.1.  LONG HEADER FIELDS

        Each header field can be viewed as a single, logical  line  of
        ASCII  characters,  comprising  a field-name and a field-body.
        For convenience, the field-body  portion  of  this  conceptual
        entity  can be split into a multiple-line representation; this
        is called "folding"

[bradfitz]

https://play.golang.org/p/dSuwBrE7pWF

[katiehockman]

So here's the dilemma:

There are two places we could fail headers of this type: in readContinuedLineSlice() or ReadMIMEHeader().

Initially, I thought it would be safe to change readContinuedLineSlice() to fail if the first line doesn't contain a colon, but this function is exported through ReadContinuedLineBytes() and we have no way of guaranteeing that people are only using this function to read header lines. If we start throwing errors if the first line doesn't contain a colon then we may break users.

So that leaves ReadMIMEHeader(). Unfortunately, ReadMIMEHeader() doesn't know the difference between a space from the normalization caused by readContinuedLineSlice() and a space that was in the request originally. We could just fail every field name with a space, but this function currently accepts field names with spaces (even if those are invalid server-side).

I think it's possible to do this check in ReadMIMEHeader() with some additional fields returned by readContinuedLineSlice() but it's adding non-insignificant complexity for something that is already rejected server-side, and likely isn't hurting anyone.

Note that Chrome and curl both accept responses with newlines and/or spaces in the header field names.

At this point I think we should just leave this alone. @bradfitz thoughts?

[bradfitz]

I'd add an optional param to:

func (r *Reader) readContinuedLineSlice() ([]byte, error) {

like:

validateFirstLine func([]byte) error

And then if it's nil, act like today.

But if it's non-nil, then call that validation func with the first line.

Then make ReadMIMEHeader pass a validation func that makes sure there's a colon on the first line or whatever.

And make the existing ReadContinuedLineBytes pass nil with no extra validation, like today.

So the end result is that only ReadMIMEHeader gets stricter. (and thus net/http, net/smtp, net/mail, etc).

[katiehockman]

Sure I can look into that. @FiloSottile and I were brainstorming a few ways of doing this last week.

My concern was the additional complexity to these functions given that this issue doesn't seem likely to be a notable security risk. Sounds like the fix is important enough to warrant the changes though.

[bradfitz]

Even if net/http rejects it, there are other uses of net/textproto.

[gopherbot]

Change https://golang.org/cl/199838 mentions this issue: textproto: do not allow multi-line header field names