net/http: the server doesn't handle whitespace before the first header field correctly #22464

crvv · 2017-10-27T07:44:57Z

What version of Go are you using (`go version`)?

go version devel +7a8e8b2f19 Fri Oct 27 05:47:09 2017 +0000 darwin/amd64
go version go1.9.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

darwin/amd64

What did you do?

Send a wrong HTTP request as
"GET / HTTP/1.1\r\n Host: host\r\n\r\n". A right request should be
"GET / HTTP/1.1\r\nHost: host\r\n\r\n"

The Host header should not be parsed according to RFC7230.
And the request may be rejected.

A recipient that receives whitespace between the
start-line and the first header field MUST either reject the message
as invalid or consume each whitespace-preceded line without further
processing of it (i.e., ignore the entire line, along with any
subsequent lines preceded by whitespace, until a properly formed
header field is received or the header section is terminated)

A runnable program is

package main

import (
	"io"
	"log"
	"net"
	"net/http"
	"os"
	"time"
)

func server() {
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
                log.Println(r.Host)
	})
	go func() {
		log.Fatal(http.ListenAndServe("localhost:8000", nil))
	}()
}

func request() {
	conn, err := net.Dial("tcp", "localhost:8000")
	if err != nil {
		panic(err)
	}
	_, err = conn.Write([]byte("GET / HTTP/1.1\r\n Host: host\r\n\r\n"))
	if err != nil {
		panic(err)
	}
	go func() {
		<-time.After(time.Millisecond * 100)
		conn.Close()
	}()
	io.Copy(os.Stdout, conn)
}
func main() {
	server()
	request()
}

What did you expect to see?

The request fails because of missing Host header or the whitespace .
The response from Nginx is

HTTP/1.1 400 Bad Request
Server: nginx/1.13.3
Date: Fri, 27 Oct 2017 07:29:15 GMT
Content-Type: text/html
Content-Length: 173
Connection: close

<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx/1.13.3</center>
</body>
</html>

What did you see instead?

A successful request.

2017/10/27 15:35:36 host
HTTP/1.1 200 OK
Date: Fri, 27 Oct 2017 07:35:36 GMT
Content-Length: 0
Content-Type: text/plain; charset=utf-8

ianlancetaylor · 2017-10-27T12:55:24Z

CC @tombergan

tombergan · 2017-10-27T16:31:51Z

Thanks for the report. I will have to do a bit more code archaeology to see if this was intentional. Per this comment, whitespace at the end of a key is intentionally stripped despite being forbidden by the RFC. It's unclear so far if whitespace at the beginning of the first key is intentionally ignored, or if that was an oversight.

bradfitz · 2017-11-02T04:17:11Z

The trailing whitespace was intentional, as seen from this test from 31e9429 from a security camera I used to have:

// Test that we read slightly-bogus MIME headers seen in the wild,                                                                                 
// with spaces before colons, and spaces in keys.                                                                                                  
func TestReadMIMEHeaderNonCompliant(t *testing.T) {
        // Invalid HTTP response header as sent by an Axis security                                                                                
        // camera: (this is handled by IE, Firefox, Chrome, curl, etc.)                                                                            
        r := reader("Foo: bar\r\n" +
                "Content-Language: en\r\n" +
                "SID : 0\r\n" +
                "Audio Mode : None\r\n" +
                "Privilege : 127\r\n\r\n")
        m, err := r.ReadMIMEHeader()
        want := MIMEHeader{
                "Foo":              {"bar"},
                "Content-Language": {"en"},
                "Sid":              {"0"},
                "Audio Mode":       {"None"},
                "Privilege":        {"127"},
        }
        if !reflect.DeepEqual(m, want) || err != nil {
                t.Fatalf("ReadMIMEHeader =\n%v, %v; want:\n%v", m, err, want)
        }
}

I know of no reason to accept leading whitespace, though, and I'm surprised we do.

gopherbot · 2017-11-02T06:51:34Z

Change https://golang.org/cl/75350 mentions this issue: net/textproto: retain leading whitespace in the first header key


        net/textproto: ignore initial lines with leading whitespaces in ReadM…

…IMEHeader A header line with leading whitespaces is not valid in HTTP as per RFC7230. This change ignores these invalid lines in ReadMIMEHeader. Updates #22464 Change-Id: Iff9f00380d28a9617a55ff7888a76fba82001402 Reviewed-on: https://go-review.googlesource.com/75350 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

tombergan · 2017-11-27T18:29:01Z

We fixed the parser to ignore headers like this:

GET / HTTP/1.1
  Host: foo.com
Accept-Encoding: gzip

However, headers like the following used to return an error, but don't anymore (note the lack of colon in the second line):

GET / HTTP/1.1
  Host foo.com
Accept-Encoding: gzip

tombergan · 2017-11-27T18:43:26Z

For Go 1.10, I will fix the above issue. For Go 1.11, we will reconsider if textproto.Reader needs multiple modes to more precisely handle the various RFCs it implements and to allow the caller to control whether we ignore invalid header lines or fail loudly.

bradfitz · 2017-11-27T18:54:23Z

I'd be in favor of reverting the net/textproto changes entirely for Go 1.10 and instead writing the few line header parser in net/http that net/http needs (also for Go 1.10).

All this effort working across two packages and differing needs isn't worth it.

tombergan · 2017-11-27T19:12:22Z

I'd be in favor of reverting the net/textproto changes entirely for Go 1.10 and instead writing the few line header parser in net/http that net/http needs (also for Go 1.10).

I agree that net/texproto is doing too much, but I don't think that's a complete solution. It solves the problem for net/http but no one else. net/textproto is still wrong: it should not accept headers where the first line begins with a space. This holds for both RFC 5322 (smtp) and RFC 7230 (http). And, there are people outside net/http using net/textproto to parse HTTP headers (I know of one fairly major use inside Google). Those parsers will be broken if we don't fix net/textproto.

gopherbot · 2017-11-27T19:41:41Z

Change https://golang.org/cl/80055 mentions this issue: net/textproto: reject all headers with a leading space


        net/textproto: reject all headers with a leading space

Previously, golang.org/cl/75350 updated ReadMIMEHeader to ignore the first header line when it begins with a leading space, as in the following example: GET / HTTP/1.1 Host: foo.com Accept-Encoding: gzip However, golang.org/cl/75350 changed ReadMIMEHeader's behavior for the following example: before the CL it returned an error, but after the CL it ignored the first line. GET / HTTP/1.1 Host foo.com Accept-Encoding: gzip This change updates ReadMIMEHeader to always fail when the first header line starts with a space. During the discussion for golang.org/cl/75350, we realized we had three competing needs: 1. HTTP clients should accept malformed response headers when possible (ignoring the malformed lines). 2. HTTP servers should reject all malformed request headers. 3. The net/textproto package is used by multiple protocols (most notably, HTTP and SMTP) which have slightly different parsing semantics. This complicates changes to net/textproto. We weren't sure how to best fix net/textproto without an API change, but it is too late for API changes in Go 1.10. We decided to ignore initial lines that begin with spaces, thinking that would have the least impact on existing users -- malformed headers would continue to parse, but the initial lines would be ignored. Instead, golang.org/cl/75350 actually changed ReadMIMEHeader to succeed in cases where it previously failed (as in the above example). Reconsidering the above two examples, there does not seem to be a good argument to silently ignore ` Host: foo.com` but fail on ` Host foo.com`. Hence, this change fails for *all* headers where the initial line begins with a space. Updates #22464 Change-Id: I68d3d190489c350b0bc1549735bf6593fe11a94c Reviewed-on: https://go-review.googlesource.com/80055 Run-TryBot: Tom Bergan <tombergan@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

bradfitz · 2017-11-27T20:36:09Z

@tombergan, what remains here?

tombergan · 2017-11-27T20:44:15Z

Nothing remains for 1.10. What remains is deciding whether or not to make a larger change for Go 1.11, which could include one or both of:

Copy ReadMIMEHeader into a private implementation of net/http and fine tune the behavior for HTTP clients and servers. Perhaps also do the same for net/mail and net/smtp.
Add a "mode" enum to textproto.Reader that would add loose/strict modes and specialize ReadMIMEHeader for each RFC. e.g., ModeSMTP, ModeLooseHTTP, ModeStrictHTTP.

bradfitz · 2017-11-27T20:49:28Z

SGTM. Thanks for the milestone update.

ianlancetaylor added the NeedsInvestigation label Oct 27, 2017

ianlancetaylor added this to the Go1.10 milestone Oct 27, 2017

bradfitz added the NeedsFix label Nov 2, 2017

gopherbot removed the NeedsInvestigation label Nov 2, 2017

bradfitz modified the milestones: Go1.10, Go1.11 Nov 10, 2017

tombergan self-assigned this Nov 27, 2017

tombergan modified the milestones: Go1.11, Go1.10 Nov 27, 2017

tombergan modified the milestones: Go1.10, Go1.11 Nov 27, 2017

tombergan removed their assignment Nov 27, 2017

bcmills mentioned this issue Apr 2, 2018

net/textproto: ReadLineBytes() returns strange []byte #24642

Closed

bradfitz modified the milestones: Go1.11, Go1.12 Jul 9, 2018

andybons removed this from the Go1.12 milestone Feb 12, 2019

andybons added this to the Go1.13 milestone Feb 12, 2019

andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019

rsc modified the milestones: Go1.14, Backlog Oct 9, 2019

golang/go

net/http: the server doesn't handle whitespace before the first header field correctly #22464

net/http: the server doesn't handle whitespace before the first header field correctly #22464

crvv commented Oct 27, 2017

This comment has been minimized.

ianlancetaylor commented Oct 27, 2017

This comment has been minimized.

tombergan commented Oct 27, 2017

This comment has been minimized.

bradfitz commented Nov 2, 2017

This comment has been minimized.

gopherbot commented Nov 2, 2017

This comment has been minimized.

tombergan commented Nov 27, 2017

This comment has been minimized.

tombergan commented Nov 27, 2017

This comment has been minimized.

bradfitz commented Nov 27, 2017

This comment has been minimized.

tombergan commented Nov 27, 2017

This comment has been minimized.

gopherbot commented Nov 27, 2017

This comment has been minimized.

bradfitz commented Nov 27, 2017

This comment has been minimized.

tombergan commented Nov 27, 2017

This comment has been minimized.

bradfitz commented Nov 27, 2017

golang/go

Join GitHub today

net/http: the server doesn't handle whitespace before the first header field correctly #22464

Comments

crvv commented Oct 27, 2017

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

This comment has been minimized.

ianlancetaylor commented Oct 27, 2017

This comment has been minimized.

tombergan commented Oct 27, 2017

This comment has been minimized.

bradfitz commented Nov 2, 2017

This comment has been minimized.

gopherbot commented Nov 2, 2017

This comment has been minimized.

tombergan commented Nov 27, 2017

This comment has been minimized.

tombergan commented Nov 27, 2017

This comment has been minimized.

bradfitz commented Nov 27, 2017

This comment has been minimized.

tombergan commented Nov 27, 2017

This comment has been minimized.

gopherbot commented Nov 27, 2017

This comment has been minimized.

bradfitz commented Nov 27, 2017

This comment has been minimized.

tombergan commented Nov 27, 2017

This comment has been minimized.

bradfitz commented Nov 27, 2017

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?