net/http: don't read Request headers after ServeHTTP starts

[bronze1man]

Please answer these questions before submitting your issue. Thanks!

1. What version of Go are you using (go version)?
   go version go1.6 linux/amd64
2. What operating system and processor architecture are you using (go env)?
   GOARCH="amd64"
   GOBIN=""
   GOEXE=""
   GOHOSTARCH="amd64"
   GOHOSTOS="linux"
   GOOS="linux"
   GOPATH=""
   GORACE=""
   GOROOT="/usr/local/go"
   GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
   GO15VENDOREXPERIMENT="1"
   CC="gcc"
   GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
   CXX="g++"
   CGO_ENABLED="1"

This issues is happend on a product api server with qps about 100/s.
It will crash the process.

fatal error: concurrent map read and map write

goroutine 226231 [running]:
runtime.throw(0x1444b00, 0x21)
    /usr/local/go/src/runtime/panic.go:530 +0x90 fp=0xc820fed438 sp=0xc820fed420
runtime.mapaccess1_faststr(0x12550e0, 0xc820858bd0, 0x12e2cd0, 0xa, 0x2daa800)
    /usr/local/go/src/runtime/hashmap_fast.go:202 +0x5b fp=0xc820fed498 sp=0xc820fed438
net/http.(*Request).wantsClose(0xc8207fc000, 0xc820859000)
    /usr/local/go/src/net/http/request.go:1071 +0x6d fp=0xc820fed510 sp=0xc820fed498
net/http.(*chunkWriter).writeHeader(0xc8200fe440, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:943 +0x7d1 fp=0xc820fed930 sp=0xc820fed510
net/http.(*chunkWriter).close(0xc8200fe440)
    /usr/local/go/src/net/http/server.go:286 +0x5d fp=0xc820fedba0 sp=0xc820fed930
net/http.(*response).finishRequest(0xc8200fe410)
    /usr/local/go/src/net/http/server.go:1252 +0x96 fp=0xc820fedbd0 sp=0xc820fedba0
net/http.(*conn).serve(0xc822e27680)
    /usr/local/go/src/net/http/server.go:1476 +0xf68 fp=0xc820fedf98 sp=0xc820fedbd0
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820fedfa0 sp=0xc820fedf98
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2137 +0x44e

...

It generated a 2MB panic log,but

• i can not found hashmap_fast.go in the rest of the log.
• All code run that is write by me in the panic log do not read/write http.Request.Header in the log.
• It happens once per day.

I may do something bad. but how can i found the bad thing?
What is the possible bad thing ?

[randall77]

Can you run your server with the race detector (-race) enabled? It may give us a better idea of where this race is coming from.

[bronze1man]

Yes,I can.
It may take time to add -race into our DevOps system.

[bradfitz]

Is your http.Handler code modifying the (*http.Request).Header map?

[bronze1man]

Is ok to write (*http.Request).Header map in the http.handler callback and in the http.Handler thread?
I just found some handlers in reverse proxy part can write that header.

[bradfitz]

I suppose net/http shouldn't read from Request.Header after ServeHTTP has been started.

[bronze1man]

Thanks for help.
I think I should apologize that I waste your time for understanding what happened.

I think I may found the reason of this crash.
I find some code in reverse proxy part that write (*http.Request).Header map in another thread which may did it after the http.handler callback return.

I will try to fix it and wait for crash for a week, and keep this issues open.

[bradfitz]

Not a waste of time. We could change net/http to look at what it needs to in the request headers before starting the user's ServeHTTP func and be more robust against this in general.

[gopherbot]

CL https://golang.org/cl/21530 mentions this issue.