Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: don't read Request headers after ServeHTTP starts #14940

Closed
bronze1man opened this issue Mar 24, 2016 · 8 comments

Comments

Projects
None yet
4 participants
@bronze1man
Copy link
Contributor

commented Mar 24, 2016

Please answer these questions before submitting your issue. Thanks!

  1. What version of Go are you using (go version)?
    go version go1.6 linux/amd64
  2. What operating system and processor architecture are you using (go env)?
    GOARCH="amd64"
    GOBIN=""
    GOEXE=""
    GOHOSTARCH="amd64"
    GOHOSTOS="linux"
    GOOS="linux"
    GOPATH=""
    GORACE=""
    GOROOT="/usr/local/go"
    GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
    GO15VENDOREXPERIMENT="1"
    CC="gcc"
    GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
    CXX="g++"
    CGO_ENABLED="1"

This issues is happend on a product api server with qps about 100/s.
It will crash the process.

fatal error: concurrent map read and map write

goroutine 226231 [running]:
runtime.throw(0x1444b00, 0x21)
    /usr/local/go/src/runtime/panic.go:530 +0x90 fp=0xc820fed438 sp=0xc820fed420
runtime.mapaccess1_faststr(0x12550e0, 0xc820858bd0, 0x12e2cd0, 0xa, 0x2daa800)
    /usr/local/go/src/runtime/hashmap_fast.go:202 +0x5b fp=0xc820fed498 sp=0xc820fed438
net/http.(*Request).wantsClose(0xc8207fc000, 0xc820859000)
    /usr/local/go/src/net/http/request.go:1071 +0x6d fp=0xc820fed510 sp=0xc820fed498
net/http.(*chunkWriter).writeHeader(0xc8200fe440, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:943 +0x7d1 fp=0xc820fed930 sp=0xc820fed510
net/http.(*chunkWriter).close(0xc8200fe440)
    /usr/local/go/src/net/http/server.go:286 +0x5d fp=0xc820fedba0 sp=0xc820fed930
net/http.(*response).finishRequest(0xc8200fe410)
    /usr/local/go/src/net/http/server.go:1252 +0x96 fp=0xc820fedbd0 sp=0xc820fedba0
net/http.(*conn).serve(0xc822e27680)
    /usr/local/go/src/net/http/server.go:1476 +0xf68 fp=0xc820fedf98 sp=0xc820fedbd0
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820fedfa0 sp=0xc820fedf98
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2137 +0x44e

...

It generated a 2MB panic log,but

  • i can not found hashmap_fast.go in the rest of the log.
  • All code run that is write by me in the panic log do not read/write http.Request.Header in the log.
  • It happens once per day.

I may do something bad. but how can i found the bad thing?
What is the possible bad thing ?

@randall77

This comment has been minimized.

Copy link
Contributor

commented Mar 24, 2016

Can you run your server with the race detector (-race) enabled? It may give us a better idea of where this race is coming from.

@bronze1man

This comment has been minimized.

Copy link
Contributor Author

commented Mar 24, 2016

Yes,I can.
It may take time to add -race into our DevOps system.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Mar 24, 2016

Is your http.Handler code modifying the (*http.Request).Header map?

@bradfitz bradfitz changed the title [net/http] fatal error: concurrent map read and map write crash in net/http package. net/http: concurrent map read and map write crash Mar 24, 2016

@bronze1man

This comment has been minimized.

Copy link
Contributor Author

commented Mar 24, 2016

Is ok to write (*http.Request).Header map in the http.handler callback and in the http.Handler thread?
I just found some handlers in reverse proxy part can write that header.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Mar 24, 2016

I suppose net/http shouldn't read from Request.Header after ServeHTTP has been started.

@bronze1man

This comment has been minimized.

Copy link
Contributor Author

commented Mar 24, 2016

Thanks for help.
I think I should apologize that I waste your time for understanding what happened.

I think I may found the reason of this crash.
I find some code in reverse proxy part that write (*http.Request).Header map in another thread which may did it after the http.handler callback return.

I will try to fix it and wait for crash for a week, and keep this issues open.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Mar 25, 2016

Not a waste of time. We could change net/http to look at what it needs to in the request headers before starting the user's ServeHTTP func and be more robust against this in general.

@bradfitz bradfitz self-assigned this Mar 25, 2016

@bradfitz bradfitz added this to the Go1.7 milestone Mar 25, 2016

@bradfitz bradfitz changed the title net/http: concurrent map read and map write crash net/http: don't read Request headers after ServeHTTP starts Mar 25, 2016

@gopherbot

This comment has been minimized.

Copy link

commented Apr 5, 2016

CL https://golang.org/cl/21530 mentions this issue.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.