Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.
Sign upnet/http: concurrent map read and map write #27811
Closed
Labels
Comments
This comment has been minimized.
This comment has been minimized.
|
Can you please provide a complete runnable code sample that demonstrates this fault, or causes the race detector to complain.
… On 22 Sep 2018, at 16:34, rchunping ***@***.***> wrote:
Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
go version go1.10.3 freebsd/amd64
Does this issue reproduce with the latest release?
What operating system and processor architecture are you using (go env)?
FreeBSD rchunping.localhost 11.0-RELEASE-p2 FreeBSD 11.0-RELEASE-p2 #0: Mon Oct 24 06:55:27 UTC 2016 ***@***.***:/usr/obj/usr/src/sys/GENERIC amd64
What did you do?
a http client application, download photo and upload photo to another server
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.
What did you expect to see?
no error
What did you see instead?
goroutine 801303 [running]:
runtime.throw(0x99a1dc, 0x21)
/usr/local/go/src/runtime/panic.go:616 +0x81 fp=0xc4200654a8 sp=0xc420065488 pc=0x42ba11
runtime.mapaccess2_fast64(0x8ec7a0, 0xc42026b6e0, 0xc420713d40, 0xc4201dadf0, 0x1)
/usr/local/go/src/runtime/hashmap_fast.go:141 +0x1aa fp=0xc4200654d0 sp=0xc4200654a8 pc=0x40b93a
net/http.(*connLRU).remove(...)
/usr/local/go/src/net/http/transport.go:2314
net/http.(*Transport).getIdleConn(0xc4df60, 0x0, 0x99c98d, 0x5, 0xc42054e180, 0x12, 0xc420713d40, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:807 +0x215 fp=0xc4200655f8 sp=0xc4200654d0 pc=0x6960c5
net/http.(*Transport).getConn(0xc4df60, 0xc420c11ad0, 0x0, 0x99c98d, 0x5, 0xc42054e180, 0x12, 0x0, 0x0, 0x433eec)
/usr/local/go/src/net/http/transport.go:921 +0x11d fp=0xc420065978 sp=0xc4200655f8 pc=0x696dcd
net/http.(*Transport).RoundTrip(0xc4df60, 0xc420312200, 0xc4df60, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:409 +0x632 fp=0xc420065bc0 sp=0xc420065978 pc=0x693a82
net/http.send(0xc420312200, 0x9ed180, 0xc4df60, 0x0, 0x0, 0x0, 0xc42000e560, 0xc42049b4a0, 0xc420065d58, 0x1)
/usr/local/go/src/net/http/client.go:252 +0x185 fp=0xc420065ce8 sp=0xc420065bc0 pc=0x643785
net/http.(*Client).send(0xc42026b9e0, 0xc420312200, 0x0, 0x0, 0x0, 0xc42000e560, 0x0, 0x1, 0x0)
/usr/local/go/src/net/http/client.go:176 +0xfa fp=0xc420065d68 sp=0xc420065ce8 pc=0x64345a
net/http.(*Client).Do(0xc42026b9e0, 0xc420312200, 0xc42057a000, 0x9ed1c0, 0xc420780340)
/usr/local/go/src/net/http/client.go:615 +0x28d fp=0xc420065ef0 sp=0xc420065d68 pc=0x644aad
main._uploadHeadimg.func1(0xc42026b9e0, 0xc420312200, 0xc420b4c4c0, 0xc420ee4840, 0xc4206c63c0, 0xc420312200)
/home/rchunping/go-projects/src/photouploader/move/move.go:268 +0x4d fp=0xc420065fb0 sp=0xc420065ef0 pc=0x86a81d
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420065fb8 sp=0xc420065fb0 pc=0x4581b1
created by main._uploadHeadimg
/home/rchunping/go-projects/src/photouploader/move/move.go:267 +0x53a
==================
transport.go:
// remove removes pc from cl.
func (cl *connLRU) remove(pc *persistConn) {
if ele, ok := cl.m[pc]; ok { <------- this line
cl.ll.Remove(ele)
delete(cl.m, pc)
}
}
===================
and some times, the panic caused by:
func (t *Transport) getIdleConnCh(cm connectMethod) chan *persistConn {
if t.DisableKeepAlives {
return nil
}
key := cm.key()
t.idleMu.Lock()
defer t.idleMu.Unlock()
t.wantIdle = false
if t.idleConnCh == nil {
t.idleConnCh = make(map[connectMethodKey]chan *persistConn)
}
ch, ok := t.idleConnCh[key] <---------- this line
if !ok {
ch = make(chan *persistConn)
t.idleConnCh[key] = ch
}
return ch
}
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
This comment has been minimized.
This comment has been minimized.
|
the main code: // download photo from url and upload to file server
func _uploadHeadimg(url string, token string) (*upload.PhotoResp, error) {
downloadUrl := url
client := &http.Client{}
if v, ok := http.DefaultTransport.(*http.Transport); ok {
tsport := *v
tsport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
client.Transport = http.RoundTripper(&tsport)
}
downloadReq, err := http.NewRequest("GET", downloadUrl, nil)
if err != nil {
return nil, err
}
var errchan chan error = make(chan error, 10)
resp, err := client.Do(downloadReq)
if err != nil {
return nil, err
}
defer resp.Body.Close()
contentType := resp.Header.Get("Content-Type")
if !strings.HasPrefix(contentType, "image/") {
contentType = "image/jpeg"
}
ur, uw := io.Pipe()
defer uw.Close()
uparts := multipart.NewWriter(uw)
defer uparts.Close()
uploadUrl := config.ApiUrl + "/upload"
client = &http.Client{}
uploadReq, err := http.NewRequest("POST", uploadUrl, ur)
if err != nil {
return nil, err
}
uploadReq.Header.Set("Access-Token", token)
uploadReq.Header.Set("Content-Type", "multipart/form-data; boundary="+uparts.Boundary())
var photoResp upload.PhotoResp
var done chan struct{} = make(chan struct{}, 1)
go func(ec chan error, done chan struct{}, req *http.Request) {
resp, err := client.Do(uploadReq)
if err != nil {
ec <- err
return
}
defer resp.Body.Close()
buf, _ := ioutil.ReadAll(resp.Body)
var errResp api.ErrResp
var okResp upload.PhotoResp
if err := json.Unmarshal(buf, &okResp); err == nil && okResp.Id > 0 {
photoResp = okResp
done <- struct{}{}
return
}
if err := json.Unmarshal(buf, &errResp); err != nil {
ec <- err
return
} else {
ec <- errResp
return
}
}(errchan, done, uploadReq)
h := make(textproto.MIMEHeader)
h.Set("Content-Disposition",
fmt.Sprintf(`form-data; name="%s"; filename="%s"`,
util.QuoteEscape("file"), util.QuoteEscape("headimg.jpg")))
h.Set("Content-Type", contentType)
fpw, err := uparts.CreatePart(h)
if err != nil {
return nil, err
}
go func(ec chan error) {
if _, err := io.Copy(fpw, resp.Body); err != nil {
ec <- err
return
}
uparts.Close()
uw.Close()
}(errchan)
select {
case err := <-errchan:
return nil, err
case <-done:
return &photoResp, nil
case <-time.After(5 * time.Minute):
return nil, errors.New("upload timeout")
}
}
```
after upload thousands of photos, the panic occurred |
This comment has been minimized.
This comment has been minimized.
|
Your error occurs because you are copying the contents of http.DefauktTransport
tsport := *v
This is not safe to do without a lock, and no lock for this operations is available with the default transport.
To fix your program you should construct your own http transport.
… On 22 Sep 2018, at 16:52, rchunping ***@***.***> wrote:
tsport := *v
|
This comment has been minimized.
This comment has been minimized.
|
I am closing this issue as it is not a bug in go. Unlike many projects on GitHub, the Go project does not use its bug tracker for general discussion or asking questions. We only use our bug tracker for tracking bugs and tracking proposals going through the Proposal Process. Please see https://golang.org/wiki/Questions for good places to ask questions. |
This comment has been minimized.
This comment has been minimized.
|
@davecheney thanks a lot! |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (
go version)?go version go1.10.3 freebsd/amd64
Does this issue reproduce with the latest release?
What operating system and processor architecture are you using (
go env)?FreeBSD rchunping.localhost 11.0-RELEASE-p2 FreeBSD 11.0-RELEASE-p2 #0: Mon Oct 24 06:55:27 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
What did you do?
a http client application, download photo and upload photo to another server
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.
What did you expect to see?
no error
What did you see instead?
==================
===================
and some times, the panic caused by: