net/http: concurrent map read and map write

[rchunping]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.10.3 freebsd/amd64

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

FreeBSD rchunping.localhost 11.0-RELEASE-p2 FreeBSD 11.0-RELEASE-p2 #0: Mon Oct 24 06:55:27 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64

What did you do?

a http client application, download photo and upload photo to another server

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

What did you expect to see?

no error

What did you see instead?

goroutine 801303 [running]:
runtime.throw(0x99a1dc, 0x21)
        /usr/local/go/src/runtime/panic.go:616 +0x81 fp=0xc4200654a8 sp=0xc420065488 pc=0x42ba11
runtime.mapaccess2_fast64(0x8ec7a0, 0xc42026b6e0, 0xc420713d40, 0xc4201dadf0, 0x1)
        /usr/local/go/src/runtime/hashmap_fast.go:141 +0x1aa fp=0xc4200654d0 sp=0xc4200654a8 pc=0x40b93a
net/http.(*connLRU).remove(...)
        /usr/local/go/src/net/http/transport.go:2314
net/http.(*Transport).getIdleConn(0xc4df60, 0x0, 0x99c98d, 0x5, 0xc42054e180, 0x12, 0xc420713d40, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:807 +0x215 fp=0xc4200655f8 sp=0xc4200654d0 pc=0x6960c5
net/http.(*Transport).getConn(0xc4df60, 0xc420c11ad0, 0x0, 0x99c98d, 0x5, 0xc42054e180, 0x12, 0x0, 0x0, 0x433eec)
        /usr/local/go/src/net/http/transport.go:921 +0x11d fp=0xc420065978 sp=0xc4200655f8 pc=0x696dcd
net/http.(*Transport).RoundTrip(0xc4df60, 0xc420312200, 0xc4df60, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:409 +0x632 fp=0xc420065bc0 sp=0xc420065978 pc=0x693a82
net/http.send(0xc420312200, 0x9ed180, 0xc4df60, 0x0, 0x0, 0x0, 0xc42000e560, 0xc42049b4a0, 0xc420065d58, 0x1)
        /usr/local/go/src/net/http/client.go:252 +0x185 fp=0xc420065ce8 sp=0xc420065bc0 pc=0x643785
net/http.(*Client).send(0xc42026b9e0, 0xc420312200, 0x0, 0x0, 0x0, 0xc42000e560, 0x0, 0x1, 0x0)
        /usr/local/go/src/net/http/client.go:176 +0xfa fp=0xc420065d68 sp=0xc420065ce8 pc=0x64345a
net/http.(*Client).Do(0xc42026b9e0, 0xc420312200, 0xc42057a000, 0x9ed1c0, 0xc420780340)
        /usr/local/go/src/net/http/client.go:615 +0x28d fp=0xc420065ef0 sp=0xc420065d68 pc=0x644aad
main._uploadHeadimg.func1(0xc42026b9e0, 0xc420312200, 0xc420b4c4c0, 0xc420ee4840, 0xc4206c63c0, 0xc420312200)
        /home/rchunping/go-projects/src/photouploader/move/move.go:268 +0x4d fp=0xc420065fb0 sp=0xc420065ef0 pc=0x86a81d
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420065fb8 sp=0xc420065fb0 pc=0x4581b1
created by main._uploadHeadimg
        /home/rchunping/go-projects/src/photouploader/move/move.go:267 +0x53a

==================

transport.go:

// remove removes pc from cl.
func (cl *connLRU) remove(pc *persistConn) {
	if ele, ok := cl.m[pc]; ok {  <------- this line
		cl.ll.Remove(ele)
		delete(cl.m, pc)
	}
}

===================

and some times, the panic caused by:

func (t *Transport) getIdleConnCh(cm connectMethod) chan *persistConn {
	if t.DisableKeepAlives {
		return nil
	}
	key := cm.key()
	t.idleMu.Lock()
	defer t.idleMu.Unlock()
	t.wantIdle = false
	if t.idleConnCh == nil {
		t.idleConnCh = make(map[connectMethodKey]chan *persistConn)
	}
	ch, ok := t.idleConnCh[key]  <---------- this line
	if !ok {
		ch = make(chan *persistConn)
		t.idleConnCh[key] = ch
	}
	return ch
}

[davecheney]

Can you please provide a complete runnable code sample that demonstrates this fault, or causes the race detector to complain.

…

On 22 Sep 2018, at 16:34, rchunping ***@***.***> wrote: Please answer these questions before submitting your issue. Thanks! What version of Go are you using (go version)? go version go1.10.3 freebsd/amd64 Does this issue reproduce with the latest release? What operating system and processor architecture are you using (go env)? FreeBSD rchunping.localhost 11.0-RELEASE-p2 FreeBSD 11.0-RELEASE-p2 #0: Mon Oct 24 06:55:27 UTC 2016 ***@***.***:/usr/obj/usr/src/sys/GENERIC amd64 What did you do? a http client application, download photo and upload photo to another server If possible, provide a recipe for reproducing the error. A complete runnable program is good. A link on play.golang.org is best. What did you expect to see? no error What did you see instead? goroutine 801303 [running]: runtime.throw(0x99a1dc, 0x21) /usr/local/go/src/runtime/panic.go:616 +0x81 fp=0xc4200654a8 sp=0xc420065488 pc=0x42ba11 runtime.mapaccess2_fast64(0x8ec7a0, 0xc42026b6e0, 0xc420713d40, 0xc4201dadf0, 0x1) /usr/local/go/src/runtime/hashmap_fast.go:141 +0x1aa fp=0xc4200654d0 sp=0xc4200654a8 pc=0x40b93a net/http.(*connLRU).remove(...) /usr/local/go/src/net/http/transport.go:2314 net/http.(*Transport).getIdleConn(0xc4df60, 0x0, 0x99c98d, 0x5, 0xc42054e180, 0x12, 0xc420713d40, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/transport.go:807 +0x215 fp=0xc4200655f8 sp=0xc4200654d0 pc=0x6960c5 net/http.(*Transport).getConn(0xc4df60, 0xc420c11ad0, 0x0, 0x99c98d, 0x5, 0xc42054e180, 0x12, 0x0, 0x0, 0x433eec) /usr/local/go/src/net/http/transport.go:921 +0x11d fp=0xc420065978 sp=0xc4200655f8 pc=0x696dcd net/http.(*Transport).RoundTrip(0xc4df60, 0xc420312200, 0xc4df60, 0x0, 0x0) /usr/local/go/src/net/http/transport.go:409 +0x632 fp=0xc420065bc0 sp=0xc420065978 pc=0x693a82 net/http.send(0xc420312200, 0x9ed180, 0xc4df60, 0x0, 0x0, 0x0, 0xc42000e560, 0xc42049b4a0, 0xc420065d58, 0x1) /usr/local/go/src/net/http/client.go:252 +0x185 fp=0xc420065ce8 sp=0xc420065bc0 pc=0x643785 net/http.(*Client).send(0xc42026b9e0, 0xc420312200, 0x0, 0x0, 0x0, 0xc42000e560, 0x0, 0x1, 0x0) /usr/local/go/src/net/http/client.go:176 +0xfa fp=0xc420065d68 sp=0xc420065ce8 pc=0x64345a net/http.(*Client).Do(0xc42026b9e0, 0xc420312200, 0xc42057a000, 0x9ed1c0, 0xc420780340) /usr/local/go/src/net/http/client.go:615 +0x28d fp=0xc420065ef0 sp=0xc420065d68 pc=0x644aad main._uploadHeadimg.func1(0xc42026b9e0, 0xc420312200, 0xc420b4c4c0, 0xc420ee4840, 0xc4206c63c0, 0xc420312200) /home/rchunping/go-projects/src/photouploader/move/move.go:268 +0x4d fp=0xc420065fb0 sp=0xc420065ef0 pc=0x86a81d runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420065fb8 sp=0xc420065fb0 pc=0x4581b1 created by main._uploadHeadimg /home/rchunping/go-projects/src/photouploader/move/move.go:267 +0x53a ================== transport.go: // remove removes pc from cl. func (cl *connLRU) remove(pc *persistConn) { if ele, ok := cl.m[pc]; ok { <------- this line cl.ll.Remove(ele) delete(cl.m, pc) } } =================== and some times, the panic caused by: func (t *Transport) getIdleConnCh(cm connectMethod) chan *persistConn { if t.DisableKeepAlives { return nil } key := cm.key() t.idleMu.Lock() defer t.idleMu.Unlock() t.wantIdle = false if t.idleConnCh == nil { t.idleConnCh = make(map[connectMethodKey]chan *persistConn) } ch, ok := t.idleConnCh[key] <---------- this line if !ok { ch = make(chan *persistConn) t.idleConnCh[key] = ch } return ch } — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[rchunping]

@davecheney

the main code:

//  download photo from url and upload to file server
func _uploadHeadimg(url string, token string) (*upload.PhotoResp, error) {
	downloadUrl := url
	client := &http.Client{}
	if v, ok := http.DefaultTransport.(*http.Transport); ok {
		tsport := *v
		tsport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
		client.Transport = http.RoundTripper(&tsport)
	}
	downloadReq, err := http.NewRequest("GET", downloadUrl, nil)
	if err != nil {
		return nil, err
	}
	var errchan chan error = make(chan error, 10)
	resp, err := client.Do(downloadReq)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()
	contentType := resp.Header.Get("Content-Type")
	if !strings.HasPrefix(contentType, "image/") {
		contentType = "image/jpeg"
	}
	ur, uw := io.Pipe()
	defer uw.Close()
	uparts := multipart.NewWriter(uw)
	defer uparts.Close()

	uploadUrl := config.ApiUrl + "/upload"
	client = &http.Client{}
	uploadReq, err := http.NewRequest("POST", uploadUrl, ur)
	if err != nil {
		return nil, err
	}

	uploadReq.Header.Set("Access-Token", token)
	uploadReq.Header.Set("Content-Type", "multipart/form-data; boundary="+uparts.Boundary())

	var photoResp upload.PhotoResp
	var done chan struct{} = make(chan struct{}, 1)
	go func(ec chan error, done chan struct{}, req *http.Request) {
		resp, err := client.Do(uploadReq)
		if err != nil {
			ec <- err
			return
		}
		defer resp.Body.Close()
		buf, _ := ioutil.ReadAll(resp.Body)
		var errResp api.ErrResp
		var okResp upload.PhotoResp
		if err := json.Unmarshal(buf, &okResp); err == nil && okResp.Id > 0 {
			photoResp = okResp
			done <- struct{}{}
			return
		}

		if err := json.Unmarshal(buf, &errResp); err != nil {
			ec <- err
			return
		} else {
			ec <- errResp
			return
		}

	}(errchan, done, uploadReq)

	h := make(textproto.MIMEHeader)
	h.Set("Content-Disposition",
		fmt.Sprintf(`form-data; name="%s"; filename="%s"`,
			util.QuoteEscape("file"), util.QuoteEscape("headimg.jpg")))
	h.Set("Content-Type", contentType) 
	fpw, err := uparts.CreatePart(h)
	if err != nil {
		return nil, err
	}

	go func(ec chan error) {
		if _, err := io.Copy(fpw, resp.Body); err != nil {
			ec <- err
			return
		}

		uparts.Close()
		uw.Close()
	}(errchan)
	select {
	case err := <-errchan:
		return nil, err
	case <-done:
		return &photoResp, nil

	case <-time.After(5 * time.Minute):
		return nil, errors.New("upload timeout")
	}
}
```

 after upload thousands of photos, the panic occurred

[davecheney]

Your error occurs because you are copying the contents of http.DefauktTransport tsport := *v This is not safe to do without a lock, and no lock for this operations is available with the default transport. To fix your program you should construct your own http transport.

…

On 22 Sep 2018, at 16:52, rchunping ***@***.***> wrote: tsport := *v

[davecheney]

I am closing this issue as it is not a bug in go.

Unlike many projects on GitHub, the Go project does not use its bug tracker for general discussion or asking questions. We only use our bug tracker for tracking bugs and tracking proposals going through the Proposal Process.

Please see https://golang.org/wiki/Questions for good places to ask questions.

[rchunping]

@davecheney thanks a lot!