Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: panic: persistConn was already in LRU #18061

Closed
fmzquant opened this issue Nov 27, 2016 · 7 comments

Comments

Projects
None yet
4 participants
@fmzquant
Copy link

commented Nov 27, 2016

What version of Go are you using (go version)?

go1.7.3

What operating system and processor architecture are you using (go env)?

Windows Server Standard 2007 SP2 32bit

panic: persistConn was already in LRU

goroutine 609111 [running]:
panic(0xa50dc0, 0x125a45d8)
        /opt/dev/gopkg/go/src/runtime/panic.go:500 +0x331
net/http.(*connLRU).add(0x125de5b4, 0x1267c510)
        /opt/dev/gopkg/go/src/net/http/transport.go:2147 +0x177
net/http.(*Transport).tryPutIdleConn(0x125de5a0, 0x1267c510, 0x0, 0x0)
        /opt/dev/gopkg/go/src/net/http/transport.go:678 +0x5d8
net/http.(*persistConn).readLoop.func2(0x0, 0x125f2b01)
        /opt/dev/gopkg/go/src/net/http/transport.go:1391 +0x32
net/http.(*persistConn).readLoop(0x1267c510)
        /opt/dev/gopkg/go/src/net/http/transport.go:1548 +0xce1
created by net/http.(*Transport).dialConn
        /opt/dev/gopkg/go/src/net/http/transport.go:1062 +0xda5

this is part of my code:


func DoRequest(/* som arguments */) {
        // Very high frequency call code
	var res *http.Response
	err = nil
	if r.Timeout > 0 {
		err = errors.New("timeout")
		c := make(chan *httpPeek)
		go func() {
			newRes, newErr := httpClient.Do(req)
			select {
			case c <- &httpPeek{newRes, newErr}:
			default:
				if newErr == nil && newRes != nil && newRes.Body != nil {
					// Discard and close so we can reuse it
					io.Copy(ioutil.Discard, newRes.Body)
					newRes.Body.Close()
				}
			}
			close(c)
		}()
		select {
		case <-time.After(r.Timeout):
		case v := <-c:
			if v != nil {
				res = v.res
				err = v.err
			}
		}
	} else {
		res, err = httpClient.Do(req)
	}
	if err != nil {
		return nil, err
	}
      // other code..
}
func init() {
	timeout := time.Minute
	httpClient = &http.Client{
		Transport: &http.Transport{
			MaxIdleConnsPerHost:   5,
			TLSClientConfig:       &tls.Config{InsecureSkipVerify: true},
			ResponseHeaderTimeout: timeout,
		},
		Timeout: timeout,
	}
}

It's very hard to reproduce, program running about 20 hours, then panic and exit !!

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Nov 27, 2016

@bradfitz bradfitz changed the title panic: persistConn was already in LRU net/http: panic: persistConn was already in LRU Nov 27, 2016

@bradfitz bradfitz added this to the Go1.8Maybe milestone Nov 28, 2016

@bradfitz

This comment has been minimized.

Copy link
Member

commented Nov 28, 2016

I don't see how this can happen, but it also sounds disconcerting.

Staring at the code, my best guess is it's something around cancelation,

func (pc *persistConn) roundTrip(req *transportRequest) (resp *Response, err error) {
        testHookEnterRoundTrip()
        if !pc.t.replaceReqCanceler(req.Request, pc.cancelRequest) {
                pc.t.putOrCloseIdleConn(pc)
                return nil, errRequestCanceled
        }

And the persistConn.readLoop also adding it... somehow. I can't quite see a path where this is possible.

Any help wanted.

@fmzquant

This comment has been minimized.

Copy link
Author

commented Dec 2, 2016

The same thing happened again, I believe net/http library causing this bug.

When I testing, I found a very strange question about map structure:

func (ctx *SessionCtx) Handle(method uint8, handler func([]byte) []byte) {
	ctx.mutex.Lock()
	ctx.dispatcher[method] = handler
	ctx.mutex.Unlock()
}

I put a method when init this Session:

ctx.Handle(TID_Ping, somefunc)

but I never change or deleted this key in map, In a high frequency code block like:

func (ctx *SessionCtx) OnPing() {
    ctx.mutex.Lock()
    handler, ok := ctx.dispatcher[TID_Ping]
    ctx.mutex.Unlock()

    // running some days, ok return is false !!!, but I never delete this key, GC's BUG ??
}

program running some days, something strange happen, ok return false, means not found the key, But I never deleted this key after I insert to the map, Is GC's bug ??

I think the problem might be related to the net/http problem

Test ENV:

Windows Server Standard 2007 SP2 32bit

@bradfitz

This comment has been minimized.

Copy link
Member

commented Dec 2, 2016

@botvs, sorry, I don't understand.

Could you please answer @davecheney's question first? Have you run your program under the race detector? https://blog.golang.org/race-detector

@fmzquant

This comment has been minimized.

Copy link
Author

commented Dec 2, 2016

@bradfitz I'm starting to test with race-detector now.

@fmzquant

This comment has been minimized.

Copy link
Author

commented Dec 2, 2016

@bradfitz
go build: -race and -msan are only supported on linux/amd64, freebsd/amd64, darwin/amd64 and windows/amd64

bug triggered only in windows 32bit.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Dec 2, 2016

Sorry. You'll have to use a 64-bit OS to run the race detector.

Maybe you can test on a cloud VM or something if you can't install a 64-bit OS on your machine. (I imagine your machine is actually a 64-bit processor)

I'm going to close this bug for now since there's nothing actionable for us at the moment. I can't find a path in the code where this can happen and without a repro I can't debug further.

Let's move conversation to https://golang.org/wiki/Questions until there's something concrete for us to fix.

@bradfitz bradfitz closed this Dec 2, 2016

@bradfitz bradfitz removed this from the Go1.8Maybe milestone Dec 2, 2016

@golang golang locked and limited conversation to collaborators Dec 2, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.