New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

database/sql: Goroutine freezes in database/sql. #23453

Closed
sundagy opened this Issue Jan 15, 2018 · 4 comments

Comments

Projects
None yet
5 participants
@sundagy
Copy link

sundagy commented Jan 15, 2018

In this weekends we came across a very unpleasant bug. The our server application is freezed after about 50min of works.
Using tool pprof/goroutine we did investigation and found that all 4 workers hang in the database/sql package:

ROUTINE ======================== database/sql.(*DB).conn in /usr/local/go/src/database/sql/sql.go
         0         20 (flat, cum) 0.0045% of Total
         .          .    965:           reqKey := db.nextRequestKeyLocked()
         .          .    966:           db.connRequests[reqKey] = req
         .          .    967:           db.mu.Unlock()
         .          .    968:
         .          .    969:           // Timeout the connection request with the context.
         .         20    970:           select {
         .          .    971:           case <-ctx.Done():
         .          .    972:                   // Remove the connection request and ensure no value has been sent
         .          .    973:                   // on it after removing.
         .          .    974:                   db.mu.Lock()
         .          .    975:                   delete(db.connRequests, reqKey)

What is most striking that until today's weekend the our server application for 2 past years worked stably. And hasn't meet such kind of bug.

Tool like netstat didn't see any network activity with mysql.

Mysql driver is here:
https://github.com/go-sql-driver/mysql

I know the sufficiency of information is poor and the bug is probably just not to repeat so easily.
But can someone tell what is going on in this part of that package. I think this will help me to find clue.

What version of Go are you using (go version)?

go version go1.9.2 linux/amd64

Does this issue reproduce with the latest release?

I don't know.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/dev-pon/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build053765279=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you expect to see?

Processing query or return error.

What did you see instead?

After 50 minutes it is impossible to make queries to the database and the whole goroutine freeze.

@davecheney

This comment has been minimized.

Copy link
Contributor

davecheney commented Jan 15, 2018

pprof is the wrong tool for this job. You want to take a stack dump by sending SIGQUIT to your process.

I am going to close this issue as it doesn't appear to be a bug in Go. Unlike many projects on GitHub, the Go project does not use its bug tracker for general discussion or asking questions. We only use our bug tracker for tracking bugs and tracking proposals going through the Proposal Process.

Please see https://golang.org/wiki/Questions for good places to ask questions. If it turns out there is a bug in Go, they'll redirect you back here.

@davecheney davecheney closed this Jan 15, 2018

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Jan 15, 2018

@kardianos

This comment has been minimized.

Copy link
Contributor

kardianos commented Jan 15, 2018

I'm slightly unsure of the exact issue you are experiencing but it sounds like a connection leak perhaps. Do make sure your transactions are being committed / rolled back.

I would also recommend you try out the go1.10 beta2 if you are able to.

@mikioh mikioh changed the title Goroutine freezes in database/sql. database/sql: Goroutine freezes in database/sql. Jan 16, 2018

@sundagy

This comment has been minimized.

Copy link
Author

sundagy commented Jan 21, 2018

Indeed, it was connection leak.

@golang golang locked and limited conversation to collaborators Jan 21, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.