Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

database/sql: deadlock opening new connections #59262

Open
sxfworks opened this issue Mar 27, 2023 · 1 comment
Open

database/sql: deadlock opening new connections #59262

sxfworks opened this issue Mar 27, 2023 · 1 comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@sxfworks
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.19.1 darwin/arm64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/sxfworks/Library/Caches/go-build"
GOENV="/Users/sxfworks/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/sxfworks/go/pkg/mod"
GONOPROXY="*.everphoto.cn,git.smartisan.com"
GONOSUMDB="*.everphoto.cn,git.smartisan.com"
GOOS="darwin"
GOPATH="/Users/sxfworks/go"
GOPRIVATE="*.everphoto.cn,git.smartisan.com"
GOPROXY="https://goproxy.byted.org,direct"
GOROOT="/opt/homebrew/Cellar/go/1.19.1/libexec"
GOSUMDB="sum.golang.google.cn"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.19.1/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.19.1"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="0"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/b1/0fd1b6hs7lz0fm_mh346lybm0000gn/T/go-build3136756528=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I run my program under high concurrency and frequently read and write databases, but the maximum number of mysql connections is limited to 10, and the program gets stuck in the following logic after running for a period of time:

goroutine 146 [select, 13 minutes]:
database/sql.(*DB).connectionOpener(0xc000502d00, {0x1a9bc28, 0xc00052d240})
/usr/local/go/src/database/sql/sql.go:1226 +0x8d
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:794 +0x18d

goroutine 203 [select, 12 minutes]:
database/sql.(*DB).conn(0xc0002ab5f0, {0x1a9bc60, 0xc000120000}, 0x1)
/usr/local/go/src/database/sql/sql.go:1343 +0x3da
database/sql.(*DB).begin(0xc0044d2000?, {0x1a9bc60, 0xc000120000}, 0xc00368d590?, 0xac?)
/usr/local/go/src/database/sql/sql.go:1869 +0x33
database/sql.(*DB).BeginTx(0xc000fe16e0?, {0x1a9bc60, 0xc000120000}, 0x0?)
/usr/local/go/src/database/sql/sql.go:1847 +0x7e
gorm.io/gorm.(*DB).Begin(0xc000b79410, {0x0, 0x0, 0xc0000441d0?})
/opt/tiger/compile_path/pkg/mod/gorm.io/gorm@v1.24.2/finisher_api.go:647 +0x11d

After analysis, I think that connectionOpener calls openNewConnection, but openNewConnection fails, causing db.maybeOpenNewConnections() to be called again, and db.openerCh <- struct{}{} will be written inside db.maybeOpenNewConnections(), and db.openerCh is filled with a large number of finisher_api write, causing connectionOpener to be stuck writing db.openerCh, and many finisher_api are also waiting for connectionOpener to wait new connections, waiting for each other to cause deadlock

What did you expect to see?

database/sql shouldn't block on conn

What did you see instead?

database/sql block on conn

@sxfworks sxfworks changed the title affected/package: database/sql affected/package: database/sql may prevent code from running because db.openerCh <- struct{}{} gets filled Mar 27, 2023
@seankhliao seankhliao changed the title affected/package: database/sql may prevent code from running because db.openerCh <- struct{}{} gets filled database/sql: deadlock opening new connections Mar 27, 2023
@seankhliao seankhliao added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 27, 2023
@bcmills
Copy link
Contributor

bcmills commented Apr 3, 2023

@bcmills bcmills added this to the Backlog milestone Apr 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

3 participants