database/sql: panic in the Scanner interface implementation causes a deadlock

[meetmorrowsolonmars]

Go version

go1.25.0 darwin/arm64

Output of go env in your module/workspace:

AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE='on'
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/Users/gopher/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/gopher/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/9f/5chpt_kx5kq9_fnv1tsb63080000gn/T/go-build696434100=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/Users/gopher/Projects/sql-db-issue/go.mod'
GOMODCACHE='/Users/gopher/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/gopher/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/gopher/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.0.darwin-arm64'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/gopher/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/gopher/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.0.darwin-arm64/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.25.0'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

I found this issue working with github.com/uptrace/bun and JSONB column type in PostgreSQL. Their implementation of Scanner interface uses reflect package to unmarshal JSONB field. In my case for type map[string]any it causes panic and database connection stays in idle in transaction state.

I wrote a small code snippet to show the problem. Let's say we work with a custom text format like XML or CSV and want to unmarshal the column data into our custom struct. In this case, if for some reason out implementation of Scan method panic, then we will see nothing. The Rows.Scan method will be blocked, and with it the database transaction.

package main

import (
	"database/sql"
	"errors"
	"fmt"
	"log"

	_ "github.com/jackc/pgx/v5/stdlib"
)

func main() {
	db, err := sql.Open("pgx", "host=127.0.0.1 port=55432 user=postgres dbname=postgres sslmode=disable")
	if err != nil {
		log.Fatalln(err)
	}

	log.Println("Connected to database")

	_, err = db.Exec("CREATE TABLE IF NOT EXISTS test (id BIGINT PRIMARY KEY, data TEXT);")
	if err != nil {
		log.Fatalln(err)
	}

	log.Println("Created \"test\" table")

	_, err = db.Exec("INSERT INTO test (id, data) VALUES ($1, $2) ON CONFLICT DO NOTHING", 1, "Hello, World!")
	if err != nil {
		log.Fatalln(err)
	}

	log.Println("Test data inserted")

	var result model

	row := db.QueryRow("SELECT id, data FROM test WHERE id = $1", 1)

	err = row.Scan(&result.ID, &result.Data)
	if err != nil {
		log.Fatalln(err)
	}

	fmt.Println(result)
}

type model struct {
	ID   int64
	Data field
}

type field struct {
	Data string
}

func (f *field) Scan(src any) error {
	switch src.(type) {
	case string:
		f.Data = src.(string)
		panic("can not unmarshal `field` data")
	default:
		return errors.New("invalid type")
	}

	return nil
}

Stack trace.

fatal error: all goroutines are asleep - deadlock!

goroutine 1 [sync.RWMutex.Lock]:
sync.runtime_SemacquireRWMutex(0x10?, 0x60?, 0x14000161801?)
	/Users/gopher/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.0.darwin-arm64/src/runtime/sema.go:105 +0x28
sync.(*RWMutex).Lock(0x14000161888?)
	/Users/gopher/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.0.darwin-arm64/src/sync/rwmutex.go:155 +0xf4
database/sql.(*Rows).close(0x1400010c500, {0x0, 0x0})
	/Users/gopher/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.0.darwin-arm64/src/database/sql/sql.go:3443 +0x80
database/sql.(*Rows).Close(0x1400010c500)
	/Users/gopher/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.0.darwin-arm64/src/database/sql/sql.go:3439 +0x30
panic({0x102d3e800?, 0x102df0470?})
	/Users/gopher/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.0.darwin-arm64/src/runtime/panic.go:783 +0x120
main.(*field).Scan(0x10310cad0?, {0x102d3e800?, 0x1400001e470?})
	/Users/gopher/Projects/sql-db-issue/main.go:59 +0xa8
database/sql.convertAssignRows({0x102d4faa0, 0x1400000c218}, {0x102d3e800, 0x1400001e470}, 0x1400010c500)
	/Users/gopher/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.0.darwin-arm64/src/database/sql/convert.go:394 +0x1a48
database/sql.(*Rows).scanLocked(0x1400010c500, {0x14000161ee0, 0x2, 0x10001029c75b0?})
	/Users/gopher/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.0.darwin-arm64/src/database/sql/sql.go:3399 +0x1f0
database/sql.(*Rows).Scan(0x1400010c500, {0x14000161ee0, 0x2, 0x102c7673c?})
	/Users/gopher/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.0.darwin-arm64/src/database/sql/sql.go:3374 +0x9c
database/sql.(*Row).Scan(0x1400000c228, {0x14000161ee0?, 0x2?, 0x2})
	/Users/gopher/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.0.darwin-arm64/src/database/sql/sql.go:3515 +0x118
main.main()
	/Users/gopher/Projects/sql-db-issue/main.go:38 +0x2ac

What did you see happen?

The Rows.Scan method does not exit and stays blocked.
The database connection are blocked.
The transaction stays in idle in transaction state with ClientRead wait event.

What did you expect to see?

The Rows.Scan method returns error and flushes all data in database connection to use it again. Otherwise, this issue can stay unhandled and cause problems for example with connection pool, for example.

[gabyhelp]

Related Issues

• database/sql: possible deadlock on Rows.awaitDone()/Rows.Err()/Rows.Close() after RawBytes change #64498
• database/sql: Tx.Rollback hangs after panic in Valuer.Value #26332 (closed)
• database/sql: panic in DB.QueryRow #61646 (closed)
• database/sql: panic in `database/sql.(*connRequestSet).deleteIndex` #68949 (closed)
• database/sql: rows.Next panic from concurrent map writes #27448 (closed)
• database/sql: sync.Mutex unlocked when panic cause by context #54110 (closed)

Related Discussions

• database/sql QueryRow Scan panic
• possible bug: Rollback hangs after panic

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

I think CL 721207 would fix this
though the correct thing to do is to let the panic propagate, not to convert it to an error.

[meetmorrowsolonmars]

I think CL 721207 would fix this though the correct thing to do is to let the panic propagate, not to convert it to an error.

Yeah, sounds nice. Thank you!

[firodj]

thanks for the investage & report, in my case it is deadlock even happen in non-transaction connection.
I was using bun.DB, and the panic happen when trying to resolve/reflect jsonb column into struct field with any field.
But the panic gone when the field type were struct, map[string]any, *string, or json.RawMessage ([]byte), and the deadlock won't happen.

go1.25.1 linux/amd64