runtime: garbage collector hangs when used as C binding in Python

[tdewolff]

What version of Go are you using (go version)?

$ go version
go version go1.19.1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/taco/.cache/go-build"
GOENV="/home/taco/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/taco/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/taco/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.1"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3914674171=/tmp/go-build -gno-record-gcc-switches"

What did you do?

See tdewolff/minify#535 for a bug report and how to reproduce. It concerns the Python binding for a library (https://github.com/tdewolff/minify/tree/master/bindings/py), which can be installed as pip install tdewolff-minify. Then running python test.py where test.py contains:

import multiprocessing
import minify

def minify_worker(number):
    minify.string('text/html', f"<p>{number}</p>"*10000)
    print(".", end="", flush=True)

if __name__ == "__main__":
    number_list = range(0, 1000)
    processing_pool = multiprocessing.Pool()
    processing_pool.map(minify_worker, number_list)
    print("Done!")

This hangs, but when running GOGC=off python test.py it works fine. Actually, calling runtime.GC() directly in the library causes a hang, suggesting that the garbage collector is responsible for hanging. Note that when using multiprocessing with Python, library function calls are put on a different thread from the one that initialized the Go package (init()).

What is causing the GC to hang? How can this be debugged?

[tdewolff]

This seems to be an issue with forking a Go thread. Since Go launches an additional thread for the GC, when forking the main thread the other GC thread is not forked as well, causing it to hang when a forked thread tries to reach the GC thread of the original process. The solution is to not fork Go, but use the spawn strategy for example.