Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: garbage collector hangs when used as C binding in Python #55844

Closed
tdewolff opened this issue Sep 23, 2022 · 1 comment
Closed

runtime: garbage collector hangs when used as C binding in Python #55844

tdewolff opened this issue Sep 23, 2022 · 1 comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime.

Comments

@tdewolff
Copy link

tdewolff commented Sep 23, 2022

What version of Go are you using (go version)?

$ go version
go version go1.19.1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/taco/.cache/go-build"
GOENV="/home/taco/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/taco/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/taco/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.1"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3914674171=/tmp/go-build -gno-record-gcc-switches"

What did you do?

See tdewolff/minify#535 for a bug report and how to reproduce. It concerns the Python binding for a library (https://github.com/tdewolff/minify/tree/master/bindings/py), which can be installed as pip install tdewolff-minify. Then running python test.py where test.py contains:

import multiprocessing
import minify

def minify_worker(number):
    minify.string('text/html', f"<p>{number}</p>"*10000)
    print(".", end="", flush=True)

if __name__ == "__main__":
    number_list = range(0, 1000)
    processing_pool = multiprocessing.Pool()
    processing_pool.map(minify_worker, number_list)
    print("Done!")

This hangs, but when running GOGC=off python test.py it works fine. Actually, calling runtime.GC() directly in the library causes a hang, suggesting that the garbage collector is responsible for hanging. Note that when using multiprocessing with Python, library function calls are put on a different thread from the one that initialized the Go package (init()).

What is causing the GC to hang? How can this be debugged?

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Sep 23, 2022
@tdewolff
Copy link
Author

tdewolff commented Sep 24, 2022

This seems to be an issue with forking a Go thread. Since Go launches an additional thread for the GC, when forking the main thread the other GC thread is not forked as well, causing it to hang when a forked thread tries to reach the GC thread of the original process. The solution is to not fork Go, but use the spawn strategy for example.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime.
Projects
None yet
Development

No branches or pull requests

2 participants