Skip to content

runtime: garbage collector hangs when used as C binding in Python #55844

Closed
@tdewolff

Description

@tdewolff

What version of Go are you using (go version)?

$ go version
go version go1.19.1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/taco/.cache/go-build"
GOENV="/home/taco/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/taco/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/taco/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.1"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3914674171=/tmp/go-build -gno-record-gcc-switches"

What did you do?

See tdewolff/minify#535 for a bug report and how to reproduce. It concerns the Python binding for a library (https://github.com/tdewolff/minify/tree/master/bindings/py), which can be installed as pip install tdewolff-minify. Then running python test.py where test.py contains:

import multiprocessing
import minify

def minify_worker(number):
    minify.string('text/html', f"<p>{number}</p>"*10000)
    print(".", end="", flush=True)

if __name__ == "__main__":
    number_list = range(0, 1000)
    processing_pool = multiprocessing.Pool()
    processing_pool.map(minify_worker, number_list)
    print("Done!")

This hangs, but when running GOGC=off python test.py it works fine. Actually, calling runtime.GC() directly in the library causes a hang, suggesting that the garbage collector is responsible for hanging. Note that when using multiprocessing with Python, library function calls are put on a different thread from the one that initialized the Go package (init()).

What is causing the GC to hang? How can this be debugged?

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions