Crashed when GOMAXPROCS > 1

[iapyeh]

I use the go-python3 to hook a snippet of python to the fasthttp package
(github.com/valyala/fasthttp)
It works like a charm if GOMAXPROCS==1 (with 10K requests/seconds)
But it crashed if GOMAXPROCS > 1 (such as 2, 4,...)
I have tried to acquire/release GIL, it crashes soon (not even 10 requests)
I have tried to use sync.Mutex.Lock/Unlock. It crashed too after 200 requests.
(Error: fatal error: unexpected signal during runtime execution)

Do you have any suggestion?

[hush-hush]

Hi @iapyeh,

tl;dr: My first guess would be that you need to call LockOSThread (https://golang.org/pkg/runtime/#LockOSThread) to prevent go switching thread outside Python.

More details:
Python acquire the lock based on the OS thread ID, but Golang might switch the OS thread of a goroutine at any time. This means that you might acquire the GIL from thread A but come back in Python from thread B.

So in the end when moving from Python to go: lockOSThread + lock the GIL, [do stuff in Go], unlock GIL, unlock thread.

We wrote an article on this https://www.datadoghq.com/blog/engineering/cgo-and-python/ (with a presentation from PyGotham).

[hush-hush]

I'm closing this issue, but feel free to reopen it (or a new one) if needed !