Skip to content

feat: matrix build python-version: ['3.11', '3.12']#399

Draft
b-long wants to merge 5 commits into
masterfrom
feature/pr-398-regression-test
Draft

feat: matrix build python-version: ['3.11', '3.12']#399
b-long wants to merge 5 commits into
masterfrom
feature/pr-398-regression-test

Conversation

@b-long

@b-long b-long commented Jul 24, 2026

Copy link
Copy Markdown
Member

This PR is serving two goals:

  1. It regression tests bind: make _gopy_clear_go_tls opt-in, off by default #398 , by enabling Python 3.12 builds
  2. It identifies and fixes (commit d41db33 ) issue Complex-returning functions crash under CPython 3.12 #400

While we shouldn't combine the two, I'm doing so to be pragmatic and move quickly (for now). Depending on feedback, I may split this PR into 2 separate parts.

Fixes: #400

b-long and others added 4 commits July 23, 2026 21:13
The unconditional _gopy_clear_go_tls() call (issue #370) performs a
hardcoded TLS store (movq $0, %fs:-8 on linux/amd64). On glibc + CPython
3.12+ that offset overlaps CPython's current-thread-state TLS slot, so the
store nulls it and the interpreter segfaults on the first CGo entry in the
common single-extension case (issue #395).

Add a -clear-go-tls build flag (default false) and gate the single call
site on it. The C helper and its pybindgen registration are left in place
so opt-in restores the previous behavior exactly. RTLD_GLOBAL-local
loading, which is what actually isolates each runtime's goroutine-pointer
TLS, is untouched and remains unconditional.

Fixes #395
Ensure GIL state before allocating; fixes stale thread-state segfault.
@satarsa

satarsa commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Thanks for merging #398! Wiring up the 3.12 matrix is exactly the coverage this needed, I'm happy to have a look. I've reproduced #400 locally end-to-end before writing this, so below is what I've found, a confirmation that your fix is correct, and one structural observation you may want to weigh for later.

As I now understand, #400 is a real bug, fully independent of #398. Your PyGILState_Ensure / Release around PyComplex_FromDoubles is correct and I verified it fixes the crash. The root cause is that gopy releases the GIL (PyEval_SaveThread) around the whole Go wrapper body, and the complex conversion is the one path that allocates a Python object inside that GIL-released window. That's why only complex crashes and, e.g., int or bytes do not.

Environment

Gentoo, glibc, system CPython 3.14.6, go 1.26.4, gcc 16, x86_64. Note 3.12 is only the floor: this class of crash is the glibc + CPython-3.12+ layout, not a specific minor, so 3.14 reproduces #400 identically. Your CI hitting it on 3.12 and me hitting it on 3.14 are the same bug.

1. #400 is real and orthogonal to #398

I built gopy from master-with-#398 (clear off by default) and ran a small probe package, each function in its own process (a fault shows up as a non-zero/signal exit):

function Go->Py conversion path result (clear off, no complex fix)
Add(i, j int) int C side (pybindgen) OK, 5
GetString() string C side, C.CString OK, hello
RealOf(complex128) float64 Go side, PyComplex_AsCComplex (read, no alloc) OK, 3.0
GetBytes() []byte C-side handle (go.Slice_byte) OK
bytes(GetBytes()) Go side, PyBytes_FromStringAndSize OK, b'\x01\x02\x03\x04'
MayFail(int) (int, error) value + raise OK / RuntimeError
Comp64Add(complex64) complex64 Go side, PyComplex_FromDoubles (alloc) SIGSEGV

So with the TLS clear already off, complex still crashes while everything else is fine; this is not #395 wearing a complex hat, it is a distinct bug. Confirmed crash signature matches your report:

Fatal Python error: Segmentation fault
Current thread (most recent call first):
  File ".../simple.py", line 107 in Comp64Add

2. Root cause: allocation inside gopy's GIL-released window

The generated wrapper releases the GIL for the entire body:

//export probe_Comp64Add
func probe_Comp64Add(i *C.PyObject, j *C.PyObject) *C.PyObject {
	_saved_thread := C.PyEval_SaveThread()        // GIL released, current tstate nulled on this thread
	defer C.PyEval_RestoreThread(_saved_thread)   // re-acquired on return
	return complex64GoToPy(probe.Comp64Add(complex64PyToGo(i), complex64PyToGo(j)))
	//     ^^^^^^^^^^^^^^^^ PyComplex_FromDoubles allocates here, GIL-released -> null tstate
}

On CPython 3.12+, object allocation dereferences the current thread state (allocator / GC bookkeeping), so allocating with the GIL released and tstate == NULL faults. This explains the whole table above:

  • int / float / bool / string are marshaled on the C side by pybindgen, where the GIL is still held -> fine.
  • []byte comes back as a handle (C side); only an explicit bytes(x) hits the Go-side PyBytes_FromStringAndSize: and that runs as its own exported call with the GIL held, so it's fine too.
  • PyComplex_AsCComplex (reading a complex arg) runs in the released window but only reads two doubles, no allocation -> fine.
  • PyComplex_FromDoubles (building the complex result) is the one Go-side allocation inside the released window -> crash.

3. As I see your fix is correct and verified

PyGILState_Ensure() re-acquires the GIL and binds a valid tstate for exactly the allocation, Release() drops it again. I rebuilt gopy with your two-function patch and:

Comp64Add(1+2j, 3+4j) -> (4+6j)     # exit 0
RealOf(3+4j)          -> 3.0        # exit 0
Add(2,3)              -> 5          # exit 0

It also matches gopy's own precedent: the Python-callback path in bind/symbols.go already wraps its C-API calls in PyGILState_Ensure / Release for the same reason. So this is consistent with the existing design, not a one-off.

4. Two a little stinky places, they're optional and may be for later

  1. The arg-read side is still GIL-released. complex64PyToGo / complex128PyToGo (PyComplex_AsCComplex) touch a PyObject while the GIL is released. It doesn't fault (no allocation), but strictly, touching any PyObject without the GIL is UB. So the complex round-trip as a whole runs Python-C-API-against-released-GIL; the allocation is just the part that visibly crashes.

  2. The tuple builder. bind/symbols.go has a Go-side PyTuple_New / gopy_build_* path. I could not get it onto a live wrapper path from ordinary signatures (plain multi-return like (int,int) is rejected: "second result value must be of type error"; (T, error) returns value+raise, no tuple), so I could not make it crash. But if any construct emits that builder inside the released window, it's the same bug.

The minimal, robust generalization would probably be to not PyEval_SaveThread for wrappers whose signature marshals *C.PyObject on the Go side (complex today), or to keep the GIL for the marshalling prologue/epilogue and only release around the pure-Go call. Your targeted PyGILState_Ensure is a perfectly good fix for the crash right now; the above is just the fuller shape of it if you ever want to close the window entirely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Complex-returning functions crash under CPython 3.12

2 participants