Building Python 3.12 on ppc32 fail with: Fatal Python error: _PyInterpreterState_GET: the function must be called with the GIL held

[thesamesam]

Crash report

Python 3.12 fails when building itself while calling ./python -E -S -m sysconfig --generate-posix-var. I first noticed this when building Gentoo's Python 3.12 package, but can reproduce with a git checkout of 3.12.0b3 using (export CFLAGS_NODIST="-O2" ; ./configure --enable-shared && make -j$(nproc) CPPFLAGS= CFLAGS= LDFLAGS= ).

The --enable-shared is required, as the crash doesn't seem to happen without it.

Error messages

Enter any relevant error message caused by the crash, including a core dump if there is one.

 # LD_LIBRARY_PATH=/root/cpython ./python -E -S -m sysconfig --generate-posix-var
Segmentation fault (core dumped)

# LD_LIBRARY_PATH=/root/cpython gdb --args ./python -E -S -m sysconfig --generate-posix-var
[...]
Program received signal SIGSEGV, Segmentation fault.
sys_audit_tstate (ts=0x312e0000, event=0xf7d5dfb4 "cpython.PyInterpreterState_New", argFormat=0x0, vargs=vargs@entry=0xffffda60) at ./Python/sysmodule.c:196
196         PyInterpreterState *is = ts->interp;
(gdb) bt
#0  sys_audit_tstate (ts=0x312e0000, event=0xf7d5dfb4 "cpython.PyInterpreterState_New", argFormat=0x0, vargs=vargs@entry=0xffffda60) at ./Python/sysmodule.c:196
#1  0xf7c3e9b0 in _PySys_Audit (tstate=tstate@entry=0x312e0000, event=event@entry=0xf7d5dfb4 "cpython.PyInterpreterState_New", argFormat=argFormat@entry=0x0) at ./Python/sysmodule.c:312
#2  0xf7c26840 in PyInterpreterState_New () at Python/pystate.c:709
#3  0xf7c23ab8 in pycore_create_interpreter (runtime=0xf7f6ff08 <_PyRuntime>, tstate_p=<synthetic pointer>, src_config=0xffffdaf8) at Python/pylifecycle.c:628
#4  pyinit_config (runtime=0xf7f6ff08 <_PyRuntime>, config=0xffffdaf8, tstate_p=0xffffdc58) at Python/pylifecycle.c:891
#5  pyinit_core (src_config=src_config@entry=0xffffdcb8, tstate_p=tstate_p@entry=0xffffdc58, runtime=0xf7f6ff08 <_PyRuntime>) at Python/pylifecycle.c:1060
#6  0xf7c23e04 in Py_InitializeFromConfig (config=0xffffdcb8) at Python/pylifecycle.c:1256
#7  Py_InitializeFromConfig (config=config@entry=0xffffdcb8) at Python/pylifecycle.c:1241
#8  0xf7c584c8 in pymain_init (args=args@entry=0xffffde5c) at Modules/main.c:67
#9  0xf7c599a8 in pymain_main (args=args@entry=0xffffde5c) at Modules/main.c:710
#10 0xf7c59ad0 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:743
#11 0x00400524 in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15
(gdb)

When building with --with-pydebug, I get:

LD_LIBRARY_PATH=/root/cpython ./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
        echo "generate-posix-vars failed" ; \
        rm -f ./pybuilddir.txt ; \
        exit 1 ; \
fi
Fatal Python error: _PyInterpreterState_GET: the function must be called with the GIL held, after Python initialization and before Python finalization, but the GIL is released (the current Python thread state is NULL)
Python runtime state: preinitialized

Current thread 0xf793c020 (most recent call first):
  <no Python frame>
make: *** [Makefile:949: pybuilddir.txt] Error 134

With UBSAN, I get:

Include/internal/pycore_pystate.h:118:18: runtime error: member access within null pointer of type 'struct PyThreadState'
    #0 0xf6cfa94c in _PyInterpreterState_GET Include/internal/pycore_pystate.h:118
    #1 0xf6cfa94c in PyDict_New Objects/dictobject.c:849
    #2 0xf6ee16ac in init_interned_dict Objects/unicodeobject.c:248
    #3 0xf6ee16ac in _PyUnicode_InitGlobalObjects Objects/unicodeobject.c:14659
    #4 0xf710a368 in pycore_init_global_objects Python/pylifecycle.c:678
    #5 0xf710a368 in pycore_interp_init Python/pylifecycle.c:826
    #6 0xf7110014 in pyinit_config Python/pylifecycle.c:897
    #7 0xf7110014 in pyinit_core Python/pylifecycle.c:1060
    #8 0xf7110388 in Py_InitializeFromConfig Python/pylifecycle.c:1256
    #9 0xf7110388 in Py_InitializeFromConfig Python/pylifecycle.c:1241
    #10 0xf719e23c in pymain_init Modules/main.c:67
    #11 0xf71a0f38 in pymain_main Modules/main.c:710
    #12 0xf71a1060 in Py_BytesMain Modules/main.c:743
    #13 0x8b0540 in main Programs/python.c:15
    #14 0x649bf0  (/usr/lib/libc.so.6+0x29bf0)
    #15 0x649e14 in __libc_start_main (/usr/lib/libc.so.6+0x29e14)

Your environment

• CPython versions tested on: 3.12.0b3 fails, 3.11.4 is okay
• Operating system and architecture: Gentoo Linux, ppc32
• Compiler: GCC 13.1.1 20230527 fails, but GCC 12.3.1 20230623 and GCC 11.3.1 20221209 work.
• Downstream report: https://bugs.gentoo.org/909544

[mgorny]

I've been able to bisect it to f8abfa3.

[thesamesam]

cc @ericsnowcurrently

[thesamesam]

I wonder if I made a mistake earlier with the initial UBSAN output (because I had to hack it in for reasons unrelated to this bug, this is it done properly - wonder if something like the instrumentation wasn't in all the objects):

LD_LIBRARY_PATH=/root/cpython ./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
        echo "generate-posix-vars failed" ; \
        rm -f ./pybuilddir.txt ; \
        exit 1 ; \
fi
Python/sysmodule.c:196:25: runtime error: member access within misaligned address 0x06969183 for type 'struct PyThreadState', which requires 8 byte alignment
0x06969183: note: pointer points here
<memory cannot be printed>
    #0 0xf76dbf44 in sys_audit_tstate Python/sysmodule.c:196
    #1 0xf76de824 in _PySys_Audit Python/sysmodule.c:312
    #2 0xf76bf640 in PyInterpreterState_New Python/pystate.c:709
    #3 0xf76bc68c in pycore_create_interpreter Python/pylifecycle.c:628
    #4 0xf76bc68c in pyinit_config Python/pylifecycle.c:891
    #5 0xf76bc68c in pyinit_core Python/pylifecycle.c:1060
    #6 0xf76bca20 in Py_InitializeFromConfig Python/pylifecycle.c:1256
    #7 0xf76bca20 in Py_InitializeFromConfig Python/pylifecycle.c:1241
    #8 0xf76ff45c in pymain_init Modules/main.c:67
    #9 0xf7700aa8 in pymain_main Modules/main.c:710
    #10 0xf7700bd8 in Py_BytesMain Modules/main.c:743
    #11 0x630540 in main Programs/python.c:15
    #12 0x339bf0  (/usr/lib/libc.so.6+0x29bf0)
    #13 0x339e14 in __libc_start_main (/usr/lib/libc.so.6+0x29e14)

[vstinner]

I fail to reproduce the issue on Fedora 38 with GCC 13.1.1. I built the 3.12 branch of the Git repository and it works fine:

$ git clean -fdx
$ echo $MAKEFLAGS 
-j14
$ ./configure --enable-shared && make 

Operating system and architecture: Gentoo Linux, ppc32

Oh. It seems like Thread-Local Variable is not well implemented on this old architecture.

Python now has a _Py_thread_local PyThreadState *_Py_tss_tstate = NULL; variable where _Py_thread_local macro is defined as __thread on GCC.

[thesamesam]

This looks to be a binutils bfd linker bug, see https://bugs.gentoo.org/909544#c12.

Reported it over there at https://sourceware.org/bugzilla/show_bug.cgi?id=30697. Thanks.

[vstinner]

This looks to be a binutils bfd linker bug, see https://bugs.gentoo.org/909544#c12. Reported it over there at https://sourceware.org/bugzilla/show_bug.cgi?id=30697. Thanks.

Great!