[lldb] Crash in _PyObject_Malloc when importing lldb in a different Python interpreter

[JDevlieghere]

I built lldb with LLDB_ENABLE_PYTHON_LIMITED_API against Python 3.14 from python.org. I then tried to import it from Python 3.12 installed with Homebrew. That causes a crash in _PyObject_Malloc during initialization:

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10)
    frame #0: 0x000000010879560c Python`_PyObject_Malloc + 52
Python`_PyObject_Malloc:
->  0x10879560c <+52>: ldr    x8, [x8, #0x10]
    0x108795610 <+56>: ldr    x19, [x8, #0x2aa8]
    0x108795614 <+60>: sub    x8, x1, #0x201
    0x108795618 <+64>: cmn    x8, #0x200
Target 0: (Python) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10)
  * frame #0: 0x000000010879560c Python`_PyObject_Malloc + 52
    frame #1: 0x0000000108803ce8 Python`PyUnicode_FromString + 548
    frame #2: 0x0000000116c16310 liblldb.22.0.0git.dylib`SWIG_Python_str_FromChar(c=<unavailable>) at LLDBWrapPython.cpp:877:10 [opt] [inlined]
    frame #3: 0x0000000116c16304 liblldb.22.0.0git.dylib`SWIG_This() at LLDBWrapPython.cpp:2473:24 [opt] [inlined]
    frame #4: 0x0000000116c162f8 liblldb.22.0.0git.dylib`::PyInit__lldb() at LLDBWrapPython.cpp:105239:3 [opt]

[llvmbot]

@llvm/issue-subscribers-lldb

Author: Jonas Devlieghere (JDevlieghere)

I built lldb with `LLDB_ENABLE_PYTHON_LIMITED_API` against Python 3.14 from python.org. I then tried to import it from Python 3.12 installed with Homebrew. That causes a crash in `_PyObject_Malloc` during initialization:

* thread #<!-- -->1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10)
    frame #<!-- -->0: 0x000000010879560c Python`_PyObject_Malloc + 52
Python`_PyObject_Malloc:
-&gt;  0x10879560c &lt;+52&gt;: ldr    x8, [x8, #<!-- -->0x10]
    0x108795610 &lt;+56&gt;: ldr    x19, [x8, #<!-- -->0x2aa8]
    0x108795614 &lt;+60&gt;: sub    x8, x1, #<!-- -->0x201
    0x108795618 &lt;+64&gt;: cmn    x8, #<!-- -->0x200
Target 0: (Python) stopped.
(lldb) bt
* thread #<!-- -->1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10)
  * frame #<!-- -->0: 0x000000010879560c Python`_PyObject_Malloc + 52
    frame #<!-- -->1: 0x0000000108803ce8 Python`PyUnicode_FromString + 548
    frame #<!-- -->2: 0x0000000116c16310 liblldb.22.0.0git.dylib`SWIG_Python_str_FromChar(c=&lt;unavailable&gt;) at LLDBWrapPython.cpp:877:10 [opt] [inlined]
    frame #<!-- -->3: 0x0000000116c16304 liblldb.22.0.0git.dylib`SWIG_This() at LLDBWrapPython.cpp:2473:24 [opt] [inlined]
    frame #<!-- -->4: 0x0000000116c162f8 liblldb.22.0.0git.dylib`::PyInit__lldb() at LLDBWrapPython.cpp:105239:3 [opt]

[JDevlieghere]

Here's the real issue:

Fatal Python error: _PyInterpreterState_GET: the function must be called with the GIL held, after Python initialization and before Python finalization, but the GIL is released (the current Python thread state is NULL)

A more informative backtrace from a debug Python build:

>>> import lldb
Fatal Python error: _PyInterpreterState_GET: the function must be called with the GIL held, after Python initialization and before Python finalization, but the GIL is released (the current Python thread state is NULL)
Python runtime state: unknown

Process 77995 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x0000000185dbf5b0 libsystem_kernel.dylib`__pthread_kill + 8
libsystem_kernel.dylib`__pthread_kill:
->  0x185dbf5b0 <+8>:  b.lo   0x185dbf5d0    ; <+40>
    0x185dbf5b4 <+12>: pacibsp
    0x185dbf5b8 <+16>: stp    x29, x30, [sp, #-0x10]!
    0x185dbf5bc <+20>: mov    x29, sp
Target 0: (Python) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x0000000185dbf5b0 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x0000000185df9888 libsystem_pthread.dylib`pthread_kill + 296
    frame #2: 0x0000000185cfe850 libsystem_c.dylib`abort + 124
    frame #3: 0x0000000106ae685c Python`fatal_error_exit(status=<unavailable>) at pylifecycle.c:3159:9 [opt]
    frame #4: 0x0000000106ae659c Python`fatal_error(fd=<unavailable>, header=<unavailable>, prefix=<unavailable>, msg=<unavailable>, status=-1) at pylifecycle.c:0 [opt]
    frame #5: 0x0000000106ae5358 Python`_Py_FatalErrorFunc(func=<unavailable>, msg=<unavailable>) at pylifecycle.c:3391:5 [opt]
    frame #6: 0x0000000106986368 Python`_Py_EnsureFuncTstateNotNULL(func=<unavailable>, tstate=<unavailable>) at pycore_pystate.h:189:9 [opt] [inlined]
    frame #7: 0x0000000106986354 Python`_PyInterpreterState_GET at pycore_pystate.h:212:5 [opt] [inlined]
    frame #8: 0x0000000106986354 Python`get_state at obmalloc.c:1472:34 [opt] [inlined]
    frame #9: 0x0000000106986354 Python`_PyObject_Malloc(ctx=0x0000000000000000, nbytes=<unavailable>) at obmalloc.c:2232:22 [opt]
    frame #10: 0x0000000106986da0 Python`_PyMem_DebugRawAlloc(use_calloc=0, ctx=0x0000000106dbc140, nbytes=45) at obmalloc.c:2792:24 [opt] [inlined]
    frame #11: 0x0000000106986d8c Python`_PyMem_DebugRawMalloc(ctx=0x0000000106dbc140, nbytes=45) at obmalloc.c:2825:12 [opt] [inlined]
    frame #12: 0x0000000106986d8c Python`_PyMem_DebugMalloc(ctx=0x0000000106dbc140, nbytes=45) at obmalloc.c:2982:12 [opt]
    frame #13: 0x00000001069c0890 Python`PyUnicode_New(size=4, maxchar=127) at unicodeobject.c:1398:24 [opt]
    frame #14: 0x00000001069c84d8 Python`unicode_decode_utf8(s=<unavailable>, size=4, error_handler=_Py_ERROR_UNKNOWN, errors=0x0000000000000000, consumed=0x0000000000000000) at unicodeobject.c:4995:19 [opt]
    frame #15: 0x000000011540346c liblldb.22.0.0git.dylib`SWIG_Python_str_FromChar(c=<unavailable>) at LLDBWrapPython.cpp:877:10 [opt] [inlined]
    frame #16: 0x0000000115403460 liblldb.22.0.0git.dylib`SWIG_This() at LLDBWrapPython.cpp:2473:24 [opt] [inlined]
    frame #17: 0x0000000115403454 liblldb.22.0.0git.dylib`::PyInit__lldb() at LLDBWrapPython.cpp:105776:3 [opt]

[JDevlieghere]

For those at home wondering why I didn't get that error message, that's because it's only there for debug builds:

frame #9: 0x0000000106986354 Python`_PyObject_Malloc(ctx=0x0000000000000000, nbytes=<unavailable>) at obmalloc.c:2232:22 [opt]
   2229 void *
   2230 _PyObject_Malloc(void *ctx, size_t nbytes)
   2231 {
-> 2232     OMState *state = get_state();
   2233     void* ptr = pymalloc_alloc(state, ctx, nbytes);
   2234     if (LIKELY(ptr != NULL)) {
   2235         return ptr;
(lldb)
frame #8: 0x0000000106986354 Python`get_state at obmalloc.c:1472:34 [opt] [inlined]
   1469 static inline OMState *
   1470 get_state(void)
   1471 {
-> 1472     PyInterpreterState *interp = _PyInterpreterState_GET();
   1473     assert(interp->obmalloc != NULL); // otherwise not initialized or freed
   1474     return interp->obmalloc;
   1475 }
(lldb)
frame #7: 0x0000000106986354 Python`_PyInterpreterState_GET at pycore_pystate.h:212:5 [opt] [inlined]
   209  static inline PyInterpreterState* _PyInterpreterState_GET(void) {
   210      PyThreadState *tstate = _PyThreadState_GET();
   211  #ifdef Py_DEBUG
-> 212      _Py_EnsureTstateNotNULL(tstate);
   213  #endif
   214      return tstate->interp;
   215  }

[JDevlieghere]

The error seems legitimate: in the working case, tstate is a valid pointer. In the broken case, tstate is NULL. In neither case there's a call to PyGILState_Ensure so it's unclear to me why the PyThreadState is set in one case but not the other.