[WIP] Re-enable Move Subinterpreter test for free-threaded Python 3.14

[rwgk]

Continuation of work done initially under PRs #5934 and #5933

The write-up below is a Cursor-generated.

In an earlier draft of the below, Cursor asked "Could this be a bug in cpython?" Cursor was offering to generate a minimal C-only reproducer. We can come back to that if we want to.

__

Investigation: Free-Threaded Python 3.14 Hang in "Move Subinterpreter" Test

To: @b-pass
From: Investigation with Cursor AI assistance
Date: December 20, 2025
Re: PR #5933 - Root cause of Py_EndInterpreter() hang on free-threaded Python 3.14.2

---

Executive Summary

We've isolated the exact cause of the "Move Subinterpreter" test hang on free-threaded Python 3.14.2. The issue is not in gil_safe_call_once_and_store, the internals capsule destructors, or any cleanup code.

The root cause is a single line in py::subinterpreter::create():

PyThreadState_Swap(prev_tstate);  // subinterpreter.h:124

When this is called after PyThreadState_DeleteCurrent() during subinterpreter creation, it leaves the system in a state where later calling Py_EndInterpreter() from a different thread causes a deadlock.

---

Background

The Failing Test

The "Move Subinterpreter" test (test_subinterpreter.cpp:94-119) does:

1. Creates a subinterpreter on the main thread
2. Activates it, imports modules, deactivates
3. Spawns a worker thread that:
   • Activates the same subinterpreter
   • Imports a module
   • Destroys the subinterpreter (sub.reset())
4. Joins the worker thread

This test passes on:

• Default (non-free-threaded) Python 3.14.2 ✅
• Free-threaded Python 3.14.0t ✅ (sporadically fails on macOS)

This test hangs on:

• Free-threaded Python 3.14.1t ❌
• Free-threaded Python 3.14.2t ❌

Prior Findings

A pure C reproducer (move_subinterpreter_redux.c) that mimics the same pattern using only CPython C API passes on both 3.14.0t and 3.14.1t. This indicated the issue was in pybind11's internals, not CPython itself.

---

Investigation Methodology

We systematically created minimal test cases, each removing one aspect of pybind11's subinterpreter handling, until we found what causes the hang.

Test Matrix

Test Case	Key Difference	Result
move_subinterpreter_redux.c	Pure C, no pybind11	✅ Pass
debug_pure_c_with_pb11_main.cpp	pybind11 main interpreter, pure C subinterpreter	✅ Pass
debug_no_swap_back.cpp	Like above, but don't swap back after creation	✅ Pass
debug_with_swap_back.cpp	Add PyThreadState_Swap(prev_tstate) after creation	❌ Hang
debug_no_get_internals.cpp	Skip get_internals() call	❌ Hang
debug_no_num_interp.cpp	Skip get_num_interpreters_seen()++	❌ Hang
Any test using py::subinterpreter::create()	Uses full pybind11 code	❌ Hang

The critical finding: The only difference between passing and failing tests is PyThreadState_Swap(prev_tstate) after PyThreadState_DeleteCurrent().

---

Root Cause Analysis

The Problematic Code Path

In subinterpreter.h, the create() function does:

static subinterpreter create(PyInterpreterConfig const &cfg) {
    error_scope err_scope;
    subinterpreter result;
    {
        // Activate main interpreter to hold its GIL
        subinterpreter_scoped_activate main_guard(main());

        auto *prev_tstate = PyThreadState_Get();  // Save main's tstate

        // Create subinterpreter - now creation_tstate_ is current
        Py_NewInterpreterFromConfig(&result.creation_tstate_, &cfg);

        result.istate_ = result.creation_tstate_->interp;
        detail::get_num_interpreters_seen() += 1;
        detail::get_internals();  // Initialize internals for subinterpreter

        // Clean up creation tstate (3.14+ style)
        PyThreadState_Clear(result.creation_tstate_);
        PyThreadState_DeleteCurrent();  // Thread state is now NULL

        // Switch back to main interpreter
        PyThreadState_Swap(prev_tstate);  // <-- THIS CAUSES THE HANG
    }
    return result;
}

Why This Causes a Hang

On free-threaded Python 3.14.2, the sequence:

1. PyThreadState_DeleteCurrent() - deletes the subinterpreter's creation tstate
2. PyThreadState_Swap(prev_tstate) - swaps to main interpreter's tstate

...appears to leave some internal state inconsistent. Specifically, when later:

3. A different thread creates a new tstate for the subinterpreter
4. That thread calls Py_EndInterpreter()

...the Py_EndInterpreter() call deadlocks.

The Pure C Pattern That Works

The working C reproducer does this instead:

static void create_subinterpreter(void) {
    PyThreadState *creation_tstate = NULL;
    Py_NewInterpreterFromConfig(&creation_tstate, &cfg);
    sub_interp = creation_tstate->interp;

    // Clean up creation tstate
    PyThreadState_Clear(creation_tstate);
    PyThreadState_DeleteCurrent();
    // NOTE: Does NOT call PyThreadState_Swap() here
    // Thread state is left as NULL
}

The key difference: no PyThreadState_Swap() after PyThreadState_DeleteCurrent().

---

Minimal Reproducer

Here's the minimal code that demonstrates the issue:

// HANGS on free-threaded Python 3.14.2
#include <pybind11/embed.h>
#include <thread>

namespace py = pybind11;

static PyInterpreterState *sub_interp = nullptr;

int main() {
    py::scoped_interpreter guard{};
    
    // Create subinterpreter
    {
        PyThreadState *prev_tstate = PyThreadState_Get();  // Save main tstate
        
        PyInterpreterConfig cfg = {0};
        cfg.allow_threads = 1;
        cfg.check_multi_interp_extensions = 1;
        cfg.gil = PyInterpreterConfig_OWN_GIL;

        PyThreadState *creation_tstate = nullptr;
        Py_NewInterpreterFromConfig(&creation_tstate, &cfg);
        sub_interp = creation_tstate->interp;
        
        PyThreadState_Clear(creation_tstate);
        PyThreadState_DeleteCurrent();
        
        PyThreadState_Swap(prev_tstate);  // <-- REMOVE THIS LINE TO FIX
    }

    // Use subinterpreter on main thread
    {
        PyThreadState* tstate = PyThreadState_New(sub_interp);
        PyThreadState* old = PyThreadState_Swap(tstate);
        PyRun_SimpleString("import datetime");
        PyThreadState_Clear(tstate);
        PyThreadState_DeleteCurrent();
        PyThreadState_Swap(old);
    }

    // Destroy from worker thread - THIS HANGS
    std::thread([&]() {
        PyThreadState* tstate = PyThreadState_New(sub_interp);
        PyThreadState_Swap(tstate);
        PyThreadState_Clear(tstate);
        PyThreadState_DeleteCurrent();
        
        PyThreadState *destroy_tstate = PyThreadState_New(sub_interp);
        PyThreadState_Swap(destroy_tstate);
        Py_EndInterpreter(destroy_tstate);  // <-- HANGS HERE
    }).join();
    
    return 0;
}

To fix: Remove the PyThreadState_Swap(prev_tstate) line after creation.

---

Attempted Fixes

We tried several quick fixes, none of which fully worked:

Attempt 1: Skip PyThreadState_Swap(prev_tstate) on free-threaded Python

#if defined(Py_GIL_DISABLED) && PY_VERSION_HEX >= 0x030D0000
    (void) prev_tstate;  // Skip the swap
#else
    PyThreadState_Swap(prev_tstate);
#endif

Result: No longer hangs, but crashes with:

Fatal Python error: PyGILState_Release: auto-releasing thread-state, 
but no thread-state for this thread

The subinterpreter_scoped_activate main_guard(main()) destructor tries to call PyGILState_Release() but there's no current thread state after we skipped the swap.

Attempt 2: Create a fresh thread state for main instead of reusing prev_tstate

#if defined(Py_GIL_DISABLED)
    (void) prev_tstate;
    PyThreadState *fresh_main_tstate = PyThreadState_New(PyInterpreterState_Main());
    PyThreadState_Swap(fresh_main_tstate);
#else
    PyThreadState_Swap(prev_tstate);
#endif

Result: Segfault. The fresh tstate doesn't properly integrate with the saved state in main_guard.

Conclusion

The fix is not trivial because the subinterpreter_scoped_activate main_guard(main()) at line 85 saves state (gil_state_ and/or old_tstate_) that becomes stale after PyThreadState_DeleteCurrent(). A proper fix likely requires restructuring how create() manages the main interpreter's GIL, possibly:

1. Not using subinterpreter_scoped_activate inside create() on free-threaded Python
2. Using a different pattern that doesn't rely on swapping back to a saved thread state
3. Some other approach?

---

Files Referenced

• include/pybind11/subinterpreter.h - lines 79-127 (create() function)
• tests/test_with_catch/test_subinterpreter.cpp - lines 94-119 ("Move Subinterpreter" test)
• Debug test files:
  • debug_no_swap_back.cpp - passes
  • debug_with_swap_back.cpp - hangs

---

Appendix: Debug Output

Passing Test (no swap back)

[DEBUG] Starting scoped_interpreter...
[DEBUG] scoped_interpreter created
[DEBUG] Creating subinterpreter...
[DEBUG] Subinterpreter created, id=1
[DEBUG] Main thread: activating...
[DEBUG] Main thread: imported
[DEBUG] Spawning worker thread...
[DEBUG] Worker thread: started
[DEBUG] Worker thread: activated
[DEBUG] Worker thread: about to Py_EndInterpreter...
[DEBUG] Worker thread: Py_EndInterpreter done!
[DEBUG] Thread joined, test complete!
Exit code: 0

Failing Test (with swap back)

[DEBUG] Starting scoped_interpreter...
[DEBUG] scoped_interpreter created
[DEBUG] Creating subinterpreter...
[DEBUG] Called PyThreadState_Swap(prev_tstate) after DeleteCurrent
[DEBUG] Subinterpreter created, id=1
[DEBUG] Main thread: activating...
[DEBUG] Main thread: imported
[DEBUG] Spawning worker thread...
[DEBUG] Worker thread: started
[DEBUG] Worker thread: activated
[DEBUG] Worker thread: about to Py_EndInterpreter...
Exit code: 124  (timeout - hung in Py_EndInterpreter)

---

This investigation was conducted using local builds of Python 3.14.2 (default and free-threaded) from commit df793163d58.

[rwgk]

@b-pass, it'd be great if we could connect directly. Could you please email me under the email address you see with git log | grep rwgkio?

[rwgk]

@b-pass This is now a real PR. Could you take it from here?