Unclosed interpreter causes segfault after fork() in child process.

[aurelwildfellner]

Bug report

Bug description:

With concurrent.interpreters now available in 3.14 it's possible to trigger a probably already known issue with cleaning up interpreters after a fork.

This minimal example creates an interpreter (on the same thread), doesn't close it and then forks. The child process segfaults.

#!/usr/bin/python3.14

import os
from concurrent import interpreters

GLOBAL_INTERP = None

def create_subinter(do_close: bool) -> None:
    global GLOBAL_INTERP
    interp = interpreters.create()
    GLOBAL_INTERP = interp
    interp.exec("""
x = 42 * 23
""")
    if do_close:
        interp.close()


def main():
    global GLOBAL_INTERP

    create_subinter(do_close=False)
    print("Before fork")
    child_pid = os.fork()

    if child_pid == 0:
        print("in child")
    else:
        print("in parent")

if __name__ == "__main__":
    main()

There seems to be a more general issue with the implementation of PyOS_AfterFork_Child() or PyInterpreterState_Clear() specifically:

cpython/Python/pystate.c

Lines 992 to 994 in a15aeec

	// XXX Won't this fail since PyInterpreterState_Clear() requires
	// the "current" tstate to be set?
	PyInterpreterState_Clear(interp); // XXX must activate?

As the comment points out, the PyInterpreterState_Clear() won't be able to clear any interpreters, as the tstate is set to NULL before calling the function. Which finally results in a Py_DECREF on NULL.

I guess the expected behavior is that manually closing interpreters before a fork is not required and a working clear in the fork. While the combination of subinterpreters and fork() seems a bit obscure, we already ran into this issue in a CI setup.

It also seems that there is a lack of test coverage with the mentioned implementation in pystate.c. If appreciated, I could also follow up with a pull request adding more tests.

Full backtrace:

#0  _Py_DECREF_DecRefTotal () at Objects/object.c:287
#1  0x0000555555816177 in Py_DECREF (filename=filename@entry=0x555555908674 "Python/import.c", lineno=lineno@entry=4099, op=0x7ffff6ea0230) at ./Include/refcount.h:400
#2  0x000055555581d0af in _PyImport_ClearCore (interp=interp@entry=0x7ffff7a59020) at Python/import.c:4099
#3  0x000055555584a008 in PyInterpreterState_Clear (interp=interp@entry=0x7ffff7a59020) at Python/pystate.c:986
#4  0x000055555584a7a3 in _PyInterpreterState_DeleteExceptMain (runtime=runtime@entry=0x555555b809c0 <_PyRuntime>) at Python/pystate.c:1085
#5  0x000055555588f202 in PyOS_AfterFork_Child () at ./Modules/posixmodule.c:741
#6  0x000055555588f49a in os_fork_impl (module=<optimized out>) at ./Modules/posixmodule.c:8159
#7  0x000055555588f516 in os_fork (module=<optimized out>, _unused_ignored=_unused_ignored@entry=0x0) at ./Modules/clinic/posixmodule.c.h:4394
#8  0x00005555556ce81a in cfunction_vectorcall_NOARGS (func=0x7ffff711d3d0, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:508
#9  0x0000555555676df0 in _PyObject_VectorcallTstate (tstate=0x555555bd1938 <_PyRuntime+331640>, callable=callable@entry=0x7ffff711d3d0, args=args@entry=0x7fffffffd628,
    nargsf=9223372036854775808, kwnames=kwnames@entry=0x0) at ./Include/internal/pycore_call.h:169
#10 0x0000555555676ecc in PyObject_Vectorcall (callable=callable@entry=0x7ffff711d3d0, args=args@entry=0x7fffffffd628, nargsf=<optimized out>, kwnames=kwnames@entry=0x0)
    at Objects/call.c:327
#11 0x00005555557b4ca6 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ffff7fb2080, throwflag=0) at Python/generated_cases.c.h:1619
#12 0x00005555557d3d57 in _PyEval_EvalFrame (tstate=tstate@entry=0x555555bd1938 <_PyRuntime+331640>, frame=frame@entry=0x7ffff7fb2020, throwflag=throwflag@entry=0)
    at ./Include/internal/pycore_ceval.h:119
#13 0x00005555557d3f28 in _PyEval_Vector (tstate=tstate@entry=0x555555bd1938 <_PyRuntime+331640>, func=func@entry=0x7ffff716ab10, locals=locals@entry=0x7ffff71782f0,
    args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:1965
#14 0x00005555557d4024 in PyEval_EvalCode (co=co@entry=0x7ffff7bd23b0, globals=globals@entry=0x7ffff71782f0, locals=locals@entry=0x7ffff71782f0) at Python/ceval.c:857
#15 0x000055555584b13f in run_eval_code_obj (tstate=tstate@entry=0x555555bd1938 <_PyRuntime+331640>, co=co@entry=0x7ffff7bd23b0, globals=globals@entry=0x7ffff71782f0,
    locals=locals@entry=0x7ffff71782f0) at Python/pythonrun.c:1365
#16 0x000055555584b2dc in run_mod (mod=mod@entry=0x555555d7f060, filename=filename@entry=0x7ffff7198400, globals=globals@entry=0x7ffff71782f0, locals=locals@entry=0x7ffff71782f0,
    flags=flags@entry=0x7fffffffda28, arena=arena@entry=0x7ffff71e4a00, interactive_src=0x0, generate_new_source=0) at Python/pythonrun.c:1459
--Type <RET> for more, q to quit, c to continue without paging--                          
#17 0x000055555584bb5c in pyrun_file (fp=fp@entry=0x555555c49a60, filename=filename@entry=0x7ffff7198400, start=start@entry=257, globals=globals@entry=0x7ffff71782f0,
    locals=locals@entry=0x7ffff71782f0, closeit=closeit@entry=1, flags=0x7fffffffda28) at Python/pythonrun.c:1293
#18 0x000055555584d3f0 in _PyRun_SimpleFileObject (fp=fp@entry=0x555555c49a60, filename=filename@entry=0x7ffff7198400, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffda28)
    at Python/pythonrun.c:521
#19 0x000055555584d5e6 in _PyRun_AnyFileObject (fp=fp@entry=0x555555c49a60, filename=filename@entry=0x7ffff7198400, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffda28)
    at Python/pythonrun.c:81
#20 0x0000555555876b67 in pymain_run_file_obj (program_name=program_name@entry=0x7ffff71b2640, filename=filename@entry=0x7ffff7198400, skip_source_first_line=0) at Modules/main.c:410
#21 0x0000555555876c8f in pymain_run_file (config=config@entry=0x555555b9c9f0 <_PyRuntime+114736>) at Modules/main.c:429
#22 0x00005555558777ca in pymain_run_python (exitcode=exitcode@entry=0x7fffffffdba4) at Modules/main.c:694
#23 0x0000555555877a7f in Py_RunMain () at Modules/main.c:775
#24 0x0000555555877af6 in pymain_main (args=args@entry=0x7fffffffdc00) at Modules/main.c:805
#25 0x0000555555877bc4 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:829
#26 0x00005555555de166 in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15

CPython versions tested on:

3.14

Operating systems tested on:

Linux

[efimov-mikhail]

Thanks for the report.
Unfortunately, I can't get a crash with this code snippet.

-> % ./python subinterpreteres_crash_139759.py
Before fork
in parent
<sys>:0: RuntimeWarning: remaining subinterpreters; destroy them with _interpreters.destroy()

There is only RuntimeWarning present.

Could you please provide more information about your OS and environment?

[aurelwildfellner]

Unfortunately, I can't get a crash with this code snippet.

This may be due to SIGSEV signal not propagating with the fork. You can see from the output that "in child" is not printed which indicates that the process has crashed.

You can also try to run it in gdb with the right follow-fork-mode:

gdb -ex "set follow-fork-mode child" --args ./python subinterpreteres_crash_139759.py

You should see a SIGSEV and get an according backtrace (running with a debug build).

Could you please provide more information about your OS and environment?

Tested with a debug build of the 3.14 branch.

gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0

Ubuntu 24.04.2 LTS (all dependencies installed from system packages).

[efimov-mikhail]

gdb -ex "set follow-fork-mode child" --args ./python subinterpreteres_crash_139759.py

Confirmed on main branch, gcc version 12.2.0 (Debian 12.2.0-14+deb12u1). Thanks!

[ZeroIntensity]

Would you like to submit a PR?

[efimov-mikhail]

Would you like to submit a PR?

You're asking me or @aurelwildfellner ?
I can try to dig into this.

[ZeroIntensity]

I was asking either of you.