You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
On one of my recent PRs, gh-118157, I was getting consistent failures on CI in test_interpreters. Different tests would fail, but the error was always the same:
(expand)
0:08:24 load avg: 4.18 Re-running 1 failed tests in verbose mode in subprocesses
0:08:24 load avg: 4.18 Run 1 test in parallel using 1 worker process (timeout: 10 min, worker timeout: 15 min)
0:08:26 load avg: 3.93 [1/1/1] test_interpreters failed (6 failures)
Re-running test_interpreters in verbose mode (matching: test_display_preserved_exception, test_gh_109793, test_sys_path_0, test_sys_path_0, test_sys_path_0, test_sys_path_0)
test_display_preserved_exception (test.test_interpreters.test_api.TestInterpreterExec.test_display_preserved_exception) ... FAIL
test_gh_109793 (test.test_interpreters.test_lifecycle.FinalizationTests.test_gh_109793) ... FAIL
test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) ...
### start subtest debug ###
# python3 script.py failed:
--- stdout ---
{
"main": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4",
"sub": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4"
}
------
--- stderr ---
Exception ignored in: <function Interpreter.__del__ at 0x2000119aed0>
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
TypeError: catching classes that do not inherit from BaseException is not allowed
Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
Python runtime state: finalizing (tstate=0x00005639aed8ea10)
------
### end subtest debug ###
test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 script.py] ... FAIL
### start subtest debug ###
# python3 -m script failed:
--- stdout ---
{
"main": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4",
"sub": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4"
}
------
--- stderr ---
Exception ignored in: <function Interpreter.__del__ at 0x2000117c0d0>
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
TypeError: catching classes that do not inherit from BaseException is not allowed
Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
Python runtime state: finalizing (tstate=0x0000564591b58a10)
------
### end subtest debug ###
test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m script] ... FAIL
### start subtest debug ###
# python3 -m pkg failed:
--- stdout ---
{
"main": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4",
"sub": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4"
}
------
--- stderr ---
Exception ignored in: <function Interpreter.__del__ at 0x2000118c0d0>
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
TypeError: catching classes that do not inherit from BaseException is not allowed
Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
Python runtime state: finalizing (tstate=0x000055aaa11eaa10)
------
### end subtest debug ###
test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m pkg] ... FAIL
### start subtest debug ###
# python3 -m pkg.script failed:
--- stdout ---
{
"main": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4",
"sub": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4"
}
------
--- stderr ---
Exception ignored in: <function Interpreter.__del__ at 0x2000118c0d0>
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
TypeError: catching classes that do not inherit from BaseException is not allowed
Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
Python runtime state: finalizing (tstate=0x00005637ca7cfa10)
------
### end subtest debug ###
test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m pkg.script] ... FAIL
======================================================================
FAIL: test_display_preserved_exception (test.test_interpreters.test_api.TestInterpreterExec.test_display_preserved_exception)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/__init__.py", line 2603, in wrapper
return func(*args, **kwargs)
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_api.py", line 764, in test_display_preserved_exception
self.assertEqual(stderr, dedent(f"""\
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
Traceback (most recent call last):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<18 lines>...
RuntimeError: uh-oh!
^^^^^^^^^^^^^^^^^^^^
"""))
^^^^^
AssertionError: 'Trac[778 chars]oh!\nException ignored in: <function Interpret[519 chars]\n\n' != 'Trac[778 chars]oh!\n'
Traceback (most recent call last):
File "/tmp/test_python_tk7ha1xh/tmpkqxiakqz/script.py", line 9, in <module>
interp.exec(script)
~~~~~~~~~~~^^^^^^^^
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 227, in exec
raise ExecutionFailed(excinfo)
test.support.interpreters.ExecutionFailed: RuntimeError: uh-oh!
Uncaught in the interpreter:
Traceback (most recent call last):
File "/tmp/test_python_tk7ha1xh/tmpkqxiakqz/script.py", line 6, in script
spam.eggs()
~~~~~~~~~^^
File "/tmp/test_python_tk7ha1xh/tmpkqxiakqz/spam.py", line 6, in eggs
ham()
~~~^^
File "/tmp/test_python_tk7ha1xh/tmpkqxiakqz/spam.py", line 3, in ham
raise RuntimeError('uh-oh!')
RuntimeError: uh-oh!
- Exception ignored in: <function Interpreter.__del__ at 0x2000119aed0>
- Traceback (most recent call last):
- File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
- File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
- TypeError: catching classes that do not inherit from BaseException is not allowed
- Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
- Python runtime state: finalizing (tstate=0x000055b06f0a1a10)
-
======================================================================
FAIL: test_gh_10[979](https://github.com/python/cpython/actions/runs/8978599353/job/24660038142#step:19:980)3 (test.test_interpreters.test_lifecycle.FinalizationTests.test_gh_109793)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 185, in test_gh_109793
self.assertEqual(proc.returncode, 1)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: -6 != 1
======================================================================
FAIL: test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 script.py]
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 29, in subTest
yield ctx
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 157, in test_sys_path_0
out = self.run_python(argv, cwd=cwd)
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 114, in run_python
self.assertEqual(proc.returncode, 0)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: -6 != 0
======================================================================
FAIL: test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m script]
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 29, in subTest
yield ctx
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 157, in test_sys_path_0
out = self.run_python(argv, cwd=cwd)
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 114, in run_python
self.assertEqual(proc.returncode, 0)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: -6 != 0
======================================================================
FAIL: test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m pkg]
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 29, in subTest
yield ctx
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 157, in test_sys_path_0
out = self.run_python(argv, cwd=cwd)
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 114, in run_python
self.assertEqual(proc.returncode, 0)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: -6 != 0
======================================================================
FAIL: test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m pkg.script]
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 29, in subTest
yield ctx
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 157, in test_sys_path_0
out = self.run_python(argv, cwd=cwd)
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 114, in run_python
self.assertEqual(proc.returncode, 0)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: -6 != 0
----------------------------------------------------------------------
Ran 3 tests in 1.720s
FAILED (failures=6)
test test_interpreters failed
1 test failed again:
test_interpreters
The error looks like this:
Exception ignored in: <function Interpreter.__del__ at 0x2000117c0d0>
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
TypeError: catching classes that do not inherit from BaseException is not allowed
Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
Python runtime state: finalizing (tstate=0x0000564591b58a10)
I'm fairly sure that means one of the 3 calls in _interpreters._decref() is raising a bogus exception.
It isn't clear to me how gh-118157 would have caused this to happen. As a very temporary solution, I've skipped test_interpreters on free-threading builds.
This leads to slightly different behavior during interpreter shutdown. A lot of finalizers normally get called during finalize_remove_modules, but here that's not happening because the module's dictionary was immortalized.
By the time Interpreter.__del__ is called, a lot of global variables including _interpreters have already been deleted from the module's dictionary so you get a bunch of weird errors, and the interpreter is not deleted.
I'm not sure. We could call the finalizers on all GC-enabled objects during shutdown in the free-threaded build (before we start clearing module's dictionaries). I think that makes logical sense: everything is going away, why not call their finalizers? That seems to fix test_interpreters, but a few other test cases would need tweaking.
Bug report
Bug description:
On one of my recent PRs, gh-118157, I was getting consistent failures on CI in test_interpreters. Different tests would fail, but the error was always the same:
(expand)
The error looks like this:
I'm fairly sure that means one of the 3 calls in
_interpreters._decref()
is raising a bogus exception.It isn't clear to me how gh-118157 would have caused this to happen. As a very temporary solution, I've skipped test_interpreters on free-threading builds.
CC @colesbury
CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
The text was updated successfully, but these errors were encountered: