-
-
Notifications
You must be signed in to change notification settings - Fork 33.1k
[3.13] gh-134163: fix: exec('_testcapi.set_nomemory(0)') hang in new repl 3.13 #138491
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool, I didn't think anyone would ever end up taking a stab at this. First and foremost, please add a test case (more on that in a second) and blurb entry, as this is user-facing.
Now, I am a little surprised to see this in the eval loop and not somewhere in _pyrepl
(or perhaps Py_Main
). That should mean that there's a reproducer available without the REPL -- would you mind finding that, and then using it for a test case?
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
b59a1ee
to
e557017
Compare
help with gpt I wrote a reproduce python script def simulate_repl_exec():
print("=" * 60)
print("Reproducing issue #134163 outside REPL")
print("Simulating interactive interpreter exec() behavior")
print("=" * 60)
print()
# First, import and set up the memory failure condition
print("Step 1: Setting up memory allocation failure...")
import _testcapi
print("Step 2: Preparing code that will trigger exception handling...")
# Create a code object that will cause exception handling
# This simulates what happens when REPL executes user input
test_code = """
# This code will trigger the problematic code path
# by causing an exception during execution when memory is constrained
import _testcapi
_testcapi.set_nomemory(0) # This line triggers the hang condition
# Any subsequent operation that might need memory allocation
# will trigger the exception handling path where the bug occurs
x = [1, 2, 3] # This should trigger MemoryError
"""
print("Step 3: Compiling test code...")
try:
compiled_code = compile(test_code, "<reproduce_script>", "exec")
except Exception as e:
print(f"Compilation failed: {e}")
return
print("Step 4: Executing code that triggers the hang condition...")
print("BEFORE FIX: This would hang indefinitely")
print("AFTER FIX: This should exit gracefully")
print()
try:
exec(compiled_code, {"__name__": "__console__"})
print("Code executed successfully (unexpected)")
except SystemExit:
print("SystemExit caught - re-raising")
raise
except Exception as e:
print(f"Exception caught during exec(): {type(e).__name__}: {e}")
print("This is the expected path - exception handling should work normally")
# The showtraceback() equivalent would be called here in real REPL
import traceback
traceback.print_exc()
if __name__ == "__main__":
print("This script reproduces the hang condition outside of REPL")
simulate_repl_exec() note the print can not drop.... This is very strange if I drop some print I can not reproduce it.... |
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Lib/test/test_repl.py
Outdated
""") | ||
p = spawn_repl() | ||
with SuppressCrashReport(): | ||
p.stdin.write(user_input) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is still just running it in the REPL. For now, let's switch to the simple repro of import _testcapi; _testcapi.set_nomemory(0)
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
seems not right if so
...it will not hang for 3.13
the test is the same as
@unittest.skipIf(support.Py_TRACE_REFS, 'cannot test Py_TRACE_REFS build')
def test_no_memory(self):
import_module("_testcapi")
# Issue #30696: Fix the interactive interpreter looping endlessly when
# no memory. Check also that the fix does not break the interactive
# loop when an exception is raised.
user_input = """
import sys, _testcapi
1/0
print('After the exception.')
_testcapi.set_nomemory(0)
sys.exit(0)
"""
user_input = dedent(user_input)
p = spawn_repl()
with SuppressCrashReport():
p.stdin.write(user_input)
output = kill_python(p)
self.assertIn('After the exception.', output)
# Exit code 120: Py_FinalizeEx() failed to flush stdout and stderr.
self.assertIn(p.returncode, (1, 120))
if we use test
import _testcapi; _testcapi.set_nomemory(0).
it always pass with or without the patch
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I meant that we don't need spawn_repl
here. subprocess.run([sys.executable, '-c', user_input])
should work as a test -- there's no need to involve the REPL.
the reason is that: When there are no four print statements, the Python command is as follows: So, it can be observed that the instruction before executing set_nomemory(0) is around 208. When I used GDB to monitor: int frame_lasti = _PyInterpreterFrame_LASTI(frame);
PyObject *lasti = PyLong_FromLong(frame_lasti);
if (lasti == NULL) {
goto exception_unwind;
} The frame_lasti here is 208, which is a small integer. Subsequently, when calling PyLong_FromLong, malloc is not invoked, so no MemError is triggered. However, once print statements are added, the Python instructions become: text #define _PY_NSMALLPOSINTS 257 At this point, PyLong_FromLong(frame_lasti) triggers malloc, causing a memory error and subsequently leading to an infinite loop. Well, it's not exactly coincidental, but the general idea is that adding print statements increases the value of frame_lasti. When this value exceeds the small integer limit, malloc is called, triggering an error and resulting in an infinite loop. The Python REPL has a complex implementation. It likely runs a series of instructions during initialization. When I checked with GDB, frame_lasti was 409, which exceeds the small integer limit. So, this should indeed be the root cause. so if we run a fake repl it will not hang it is not into the recursvie |
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
btw I tested 3.12 there's no such problem it only exists in 3.13 -> 3.14a2 |
Ah, that makes sense. Sounds like we can definitely repro without the prints, though. |
yes, going to try |
a simple way can you try: 14 times malloc a = list(range(1000, 2000))
a = list(range(1000, 2000))
a = list(range(1000, 2000))
a = list(range(1000, 2000))
a = list(range(1000, 2000))
a = list(range(1000, 2000))
a = list(range(1000, 2000))
a = list(range(1000, 2000))
a = list(range(1000, 2000))
a = list(range(1000, 2000))
a = list(range(1000, 2000))
a = list(range(1000, 2000))
a = list(range(1000, 2000))
a = list(range(1000, 2000))
try:
import _testcapi
_testcapi.set_nomemory(0)
b = list(range(1000, 2000)) # one time again
except Exception as e:
import traceback
traceback.print_exc() update list(range(0,1 ) can also trigger the bug |
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
@ZeroIntensity if you have better way to save some code you can continue commit and I tested 3.13 hang in test in the branch passed ok |
Misc/NEWS.d/next/Core_and_Builtins/2025-09-04-11-52-23.gh-issue-134163.EqKyn8.rst
Outdated
Show resolved
Hide resolved
…e-134163.EqKyn8.rst Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
addressed thank you |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Getting close!
🤖 New build scheduled with the buildbot fleet by @ZeroIntensity for commit 52310f0 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F138491%2Fmerge The command will test the builders whose names match following regular expression: The builders matched are:
|
What....we set pymemory err here, it still fail... |
reproduced we must set configure with
|
now all things clear cc @ZeroIntensity we also have this issue in 3.13 or 3.14 even main
Current thread 0x0000ffff9c52e020 (most recent call first): Extension modules: _testcapi (total: 1)
Python 3.14.0rc2+ (heads/3.14:161543f4f11, Sep 9 2025, 09:43:56) [GCC 11.5.0 20240719 (Red Hat 11.5.0-5)] on linux
Current thread 0x0000ffffa00c6020 [python] (most recent call first): Extension modules: _testcapi (total: 1)
Python 3.15.0a0 (heads/main:5edfe55acf2, Sep 9 2025, 09:49:31) [GCC 11.5.0 20240719 (Red Hat 11.5.0-5)] on linux
Current thread 0x0000ffffa2ca8020 [python] (most recent call first): Extension modules: _testcapi (total: 1) |
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
update we already meet this
|
…python into hy/fix_issue_134163
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alas, this looks good to me. I've left two minor comment changes as suggestions below. I'll run the buildbots one more time and then I'll merge this. Thanks for being responsive and respectful to my feedback :)
🤖 New build scheduled with the buildbot fleet by @ZeroIntensity for commit 5ab08c7 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F138491%2Fmerge If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again. |
Thank you learned a lot from this. |
test failed seems not about this patch merge the review comments first |
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
debug process:
First step, of course, was trying to reproduce it. On 3.13, running the statement from the issue in the REPL hangs. On 3.14, it doesn’t. So it’s a 3.13-only problem.
cc @ZeroIntensity you can try this patch in 3.13