test_profiling results in a segmentation fault when Python 3.15.0a2 is built for i686 architecture

[befeleme]

Bug report

Bug description:

Hello,

test_gc_frames_enabled has been added very recently in: #140643
In Fedora Linux, during the i686 build, the test segfaults on all supported Fedora versions (42, 43, 44).

The trace from the test rerun with the debug Python is below. Let me know if you need more data to make the issue more robust.

(gdb) run  -m test.regrtest test_profiling
Starting program: /usr/bin/python3.15d -m test.regrtest test_profiling
Program received signal SIGSEGV, Segmentation fault.
0xf79fc6cc in _PyType_HasFeature (type=0x70000000, feature=16384)
    at /usr/src/debug/python3.15-3.15.0~a2-1.fc44.i386/Include/internal/pycore_object.h:316
316         return ((type->tp_flags) & feature) != 0;
(gdb) bt
#0  0xf79fc6cc in _PyType_HasFeature (type=0x70000000, feature=16384)
    at /usr/src/debug/python3.15-3.15.0~a2-1.fc44.i386/Include/internal/pycore_object.h:316
#1  0xf79fceef in maybe_tracked (ob=<unknown at remote 0xf7f6ce24>)
    at /usr/src/debug/python3.15-3.15.0~a2-1.fc44.i386/Objects/tupleobject.c:168
#2  0xf79fd7ba in _PyTuple_FromStackRefStealOnSuccess (src=0xf7fb7cf0, n=3)
    at /usr/src/debug/python3.15-3.15.0~a2-1.fc44.i386/Objects/tupleobject.c:426
#3  0xf7b04689 in _PyEval_EvalFrameDefault (tstate=0xf7f89e78 <_PyRuntime+186616>, frame=0xf7fb7c98, throwflag=0)
    at /usr/src/debug/python3.15-3.15.0~a2-1.fc44.i386/Python/generated_cases.c.h:1486
#4  0xf7afb02f in _PyEval_EvalFrame (tstate=0xf7f89e78 <_PyRuntime+186616>, frame=0xf7fb79ec, throwflag=0)
    at /usr/src/debug/python3.15-3.15.0~a2-1.fc44.i386/Include/internal/pycore_ceval.h:121
#5  0xf7b37c90 in _PyEval_Vector (tstate=0xf7f89e78 <_PyRuntime+186616>, func=0xf7078ed0, locals=0x0, 
    args=0xfffe8ad8, argcount=2, kwnames=0x0) at /usr/src/debug/python3.15-3.15.0~a2-1.fc44.i386/Python/ceval.c:2145
#6  0xf793f953 in _PyFunction_Vectorcall (func=<function at remote 0xf7078ed0>, stack=0xfffe8ad8, nargsf=2, 
    kwnames=0x0) at /usr/src/debug/python3.15-3.15.0~a2-1.fc44.i386/Objects/call.c:413
#7  0xf7942ce8 in _PyObject_VectorcallTstate (tstate=0xf7f89e78 <_PyRuntime+186616>, 
    callable=<function at remote 0xf7078ed0>, args=0xfffe8ad8, nargsf=2, kwnames=0x0)
    at /usr/src/debug/python3.15-3.15.0~a2-1.fc44.i386/Include/internal/pycore_call.h:169
#8  0xf794379d in method_vectorcall (method=<method at remote 0xf6432cf0>, args=0xf65db0b8, nargsf=1, kwnames=0x0)
    at /usr/src/debug/python3.15-3.15.0~a2-1.fc44.i386/Objects/classobject.c:95
#9  0xf793f38a in _PyVectorcall_Call (tstate=0xf7f89e78 <_PyRuntime+186616>, func=0xf7943538 <method_vectorcall>, 
    callable=<method at remote 0xf6432cf0>, 
    tuple=(<RegressionTestResult(failfast=False, failures=[], errors=[], testsRun=6, skipped=[], expectedFailures=[], unexpectedSuccesses=[], collectedDurations=[('test_cpu_mode_integration_filtering (test.test_profiling.test_sampling_profiler.TestCpuModeFiltering.test_cpu_mode_integration_filtering)', <float at remote 0xf642c068>), ('test_cpu_mode_with_no_samples (test.test_profiling.test_sampling_profiler.TestCpuModeFiltering.test_cpu_mode_with_no_samples)', <float at remote 0xf6484e68>), ('test_frames_filtered_with_skip_idle (test.test_profiling.test_sampling_profiler.TestCpuModeFiltering.test_frames_filtered_with_skip_idle)', <float at remote 0xf6484e88>), ('test_mode_validation (test.test_profiling.test_sampling_profiler.TestCpuModeFiltering.test_mode_validation)', <float at remote 0xf6484d68>), ('test_gc_frames_disabled (test.test_profiling.test_sampling_profiler.TestGCFrameTracking.test_gc_frames_disabled)', <float at remote 0xf62b9708>)], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _st...(truncated), kwargs={})
    at /usr/src/debug/python3.15-3.15.0~a2-1.fc44.i386/Objects/call.c:273

CPython versions tested on:

3.15

Operating systems tested on:

Linux

Linked PRs

• gh-141784: Fix _remote_debugging_module.c compilation on 32-bit Linux #141796
• [3.14] gh-141784: Fix _remote_debugging_module.c compilation on 32-bit Linux (#141796) #141800

[Eclips4]

cc @brandtbucher @pablogsal

[pablogsal]

Hummm unfortunately the trace is not very useful because it basically points out to a corrupted object. Any chance we can get ssh access into one of these machines?

[hroncok]

FWIW, I was able to reproduce this via mock, our chroot thinggy from Fedora, from cpython git:

If you don't run Fedora yet, start with that:

$ podman run --rm --privileged -ti fedora:43

If you already are on Fedora, go ahead:

bash-5.3# dnf install -y mock
bash-5.3# useradd mockbuilder  # or use your personal user
bash-5.3# usermod -a -G mock mockbuilder
bash-5.3# su - mockbuilder

[mockbuilder@8d9a3acca7bc ~]$ mock -r fedora-rawhide-i386 init
[mockbuilder@8d9a3acca7bc ~]$ mock -r fedora-rawhide-i386 install gcc python3.15 autoconf git-core make gdb
[mockbuilder@8d9a3acca7bc ~]$ mock -r fedora-rawhide-i386 shell --unpriv --enable-network
<mock-chroot> sh-5.3$ git clone https://github.com/python/cpython.git --depth 1
<mock-chroot> sh-5.3$ cd cpython/
<mock-chroot> sh-5.3$ ./configure --config-cache --without-ensurepip
<mock-chroot> sh-5.3$ make -j10
<mock-chroot> sh-5.3$ ./python -m test.regrtest test_profiling
Fatal Python error: Segmentation fault

Current thread 0xf7c97700 [python] (most recent call first):
  File "/builddir/cpython/Lib/profiling/sampling/pstats_collector.py", line 26 in _process_frames
  File "/builddir/cpython/Lib/profiling/sampling/pstats_collector.py", line 45 in collect
  File "/builddir/cpython/Lib/profiling/sampling/sample.py", line 175 in sample
  File "/builddir/cpython/Lib/profiling/sampling/sample.py", line 654 in sample
  File "/builddir/cpython/Lib/test/test_profiling/test_sampling_profiler/test_advanced.py", line 72 in test_gc_frames_enabled
  File "/builddir/cpython/Lib/unittest/case.py", line 613 in _callTestMethod
  File "/builddir/cpython/Lib/unittest/case.py", line 667 in run
  File "/builddir/cpython/Lib/unittest/case.py", line 723 in __call__
  File "/builddir/cpython/Lib/unittest/suite.py", line 122 in run
  File "/builddir/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/builddir/cpython/Lib/unittest/suite.py", line 122 in run
  File "/builddir/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/builddir/cpython/Lib/unittest/suite.py", line 122 in run
  File "/builddir/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/builddir/cpython/Lib/unittest/suite.py", line 122 in run
  File "/builddir/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/builddir/cpython/Lib/unittest/suite.py", line 122 in run
  File "/builddir/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/builddir/cpython/Lib/test/libregrtest/testresult.py", line 148 in run
  File "/builddir/cpython/Lib/test/libregrtest/single.py", line 84 in _run_suite
  File "/builddir/cpython/Lib/test/libregrtest/single.py", line 42 in run_unittest
  File "/builddir/cpython/Lib/test/libregrtest/single.py", line 162 in test_func
  File "/builddir/cpython/Lib/test/libregrtest/single.py", line 118 in regrtest_runner
  File "/builddir/cpython/Lib/test/libregrtest/single.py", line 165 in _load_run_test
  File "/builddir/cpython/Lib/test/libregrtest/single.py", line 210 in _runtest_env_changed_exc
  File "/builddir/cpython/Lib/test/libregrtest/single.py", line 319 in _runtest
  File "/builddir/cpython/Lib/test/libregrtest/single.py", line 348 in run_single_test
  File "/builddir/cpython/Lib/test/libregrtest/main.py", line 393 in run_test
  File "/builddir/cpython/Lib/test/libregrtest/main.py", line 423 in run_tests_sequentially
  File "/builddir/cpython/Lib/test/libregrtest/main.py", line 563 in _run_tests
  File "/builddir/cpython/Lib/test/libregrtest/main.py", line 598 in run_tests
  File "/builddir/cpython/Lib/test/libregrtest/main.py", line 779 in main
  File "/builddir/cpython/Lib/test/libregrtest/main.py", line 787 in main
  File "/builddir/cpython/Lib/test/regrtest.py", line 43 in _main
  File "/builddir/cpython/Lib/test/regrtest.py", line 47 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Current thread's C stack trace (most recent call first):
  Binary file "./python", at _Py_DumpStack+0x29 [0x82b50d9]
  Binary file "./python" [0x82ce708]
  Binary file "linux-gate.so.1", at __kernel_sigreturn+0x0 [0xf7fa95a0]
  Binary file "./python", at _PyTuple_FromStackRefStealOnSuccess+0x60 [0x81484e0]
  Binary file "./python", at _PyEval_EvalFrameDefault+0x49ac [0x821042c]
  Binary file "./python" [0x821d97f]
  Binary file "./python", at _PyFunction_Vectorcall+0x38 [0x80a9568]
  Binary file "./python" [0x80ac87d]
  Binary file "./python", at PyObject_Call+0xb4 [0x80aa754]
  Binary file "./python", at _PyEval_EvalFrameDefault+0x6e94 [0x8212914]
  Binary file "./python" [0x821d97f]
  Binary file "./python", at _PyFunction_Vectorcall+0x38 [0x80a9568]
  Binary file "./python" [0x80aac60]
  Binary file "./python" [0x8159a26]
  Binary file "./python", at _PyObject_MakeTpCall+0xaa [0x80a8b6a]
  Binary file "./python", at PyObject_Vectorcall+0x7c [0x80a938c]
  Binary file "./python", at _PyEval_EvalFrameDefault+0x5c70 [0x82116f0]
  Binary file "./python" [0x821d97f]
  Binary file "./python", at _PyFunction_Vectorcall+0x38 [0x80a9568]
  Binary file "./python" [0x80ac87d]
  Binary file "./python", at PyObject_Call+0xb4 [0x80aa754]
  Binary file "./python", at _PyEval_EvalFrameDefault+0x6e94 [0x8212914]
  Binary file "./python" [0x821d97f]
  Binary file "./python", at _PyFunction_Vectorcall+0x38 [0x80a9568]
  Binary file "./python" [0x80aac60]
  Binary file "./python" [0x8159a26]
  Binary file "./python", at _PyObject_MakeTpCall+0xaa [0x80a8b6a]
  Binary file "./python", at PyObject_Vectorcall+0x7c [0x80a938c]
  Binary file "./python", at _PyEval_EvalFrameDefault+0x5c70 [0x82116f0]
  Binary file "./python" [0x821d97f]
  Binary file "./python", at _PyFunction_Vectorcall+0x38 [0x80a9568]
  Binary file "./python" [0x80ac87d]
  <truncated rest of calls>

Extension modules: math.integer (total: 1)
Segmentation fault         (core dumped) ./python -m test.regrtest test_profiling

[vstinner]

I modified Modules/_remote_debugging_module.c to have this code:

            // Use "~" as file and 0 as line, since that's what pstats uses:
            PyObject *file = _Py_LATIN1_CHR('~');
            PyObject *line = _PyLong_GetZero();
            PyObject *func = extra_frame;
            PyObject *extra_frame_info = make_frame_info(
                unwinder, file, line, func);

I put a breakpoint on make_frame_info. In gdb, I get:

(gdb) p *file
$4 = {{ob_refcnt = 125, _aligner = 125 '}'}, ob_type = 0x70000000}

(gdb) p *line
$5 = {{ob_refcnt = 1, _aligner = 1 '\001'}, ob_type = 0x70000000}

(gdb) p *func
$6 = {{ob_refcnt = 7627117, _aligner = 109 'm'}, ob_type = 0x70000000}

In fact, static objects seem to be corrupted for an unknown reason:

(gdb) p _PyRuntime.static_objects.singletons.strings.ascii['~']
$7 = {_ascii = {ob_base = {{ob_refcnt = 125, _aligner = 125 '}'}, ob_type = 0x70000000}, 
    length = 140990720, hash = 1, state = {interned = 0, kind = 1, compact = 1, ascii = 1, 
      statically_allocated = 1}}, _data = <incomplete sequence \347>}

(gdb) p _PyRuntime.static_objects.singletons.strings.ascii['~']._ascii.ob_base.ob_type
$8 = (PyTypeObject *) 0x70000000

I tried to put a breakpoint (hardware watchpoint) on _PyRuntime.static_objects.singletons.strings.ascii['~']._ascii.ob_base.ob_type, but it didn't work (it doesn't stop, even if the value is modified).

[vstinner]

Depending on the frame, _PyRuntime doesn't have the same size!? In _remote_debugging_module.c, the structure is 4 bytes smaller!

(gdb) frame 0
#0  make_frame_info (unwinder=0x89b7410, file=0xf71b3088, line=0x868de90 <_PyRuntime+16816>, 
    func=0xf71b3058) at ./Modules/_remote_debugging_module.c:1823
1823        RemoteDebuggingState *state = RemoteDebugging_GetStateFromObject((PyObject*)unwinder);
(gdb) p sizeof(_PyRuntime)
$11 = 187392

(gdb) frame 7
#7  0x080fe71f in method_vectorcall_NOARGS (func=0xf7521d80, args=0xfffec1c0, 
    nargsf=2147483649, kwnames=0x0) at Objects/descrobject.c:448
448         PyObject *result = _PyCFunction_TrampolineCall(meth, args[0], NULL);
(gdb) p sizeof(_PyRuntime)
$13 = 187396

[vstinner]

Ok, the guilty is _PyRuntime.pyhash_state.urandom_cache.st_ino:

• in Python, it's 64 bits
• in _remote_debugging_module.c, it's 32 bits

[vstinner]

• pyconfig.h defines #define _FILE_OFFSET_BITS 64
• Python.h includes pyconfig.h
• ino_t type indirectly depends indirectly on _FILE_OFFSET_BITS: depending on _FILE_OFFSET_BITS, it's 32 or 64 bits.
• In _remote_debugging_module.c, ino_t type is defined before Python.h is included.

The fix is to include Python.h first. I'm working on a fix!