`Error: Process completed with exit code 139.` on Python 3.11.0-alpha.3

[oittaa]

Describe the bug

I noticed that my GitHub tests that use 3.11-dev as one of their Python versions started to fail after alpha 3 release.

/home/runner/work/_temp/2b7509c5-dbf2-4144-be0a-3be053d8594c.sh: line 1:  1866 Segmentation fault      (core dumped) coverage run -m unittest discover
Error: Process completed with exit code 139.

To Reproduce

1. Create a GitHub action that uses 3.11-dev as a Python version.
2. Create a step with coverage run -m unittest discover

Expected behavior
No segmentation faults.

Additional context
For example https://github.com/oittaa/ibkr-report-parser/runs/4615838413?check_suite_focus=true#step:8:12 and https://github.com/oittaa/uuid6-python/runs/4615723292?check_suite_focus=true#step:7:23

[jepler]

I am seeing similar. Some fraction of the time, it does succeed. (about 50/50)

As this seems to devolve to a problem with MyFrame_Lasti it is maybe

• frame->f_lasti will be removed in Python 3.11 #1241

1. Use a Debian / Ubuntu amd64 system
2. Un-tar https://github.com/actions/python-versions/releases/download/3.11.0-alpha.3-120611/python-3.11.0-alpha.3-win32-x64.zip in /tmp/py311
3. Fix the binary so it doesn't need LD_LIBRARY_PATH set, or to be untarred in a specific location (not strictly necessary, can use LD_LIBRARY_PATH=/tmp/py311/lib instead): chrpath -r '${ORIGIN}/../lib' /tmp/py311/bin/python3
4. Create a virtual environment in /tmp/env311: /tmp/py311/bin/python -mvenv create /tmp/env311; . /tmp/env311/bin/activate
5. git clone https://github.com/jepler/wwvbpy && cd wwvbpy && pip install -r requirements_dev.txt (tested at 2afca67091e607c4032ad0600361d578bfdafaf6)
6. gdb --args python -m coverage run --branch -p -m wwvb.gen -m 1 2020 1 1 12 30
7. At the gdb prompt, run.

(gdb) r
Starting program: /tmp/311_env/bin/python -m coverage run --branch -p -m wwvb.gen -m 1 2020 1 1 12 30
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
CTracer_handle_return (frame=0x7ffff6797d70, self=0x7ffff67e6c70) at coverage/ctracer/tracer.c:698
698	coverage/ctracer/tracer.c: No such file or directory.
(gdb) where
#0  CTracer_handle_return (frame=0x7ffff6797d70, self=0x7ffff67e6c70) at coverage/ctracer/tracer.c:698
#1  CTracer_trace (self=0x7ffff67e6c70, frame=0x7ffff6797d70, what=3, arg_unused=<optimized out>) at coverage/ctracer/tracer.c:784
#2  0x00007ffff7cabcb3 in call_trace (func=0x7ffff7108f70 <CTracer_trace>, obj=0x7ffff67e6c70, tstate=0x555555576b70, frame=0x7ffff7fc1ff0, what=<optimized out>, arg=<optimized out>)
    at Python/ceval.c:6331
#3  0x00007ffff7cabdcc in call_trace_protected (func=0x7ffff7108f70 <CTracer_trace>, obj=0x7ffff67e6c70, tstate=0x555555576b70, frame=0x7ffff7fc1ff0, what=3, 
    arg=0x7ffff7e4a060 <_Py_NoneStruct>) at Python/ceval.c:6286
#4  0x00007ffff7cabe50 in trace_function_exit (tstate=0x555555576b70, frame=0x7ffff7fc1ff0, retval=0x7ffff7e4a060 <_Py_NoneStruct>) at Python/ceval.c:1612
#5  0x00007ffff7c279f8 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2475
#6  0x00007ffff7c1d7cc in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fc1ff0, tstate=0x555555576b70) at ./Include/internal/pycore_ceval.h:48
#7  _PyEval_Vector (tstate=<optimized out>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=0, kwnames=<optimized out>) at Python/ceval.c:5937
#8  0x00007ffff7cab4ce in PyEval_EvalCode (co=co@entry=0x7ffff644b670, globals=globals@entry=0x7ffff6444580, locals=locals@entry=0x7ffff6444580) at Python/ceval.c:1138
#9  0x00007ffff7caa44f in builtin_exec_impl (module=<optimized out>, locals=0x7ffff6444580, globals=0x7ffff6444580, source=0x7ffff644b670) at Python/bltinmodule.c:1057
#10 builtin_exec (module=<optimized out>, args=args@entry=0x7ffff665abd8, nargs=<optimized out>) at Python/clinic/bltinmodule.c.h:371
#11 0x00007ffff7bd8012 in cfunction_vectorcall_FASTCALL (func=0x7ffff72f8ae0, args=0x7ffff665abd8, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:430
#12 0x00007ffff7ba1a5a in PyObject_Call () from /tmp/py311/bin/../lib/libpython3.11.so.1.0
#13 0x00007ffff7c2678e in do_call_core (use_tracing=<optimized out>, kwdict=0x7ffff64449c0, callargs=0x7ffff665abc0, func=0x7ffff72f8ae0, tstate=<optimized out>) at Python/ceval.c:6736
#14 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4903
#15 0x00007ffff7c1d7cc in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fc1c88, tstate=0x555555576b70) at ./Include/internal/pycore_ceval.h:48
#16 _PyEval_Vector (tstate=<optimized out>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=2, kwnames=<optimized out>) at Python/ceval.c:5937
#17 0x00007ffff7ba0b8e in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffffffccd0, callable=0x7ffff7300540, tstate=0x555555576b70)
    at ./Include/internal/pycore_call.h:89
#18 object_vacall (tstate=0x555555576b70, base=<optimized out>, callable=0x7ffff7300540, vargs=<optimized out>) at Objects/call.c:784
#19 0x00007ffff7ba206c in _PyObject_CallMethodIdObjArgs (obj=0x0, name=name@entry=0x7ffff7e543a0 <PyId__find_and_load.17574>) at Objects/call.c:874
#20 0x00007ffff7c405d0 in import_find_and_load (abs_name=0x7ffff64547f0, tstate=0x555555576b70) at Python/import.c:1739
#21 PyImport_ImportModuleLevelObject (name=name@entry=0x7ffff6cb31b0, globals=<optimized out>, locals=<optimized out>, fromlist=fromlist@entry=0x7ffff7326e60, level=1)
    at Python/import.c:1840
#22 0x00007ffff7c23e99 in import_name (level=0x7ffff7f4c418 <_PyRuntime+280>, fromlist=0x7ffff7326e60, name=0x7ffff6cb31b0, frame=0x7ffff7fc1bc0, tstate=<optimized out>)
    at Python/ceval.c:6837
#23 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3953
#24 0x00007ffff7c1d7cc in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fc1bc0, tstate=0x555555576b70) at ./Include/internal/pycore_ceval.h:48
#25 _PyEval_Vector (tstate=<optimized out>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=0, kwnames=<optimized out>) at Python/ceval.c:5937
#26 0x00007ffff7cab4ce in PyEval_EvalCode (co=co@entry=0x7ffff644b4b0, globals=globals@entry=0x7ffff6444b00, locals=locals@entry=0x7ffff6444b00) at Python/ceval.c:1138
#27 0x00007ffff7caa44f in builtin_exec_impl (module=<optimized out>, locals=0x7ffff6444b00, globals=0x7ffff6444b00, source=0x7ffff644b4b0) at Python/bltinmodule.c:1057
#28 builtin_exec (module=<optimized out>, args=args@entry=0x7ffff665af18, nargs=<optimized out>) at Python/clinic/bltinmodule.c.h:371
#29 0x00007ffff7bd8012 in cfunction_vectorcall_FASTCALL (func=0x7ffff72f8ae0, args=0x7ffff665af18, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:430
#30 0x00007ffff7ba1a5a in PyObject_Call () from /tmp/py311/bin/../lib/libpython3.11.so.1.0
#31 0x00007ffff7c2678e in do_call_core (use_tracing=<optimized out>, kwdict=0x7ffff6444840, callargs=0x7ffff665af00, func=0x7ffff72f8ae0, tstate=<optimized out>) at Python/ceval.c:6736
#32 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4903
#33 0x00007ffff7c1d7cc in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fc1858, tstate=0x555555576b70) at ./Include/internal/pycore_ceval.h:48
#34 _PyEval_Vector (tstate=<optimized out>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=2, kwnames=<optimized out>) at Python/ceval.c:5937
#35 0x00007ffff7ba0b8e in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffffffd4d0, callable=0x7ffff7300540, tstate=0x555555576b70)
    at ./Include/internal/pycore_call.h:89
#36 object_vacall (tstate=0x555555576b70, base=<optimized out>, callable=0x7ffff7300540, vargs=<optimized out>) at Objects/call.c:784
#37 0x00007ffff7ba206c in _PyObject_CallMethodIdObjArgs (obj=0x0, name=name@entry=0x7ffff7e543a0 <PyId__find_and_load.17574>) at Objects/call.c:874
#38 0x00007ffff7c405d0 in import_find_and_load (abs_name=0x7ffff64445f0, tstate=0x555555576b70) at Python/import.c:1739
#39 PyImport_ImportModuleLevelObject (name=0x7ffff64445f0, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ffff666ed00, level=0) at Python/import.c:1840
#40 0x00007ffff7c1bc6c in builtin___import__ (self=<optimized out>, args=<optimized out>, kwds=<optimized out>) at Python/bltinmodule.c:272
#41 0x00007ffff7bd7f62 in cfunction_call (func=0x7ffff72f8680, args=<optimized out>, kwargs=<optimized out>) at Objects/methodobject.c:543
#42 0x00007ffff7ba0199 in _PyObject_MakeTpCall (tstate=0x555555576b70, callable=0x7ffff72f8680, args=<optimized out>, nargs=<optimized out>, keywords=<optimized out>) at Objects/call.c:214
#43 0x00007ffff7c27329 in trace_call_function (kwnames=0x7ffff7327b80, nargs=1, args=<optimized out>, func=0x7ffff72f8680, tstate=<optimized out>) at Python/ceval.c:6700
#44 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4658
#45 0x00007ffff7c1d7cc in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fc11d8, tstate=0x555555576b70) at ./Include/internal/pycore_ceval.h:48
#46 _PyEval_Vector (tstate=<optimized out>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=0, kwnames=<optimized out>) at Python/ceval.c:5937
#47 0x00007ffff7cab4ce in PyEval_EvalCode (co=co@entry=0x7ffff713e170, globals=globals@entry=0x7ffff73613c0, locals=locals@entry=0x7ffff73613c0) at Python/ceval.c:1138
#48 0x00007ffff7caa44f in builtin_exec_impl (module=<optimized out>, locals=0x7ffff73613c0, globals=0x7ffff73613c0, source=0x7ffff713e170) at Python/bltinmodule.c:1057
#49 builtin_exec (module=<optimized out>, args=args@entry=0x7ffff7fc1190, nargs=<optimized out>) at Python/clinic/bltinmodule.c.h:371
#50 0x00007ffff7bd8012 in cfunction_vectorcall_FASTCALL (func=func@entry=0x7ffff72f8ae0, args=0x7ffff7fc1190, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:430
#51 0x00007ffff7ba098b in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=0x7ffff72f8ae0, tstate=0x555555576b70)
    at ./Include/internal/pycore_call.h:89
#52 PyObject_Vectorcall (callable=0x7ffff72f8ae0, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:298
#53 0x00007ffff7c1f354 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4661
#54 0x00007ffff7c1d7cc in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fc1020, tstate=0x555555576b70) at ./Include/internal/pycore_ceval.h:48
#55 _PyEval_Vector (tstate=<optimized out>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=2, kwnames=<optimized out>) at Python/ceval.c:5937
#56 0x00007ffff7ba1a5a in PyObject_Call () from /tmp/py311/bin/../lib/libpython3.11.so.1.0
#57 0x00007ffff7ccac07 in pymain_run_module (modname=<optimized out>, set_argv0=<optimized out>) at Modules/main.c:294
#58 0x00007ffff7cca96b in pymain_run_python (exitcode=0x7fffffffddc0) at Modules/main.c:587
#59 Py_RunMain () at Modules/main.c:672
#60 0x00007ffff7cca4cd in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:726
#61 0x00007ffff77ead0a in __libc_start_main (main=0x555555555060 <main>, argc=15, argv=0x7fffffffdff8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, 
    stack_end=0x7fffffffdfe8) at ../csu/libc-start.c:308
#62 0x000055555555509e in _start ()

local variables of CTracer_handle_return, from a different run:

Program received signal SIGSEGV, Segmentation fault.
CTracer_handle_return (frame=0x7ffff67a7cc0, self=0x7ffff67e6d30)
    at coverage/ctracer/tracer.c:698
698	coverage/ctracer/tracer.c: No such file or directory.
(gdb) info local
bytecode = <optimized out>
pCode = 0x7ffff66ae040
lasti = -268943664
ret = -1
error = <optimized out>
ret = <optimized out>
error = <optimized out>
bytecode = <optimized out>
pCode = <optimized out>
lasti = <optimized out>
first = <optimized out>
val = <optimized out>

It looks like the erroneous-seeming value in lasti is a step in the chain that causes the segfault. It's the actual subscript being used on PyBytes_AS_STRING(pCode).

 693             int lasti = MyFrame_lasti(frame);
 694 
 695             if (lasti < PyBytes_GET_SIZE(pCode)) {    
 696                 bytecode = PyBytes_AS_STRING(pCode)[lasti];
 697             }
 698             if (bytecode != YIELD_VALUE) {
 699                 int first = MyFrame_GetCode(frame)->co_firstlineno;

(gdb) x/i $pc
=> 0x7ffff71093ef <CTracer_trace+1151>:	cmpb   $0x56,0x20(%rdx,%rax,1)
(gdb) p PyBytes_AsString(pCode)
$14 = 0x7ffff66ae060 "U"
(gdb) p/x $rdx
$15 = 0x7ffff66ae040
(gdb) p $rax
$16 = -268943664
(gdb) p lasti
$17 = -268943664

Output of coverage debug sys:

-- sys -------------------------------------------------------
               coverage_version: 6.2
                coverage_module: /tmp/311_env/lib/python3.11/site-packages/coverage/__init__.py
                         tracer: -none-
                        CTracer: available
           plugins.file_tracers: -none-
            plugins.configurers: -none-
      plugins.context_switchers: -none-
              configs_attempted: .coveragerc
                   configs_read: /home/jepler/src/wwvbpy/.coveragerc
                    config_file: /home/jepler/src/wwvbpy/.coveragerc
                config_contents: b'# SPDX-FileCopyrightText: 2021 Jeff Epler\n#\n# SPDX-License-Identifier: GPL-3.0-only\n[run]\nomit =\n    */site-packages/*\n    test*.py\n'
                      data_file: -none-
                         python: 3.11.0a3 (main, Dec 22 2021, 10:17:10) [GCC 9.3.0]
                       platform: Linux-5.10.0-8-amd64-x86_64-with-glibc2.31
                 implementation: CPython
                     executable: /tmp/311_env/bin/python
                   def_encoding: utf-8
                    fs_encoding: utf-8
                            pid: 2552809
                            cwd: /home/jepler/src/wwvbpy
                           path: /home/jepler/src/wwvbpy
                                 /opt/hostedtoolcache/Python/3.11.0-alpha.3/x64/lib/python311.zip
                                 /tmp/py311/lib/python3.11
                                 /tmp/py311/lib/python3.11/lib-dynload
                                 /tmp/311_env/lib/python3.11/site-packages
                    environment: HOME = /home/jepler
                   command_line: /tmp/311_env/lib/python3.11/site-packages/coverage/__main__.py debug sys
                sqlite3_version: 2.6.0
         sqlite3_sqlite_version: 3.34.1
             sqlite3_temp_store: 0
        sqlite3_compile_options: COMPILER=gcc-10.2.1 20210110; ENABLE_COLUMN_METADATA; ENABLE_DBSTAT_VTAB
                                 ENABLE_FTS3; ENABLE_FTS3_PARENTHESIS; ENABLE_FTS3_TOKENIZER
                                 ENABLE_FTS4; ENABLE_FTS5; ENABLE_JSON1
                                 ENABLE_LOAD_EXTENSION; ENABLE_PREUPDATE_HOOK; ENABLE_RTREE
                                 ENABLE_SESSION; ENABLE_STMTVTAB; ENABLE_UNLOCK_NOTIFY
                                 ENABLE_UPDATE_DELETE_LIMIT; HAVE_ISNAN; LIKE_DOESNT_MATCH_BLOBS
                                 MAX_SCHEMA_RETRY=25; MAX_VARIABLE_NUMBER=250000; OMIT_LOOKASIDE
                                 SECURE_DELETE; SOUNDEX; TEMP_STORE=1
                                 THREADSAFE=1; USE_URI

Frustratingly, running under valgrind never reproduced the problem in 30 runs.

[jepler]

Omitting --branch or adding --timid seems to make my test case succeed consistently.

[parafoxia]

If you change your action to use 3.11.0-alpha.2 instead of 3.11-dev it will work. For some reason this seems to be an issue related specifically to 3.11.0a3. I've been trying to work out exactly what causes it but I can't seem to pinpoint it.

[jouve]

I think the issue is related to this binary incompatible change to struct _interpreter_frame between alpha.2 and alpha.3 : python/cpython@135cabd#diff-dc43b169f3202922a7787b97210ce299172f0cf03bd88529d473128f32e49fe4

and my guess that 6.2 was compiled before the change.

This show in disassembly of tracer.cpython-311-x86_64-linux-gnu.so from the wheel

693     in coverage/ctracer/tracer.c
   0x00000000000033de <+1134>:  mov    0x18(%r15),%rax       # frame->f_frame
   0x00000000000033e2 <+1138>:  mov    0x38(%rax),%eax       # f_frame->f_lasti
   0x00000000000033e5 <+1141>:  add    %eax,%eax

f_lasti is acceced at 0x38 = 0x08 * 7 (PyObject *f_globals to PyObject *generator)
where the correct offset should be 0x40 = 0x08 * 8 (PyFunctionObject *f_func to PyObject *generator)

The can be verified by accessing the value at 0x40 and comparing to python's own frame_getlasti :

(gdb) p *(int*)((void*)frame->f_frame + 0x40) * 2
$189 = 16
(gdb) p PyLong_AsLong(frame_getlasti(frame, 0))     
$186 = 16

[nedbat]

I guess I shouldn't publish wheels for alpha versions of Python? Everyone here: does the problem go away if you install coverage from git?

pip install git+https://github.com/nedbat/coveragepy.git@6.2

[jouve]

@nedbat, I think MyFrame_lasti in both
https://github.com/nedbat/coveragepy/blob/master/coverage/ctracer/util.h#L20
and
https://github.com/nedbat/coveragepy/blob/master/coverage/ctracer/util.h#L24

could be replaced with something like

#define MyFrame_lasti PyLong_AsLong(PyObject_GetAttrString(frame, "lasti"))

ref: https://github.com/python/cpython/blob/v3.10.0a7/Objects/frameobject.c#L64 was added in v3.10.0a7 and https://github.com/python/cpython/blob/main/Objects/frameobject.c#L68 (main)

[jepler]

I guess I shouldn't publish wheels for alpha versions of Python? Everyone here: does the problem go away if you install coverage from git?

pip install git+https://github.com/nedbat/coveragepy.git@6.2

yes, that worked for me.