Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash on AST with misordered linenos #92597

Closed
JelleZijlstra opened this issue May 10, 2022 · 12 comments · Fixed by #93359
Closed

Crash on AST with misordered linenos #92597

JelleZijlstra opened this issue May 10, 2022 · 12 comments · Fixed by #93359
Labels
3.11 only security fixes 3.12 bugs and security fixes type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@JelleZijlstra
Copy link
Member

  • Compile 3.11 branch
  • ./python.exe -m ensurepip
  • ./python.exe -m pip install pytest-cov
  • ./python.exe -m pytest

Crashes with Assertion failed: (i->i_end_lineno >= i->i_lineno), function write_location_info_long_form, file compile.c, line 7532.

lldb stack trace:

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = hit program assert
    frame #0: 0x00007fff203cd92e libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff203fc5bd libsystem_pthread.dylib`pthread_kill + 263
    frame #2: 0x00007fff20351406 libsystem_c.dylib`abort + 125
    frame #3: 0x00007fff203507d8 libsystem_c.dylib`__assert_rtn + 314
  * frame #4: 0x000000010027c183 python.exe`write_location_info_long_form.cold.2 at compile.c:7532:5 [opt]
    frame #5: 0x000000010018f469 python.exe`write_location_info_long_form(a=<unavailable>, i=<unavailable>, length=<unavailable>) at compile.c:7532:5 [opt]
    frame #6: 0x000000010018f27d python.exe`write_location_info_entry(a=0x00007ffeefbfe5d0, i=0x000000010388ac98, isize=1) at compile.c:7590:5 [opt]
    frame #7: 0x000000010018d2cd python.exe`assemble_emit_location(a=0x00007ffeefbfe5d0, i=0x000000010388ac98) at compile.c:7605:12 [opt]
    frame #8: 0x000000010017e9fa python.exe`assemble(c=<unavailable>, addNone=<unavailable>) at compile.c:8411:18 [opt]
    frame #9: 0x000000010017d4a3 python.exe`compiler_mod(c=0x00007ffeefbfe6b8, mod=<unavailable>) at compile.c:2203:10 [opt]
    frame #10: 0x000000010017d2b3 python.exe`_PyAST_Compile(mod=0x00000001038852b0, filename=0x000000010654d630, flags=<unavailable>, optimize=<unavailable>, arena=0x00000001064b0100) at compile.c:581:10 [opt]
    frame #11: 0x00000001001634e0 python.exe`builtin_compile_impl(module=<unavailable>, source=0x00000001065a3fc0, filename=0x000000010654d630, mode=<unavailable>, flags=<unavailable>, dont_inherit=1, optimize=-1, feature_version=-1) at bltinmodule.c:818:33 [opt]
    frame #12: 0x0000000100161fb6 python.exe`builtin_compile(module=<unavailable>, args=0x00007ffeefbfe7f0, nargs=<unavailable>, kwnames=<unavailable>) at bltinmodule.c.h:328:20 [opt]
    frame #13: 0x00000001000cbb08 python.exe`cfunction_vectorcall_FASTCALL_KEYWORDS(func=0x0000000100b9f4d0, args=0x0000000100c6fe00, nargsf=<unavailable>, kwnames=0x0000000102c72260) at methodobject.c:443:24 [opt]
    frame #14: 0x0000000100087018 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100486930, callable=0x0000000100b9f4d0, args=0x0000000100c6fe00, nargsf=9223372036854775811, kwnames=0x0000000102c72260) at pycore_call.h:92:11 [opt]
    frame #15: 0x0000000100086fa0 python.exe`PyObject_Vectorcall(callable=<unavailable>, args=<unavailable>, nargsf=<unavailable>, kwnames=<unavailable>) at call.c:299:12 [opt]
    frame #16: 0x0000000100170d7c python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100486930, frame=0x0000000100c6fd70, throwflag=<unavailable>) at ceval.c:4826:23 [opt]
    frame #17: 0x000000010016680d python.exe`_PyEval_EvalFrame(tstate=<unavailable>, frame=<unavailable>, throwflag=<unavailable>) at pycore_ceval.h:66:16 [opt]
    frame #18: 0x0000000100166726 python.exe`_PyEval_Vector(tstate=0x0000000100486930, func=0x0000000104737d80, locals=0x0000000000000000, args=0x00007ffeefbfeb80, argcount=3, kwnames=<unavailable>) at ceval.c:6468:24 [opt]
    frame #19: 0x0000000100087330 python.exe`_PyFunction_Vectorcall(func=0x0000000104737d80, stack=0x00007ffeefbfeb80, nargsf=<unavailable>, kwnames=0x0000000000000000) at call.c:0:45 [opt]
    frame #20: 0x0000000100089fd8 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100486930, callable=0x0000000104737d80, args=0x00007ffeefbfeb80, nargsf=3, kwnames=0x0000000000000000) at pycore_call.h:92:11 [opt]
    frame #21: 0x0000000100089644 python.exe`method_vectorcall(method=<unavailable>, args=0x00000001064c9788, nargsf=<unavailable>, kwnames=0x0000000000000000) at classobject.c:89:18 [opt]
    frame #22: 0x0000000100086f3d python.exe`_PyVectorcall_Call(tstate=0x0000000100486930, func=(python.exe`method_vectorcall at classobject.c:44), callable=0x000000010640d310, tuple=0x00000001064c9770, kwargs=0x0000000000000000) at call.c:245:16 [opt]
    frame #23: 0x0000000100087148 python.exe`_PyObject_Call(tstate=0x0000000100486930, callable=0x000000010640d310, args=0x00000001064c9770, kwargs=0x0000000000000000) at call.c:328:16 [opt]
    frame #24: 0x0000000100087209 python.exe`PyObject_Call(callable=<unavailable>, args=<unavailable>, kwargs=<unavailable>) at call.c:355:12 [opt]
    frame #25: 0x0000000100177419 python.exe`do_call_core(tstate=0x0000000100486930, func=0x000000010640d310, callargs=0x00000001064c9770, kwdict=0x0000000000000000, use_tracing=<unavailable>) at ceval.c:0:13 [opt]
    frame #26: 0x00000001001733b0 python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100486930, frame=0x0000000100c6f580, throwflag=<unavailable>) at ceval.c:5431:22 [opt]
    frame #27: 0x000000010016680d python.exe`_PyEval_EvalFrame(tstate=<unavailable>, frame=<unavailable>, throwflag=<unavailable>) at pycore_ceval.h:66:16 [opt]
    frame #28: 0x0000000100166726 python.exe`_PyEval_Vector(tstate=0x0000000100486930, func=0x0000000104053490, locals=0x0000000000000000, args=0x00000001064c0a48, argcount=1, kwnames=<unavailable>) at ceval.c:6468:24 [opt]
    frame #29: 0x0000000100087330 python.exe`_PyFunction_Vectorcall(func=0x0000000104053490, stack=0x00000001064c0a48, nargsf=<unavailable>, kwnames=0x00000001064c8f50) at call.c:0:45 [opt]
    frame #30: 0x00000001000866b2 python.exe`_PyObject_FastCallDictTstate(tstate=0x0000000100486930, callable=0x0000000104053490, args=0x00007ffeefbfef70, nargsf=1, kwargs=0x00000001064c6d50) at call.c:152:15 [opt]
    frame #31: 0x00000001000875e4 python.exe`_PyObject_Call_Prepend(tstate=<unavailable>, callable=<unavailable>, obj=0x000000010622f3a0, args=<unavailable>, kwargs=<unavailable>) at call.c:482:24 [opt]
    frame #32: 0x00000001000ee1f9 python.exe`slot_tp_call(self=0x000000010622f3a0, args=0x000000010046c5a0, kwds=0x00000001064c6d50) at typeobject.c:7604:15 [opt]
    frame #33: 0x000000010008689c python.exe`_PyObject_MakeTpCall(tstate=0x0000000100486930, callable=0x000000010622f3a0, args=0x0000000100c6f408, nargs=<unavailable>, keywords=0x0000000104694960) at call.c:214:18 [opt]
    frame #34: 0x0000000100087049 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100486930, callable=0x000000010622f3a0, args=0x0000000100c6f408, nargsf=9223372036854775808, kwnames=0x0000000104694960) at pycore_call.h:90:16 [opt]
    frame #35: 0x0000000100086fa0 python.exe`PyObject_Vectorcall(callable=<unavailable>, args=<unavailable>, nargsf=<unavailable>, kwnames=<unavailable>) at call.c:299:12 [opt]
    frame #36: 0x0000000100170d7c python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100486930, frame=0x0000000100c6f380, throwflag=<unavailable>) at ceval.c:4826:23 [opt]
    frame #37: 0x000000010016680d python.exe`_PyEval_EvalFrame(tstate=<unavailable>, frame=<unavailable>, throwflag=<unavailable>) at pycore_ceval.h:66:16 [opt]
    frame #38: 0x0000000100166726 python.exe`_PyEval_Vector(tstate=0x0000000100486930, func=0x0000000102881b20, locals=0x0000000102830350, args=0x0000000000000000, argcount=0, kwnames=<unavailable>) at ceval.c:6468:24 [opt]
    frame #39: 0x0000000100166615 python.exe`PyEval_EvalCode(co=<unavailable>, globals=0x0000000102830350, locals=0x0000000102830350) at ceval.c:1207:21 [opt]
    frame #40: 0x0000000100163b09 python.exe`builtin_exec_impl(module=<unavailable>, source=0x000000010617f230, globals=0x0000000102830350, locals=0x0000000102830350, closure=<unavailable>) at bltinmodule.c:1080:17 [opt]
    frame #41: 0x000000010016224d python.exe`builtin_exec(module=<unavailable>, args=0x0000000100c6f180, nargs=2, kwnames=0x0000000000000000) at bltinmodule.c.h:465:20 [opt]
    frame #42: 0x00000001000cbb08 python.exe`cfunction_vectorcall_FASTCALL_KEYWORDS(func=0x0000000100b9f6b0, args=0x0000000100c6f180, nargsf=<unavailable>, kwnames=0x0000000000000000) at methodobject.c:443:24 [opt]
    frame #43: 0x0000000100087018 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100486930, callable=0x0000000100b9f6b0, args=0x0000000100c6f180, nargsf=9223372036854775810, kwnames=0x0000000000000000) at pycore_call.h:92:11 [opt]
    frame #44: 0x0000000100086fa0 python.exe`PyObject_Vectorcall(callable=<unavailable>, args=<unavailable>, nargsf=<unavailable>, kwnames=<unavailable>) at call.c:299:12 [opt]
    frame #45: 0x0000000100170d7c python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100486930, frame=0x0000000100c6f0d8, throwflag=<unavailable>) at ceval.c:4826:23 [opt]
    frame #46: 0x000000010016680d python.exe`_PyEval_EvalFrame(tstate=<unavailable>, frame=<unavailable>, throwflag=<unavailable>) at pycore_ceval.h:66:16 [opt]
    frame #47: 0x0000000100166726 python.exe`_PyEval_Vector(tstate=0x0000000100486930, func=0x0000000102c5eaf0, locals=0x0000000000000000, args=0x0000000102894ba8, argcount=2, kwnames=<unavailable>) at ceval.c:6468:24 [opt]
    frame #48: 0x0000000100087330 python.exe`_PyFunction_Vectorcall(func=0x0000000102c5eaf0, stack=0x0000000102894ba8, nargsf=<unavailable>, kwnames=0x0000000000000000) at call.c:0:45 [opt]
    frame #49: 0x0000000100086f3d python.exe`_PyVectorcall_Call(tstate=0x0000000100486930, func=(python.exe`_PyFunction_Vectorcall at call.c:385), callable=0x0000000102c5eaf0, tuple=0x0000000102894b90, kwargs=0x0000000000000000) at call.c:245:16 [opt]
    frame #50: 0x0000000100087148 python.exe`_PyObject_Call(tstate=0x0000000100486930, callable=0x0000000102c5eaf0, args=0x0000000102894b90, kwargs=0x0000000000000000) at call.c:328:16 [opt]
    frame #51: 0x0000000100087209 python.exe`PyObject_Call(callable=<unavailable>, args=<unavailable>, kwargs=<unavailable>) at call.c:355:12 [opt]
    frame #52: 0x00000001001d9f3d python.exe`pymain_run_module(modname=<unavailable>, set_argv0=<unavailable>) at main.c:300:14 [opt]
    frame #53: 0x00000001001d982d python.exe`pymain_run_python(exitcode=0x00007ffeefbff7dc) at main.c:595:21 [opt]
    frame #54: 0x00000001001d9718 python.exe`Py_RunMain at main.c:680:5 [opt]
    frame #55: 0x00000001001d99af python.exe`pymain_main(args=<unavailable>) at main.c:710:12 [opt]
    frame #56: 0x00000001001d99fb python.exe`Py_BytesMain(argc=<unavailable>, argv=<unavailable>) at main.c:734:12 [opt]
    frame #57: 0x0000000100001999 python.exe`main(argc=<unavailable>, argv=<unavailable>) at python.c:15:12 [opt]
    frame #58: 0x00007fff20417f3d libdyld.dylib`start + 1

Your environment

  • CPython versions tested on: 3.11 current head
  • Operating system and architecture: macOS

I put print statements in that show the file it crashes on is '/usr/local/lib/python3.11/site-packages/pytest_cov/plugin.py', but just running Python on that file doesn't reproduce the crash. I'll try to debug some more.

@JelleZijlstra JelleZijlstra added type-crash A hard crash of the interpreter, possibly with a core dump 3.11 only security fixes 3.12 bugs and security fixes labels May 10, 2022
@JelleZijlstra
Copy link
Member Author

JelleZijlstra commented May 10, 2022

Simple repro:

from ast import *
tree = Module(body=[
    Import(names=[alias(name='builtins', lineno=1, col_offset=0)], lineno=1, col_offset=0), 
    Import(names=[alias(name='traceback', lineno=0, col_offset=0)], lineno=0, col_offset=1)
], type_ignores=[])
compile(tree, "x", "exec", dont_inherit=True)
% ./python.exe wronglinenos.py 
Assertion failed: (i->i_end_lineno >= i->i_lineno), function write_location_info_long_form, file compile.c, line 7532.
zsh: abort      ./python.exe wronglinenos.py

@JelleZijlstra JelleZijlstra changed the title Compiler crash on pytest-cov Crash on AST with misordered linenos May 10, 2022
@kumaraditya303
Copy link
Contributor

944fffe is the first bad commit

commit 944fffee8916cb94321fa33cd3a43f4108717746
Author: Mark Shannon <mark@hotpy.org>
Date:   Thu Apr 21 16:10:37 2022 +0100

    GH-88116: Use a compact format to represent end line and column offsets. (GH-91666)
    
    * Stores all location info in linetable to conform to PEP 626.
    
    * Remove column table from code objects.
    
    * Remove end-line table from code objects.
    
    * Document new location table format

@iritkatriel
Copy link
Member

@markshannon

Looks like the ast defaults the end_lineno to None, which then becomes 0 in the compiler. Maybe the ast should default it be equal to lineno? Or should None become -1 in c?

@markshannon
Copy link
Member

Both, perhaps.
If no end_lineno is specified, then it should probably default to lineno
If the end_lineno is None then we should convert that to -1, as we do for lineno.

@iritkatriel
Copy link
Member

cc @pablogsal

@pablogsal
Copy link
Member

Both, perhaps.
If no end_lineno is specified, then it should probably default to lineno
If the end_lineno is None then we should convert that to -1, as we do for lineno.

That's what we do in some parts that deal with these information. I can prepare a PR

@pablogsal
Copy link
Member

The question is if we should manage this in the AST creation routines or in the compiler. I assume that the first is more resilient but what do others think?

@The-Compiler
Copy link
Contributor

FWIW, this seems to affect pytest's assertion rewriting as well. As soon as I'm running pytest against e.g.:

def test_foo():
    pass

with a Python 3.11.0b1 configured with --with-pydebug, I get:

collecting ... python: Python/compile.c:7531: write_location_info_long_form: Assertion `i->i_end_lineno >= i->i_lineno' failed.
Fatal Python error: Aborted

Current thread 0x00007f2555185740 (most recent call first):
  File ".../site-packages/_pytest/assertion/rewrite.py", line 361 in _rewrite_test
  File ".../site-packages/_pytest/assertion/rewrite.py", line 159 in exec_module
  ...

which is here:

https://github.com/pytest-dev/pytest/blob/c988e49af63a50f0704a46a93e73463666451af0/src/_pytest/assertion/rewrite.py#L361

and indeed i_end_lineno is 0:

(gdb) pp i
i = 
   autoderefcount="1",[
      i_opcode = <int> = {"100"}
      i_oparg = <int> = {"0"}
      i_target = <struct basicblock_*> = {"0x0"}
      i_except = <struct basicblock_*> = {"0x0"}
      i_lineno = <int> = {"1"}
      i_end_lineno = <int> = {"0"}
      i_col_offset = <int> = {"0"}
      i_end_col_offset = <int> = {"0"}
   ],<struct instr> = {"{...}"}

Turning pytest's assertion rewriting off with --assert=plain helps, and I can confirm it works fine before 944fffe.

Passing the end_lineno where we generate fake imports for the pytest assertion rewriting helpers seems to help:

diff --git i/src/_pytest/assertion/rewrite.py w/src/_pytest/assertion/rewrite.py
index 81096764e..5d7392dd4 100644
--- i/src/_pytest/assertion/rewrite.py
+++ w/src/_pytest/assertion/rewrite.py
@@ -727,7 +727,7 @@ def run(self, mod: ast.Module) -> None:
                 ast.alias("_pytest.assertion.rewrite", "@pytest_ar"),
             ]
         imports = [
-            ast.Import([alias], lineno=lineno, col_offset=0) for alias in aliases
+            ast.Import([alias], lineno=lineno, end_lineno=item.end_lineno, col_offset=0) for alias in aliases
         ]
         mod.body[pos:pos] = imports
 

cc @bluetech - though I suppose we don't necessarily need to fix this in pytest, hence I didn' t open an issue over there.

@The-Compiler
Copy link
Contributor

Another project affected in the wild seems to be flask:

$ python3.11 -X faulthandler -c "import flask; flask.Flask('test')"
python: Python/compile.c:7534: write_location_info_long_form: Assertion `i->i_end_lineno >= i->i_lineno' failed.
Fatal Python error: Aborted

Current thread 0x00007fd44da08740 (most recent call first):
  File ".../lib/python3.11/site-packages/werkzeug/routing.py", line 1073 in _compile_builder
  File ".../lib/python3.11/site-packages/werkzeug/routing.py", line 885 in compile
  File ".../lib/python3.11/site-packages/werkzeug/routing.py", line 805 in bind
  File ".../lib/python3.11/site-packages/werkzeug/routing.py", line 1546 in add
  File ".../lib/python3.11/site-packages/flask/app.py", line 1086 in add_url_rule
  File ".../lib/python3.11/site-packages/flask/scaffold.py", line 56 in wrapper_func
  File ".../lib/python3.11/site-packages/flask/app.py", line 511 in __init__
  File "<string>", line 1 in <module>

@pablogsal
Copy link
Member

I'm working on a patch, will have a draft PR ready soon

@markshannon
Copy link
Member

@pablogsal Do you have time to work on this, or should I fix it?

@pablogsal
Copy link
Member

@pablogsal Do you have time to work on this, or should I fix it?

I have opened a PR for this.

pablogsal added a commit to pablogsal/cpython that referenced this issue May 30, 2022
miss-islington pushed a commit to miss-islington/cpython that referenced this issue May 31, 2022
…can be compiled (pythonGH-93359)

(cherry picked from commit 705eaec)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
hroncok pushed a commit to fedora-python/cpython that referenced this issue May 31, 2022
…itions can be compiled

(cherry picked from commit 705eaec)
miss-islington added a commit that referenced this issue May 31, 2022
… compiled (GH-93359)

(cherry picked from commit 705eaec)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 1, 2022
…es (pythonGH-93398)

(cherry picked from commit 8a221a8)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
pablogsal added a commit that referenced this issue Jun 1, 2022
…-93398) (GH-93414)

(cherry picked from commit 8a221a8)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.11 only security fixes 3.12 bugs and security fixes type-crash A hard crash of the interpreter, possibly with a core dump
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants