C assertion error from the runtime while expecting a SyntaxError

[yilei]

Crash report

In a --with-pydebug build, run the following code:

$ cat ~/tmp/t.py
import ast
ast.parse("""
func(
    a=["unclosed], # Need a quote in this comment: "
    b=2,
)
""")
$ ./python ~/tmp/t.py

Error messages

python: Objects/call.c:324: _PyObject_Call: Assertion `!_PyErr_Occurred(tstate)' failed.
fish: Job 1, './python ~/tmp/t.py' terminated by signal SIGABRT (Abort)

(How do I obtain a core dump?)

I think the error happens inside this call:

cpython/Parser/pegen_errors.c

Line 175 in 5c19050

raise_unclosed_parentheses_error(p);

What happened is that the _PyTokenizer_Get(p->tok, &new_token) call earlier also sets an error here.

Your environment

• CPython versions tested on: main branch at 5c19050, but this also happens since Python 3.10
• Operating system and architecture: Linux x86_64

Linked PRs

• gh-100050: Fix an assertion error when raising unclosed parenthesis errors in the tokenizer #100065
• [3.11] gh-100050: Fix an assertion error when raising unclosed parenthesis errors in the tokenizer (GH-100065) #100067
• [3.10] gh-100050: Fix an assertion error when raising unclosed parenthesis errors in the tokenizer (GH-100065) #100073

[yilei]

Here is a traceback from lldb:

* thread #1, name = 'python', stop reason = signal SIGABRT
  * frame #0: 0x00007ffff7c8957c libc.so.6`__pthread_kill_implementation(threadid=<unavailable>, signo=6, no_tid=<unavailable>) at pthread_kill.c:44:76
    frame #1: 0x00007ffff7c3da02 libc.so.6`__GI_raise(sig=6) at raise.c:26:13
    frame #2: 0x00007ffff7c28469 libc.so.6`__GI_abort at abort.c:79:7
    frame #3: 0x00007ffff7c28395 libc.so.6`__assert_fail_base.cold at assert.c:92:3
    frame #4: 0x00007ffff7c36ab2 libc.so.6`__GI___assert_fail(assertion=<unavailable>, file=<unavailable>, line=<unavailable>, function=<unavailable>) at assert.c:101:3
    frame #5: 0x00005555556dda90 python`_PyObject_Call(tstate=0x0000555555be3ed0, callable=0x0000555555a5de80, args=0x00007ffff774f2a0, kwargs=0x0000000000000000) at call.c:324:5
    frame #6: 0x00005555556ddb2a python`PyObject_Call(callable=0x0000555555a5de80, args=0x00007ffff774f2a0, kwargs=0x0000000000000000) at call.c:357:12
    frame #7: 0x00005555557ea3cf python`PyErr_SetFromErrnoWithFilenameObjects(exc=0x0000555555a5de80, filenameObject=0x00007ffff77d2200, filenameObject2=0x0000000000000000) at errors.c:821:13
    frame #8: 0x00005555557ea4b8 python`PyErr_SetFromErrnoWithFilenameObject(exc=<unavailable>, filenameObject=0x00007ffff77d2200) at errors.c:732:12
    frame #9: 0x00005555558376a4 python`_Py_fopen_obj(path=0x00007ffff77d2200, mode="rb") at fileutils.c:1708:9
    frame #10: 0x00005555557eb462 python`_PyErr_ProgramDecodedTextObject(filename=<unavailable>, lineno=2, encoding="utf-8") at errors.c:1812:16
    frame #11: 0x00005555556518cc python`_PyPegen_raise_error_known_location(p=0x00007ffff77359c0, errtype=0x0000555555a5b3e0, lineno=2, col_offset=9, end_lineno=2, end_col_offset=0, errmsg="'%c' was never closed", va=0x00007fffffffd738) at pegen_errors.c:322:22
    frame #12: 0x0000555555651b04 python`RAISE_ERROR_KNOWN_LOCATION(p=<unavailable>, errtype=<unavailable>, lineno=<unavailable>, col_offset=<unavailable>, end_lineno=<unavailable>, end_col_offset=<unavailable>, errmsg="'%c' was never closed") at pegen.h:174:5
    frame #13: 0x0000555555651b5c python`raise_unclosed_parentheses_error(p=<unavailable>) at pegen_errors.c:53:8
    frame #14: 0x0000555555651c3c python`_PyPegen_tokenize_full_source_to_check_for_errors(p=0x00007ffff77359c0) at pegen_errors.c:175:25
    frame #15: 0x00005555556520be python`_Pypegen_set_syntax_error(p=0x00007ffff77359c0, last_token=0x00007ffff768f240) at pegen_errors.c:408:13
    frame #16: 0x00005555556511a4 python`_PyPegen_run_parser(p=0x00007ffff77359c0) at pegen.c:847:9
    frame #17: 0x00005555556513dd python`_PyPegen_run_parser_from_string(str="\na = func(a=[\"test], # Comment\"\nb=3,\n)", start_rule=257, filename_ob=0x00007ffff77d2200, flags=0x00007fffffffd9c8, arena=0x00007ffff779e4a0) at pegen.c:941:14
    frame #18: 0x00005555556bd4a4 python`_PyParser_ASTFromString(str="\na = func(a=[\"test], # Comment\"\nb=3,\n)", filename=0x00007ffff77d2200, mode=257, flags=0x00007fffffffd9c8, arena=0x00007ffff779e4a0) at peg_api.c:14:21
    frame #19: 0x000055555581c5f2 python`Py_CompileStringObject(str="\na = func(a=[\"test], # Comment\"\nb=3,\n)", filename=0x00007ffff77d2200, start=257, flags=0x00007fffffffd9c8, optimize=-1) at pythonrun.c:1781:11
    frame #20: 0x00005555557afe4c python`builtin_compile_impl(module=0x00007ffff7ba8670, source=0x00007ffff7795dc0, filename=0x00007ffff77d2200, mode="exec", flags=1024, dont_inherit=0, optimize=-1, feature_version=-1) at bltinmodule.c:828:14
    frame #21: 0x00005555557b00c3 python`builtin_compile(module=0x00007ffff7ba8670, args=<unavailable>, nargs=4, kwnames=0x00007ffff779e9f0) at bltinmodule.c.h:383:20
    frame #22: 0x0000555555726356 python`cfunction_vectorcall_FASTCALL_KEYWORDS(func=0x00007ffff7ba8bb0, args=0x00007ffff7fb5118, nargsf=<unavailable>, kwnames=0x00007ffff779e9f0) at methodobject.c:438:24
    frame #23: 0x00005555556ddf0b python`_PyObject_VectorcallTstate(tstate=0x0000555555be3ed0, callable=0x00007ffff7ba8bb0, args=0x00007ffff7fb5118, nargsf=9223372036854775812, kwnames=0x00007ffff779e9f0) at pycore_call.h:92:11
    frame #24: 0x00005555556ddfd7 python`PyObject_Vectorcall(callable=<unavailable>, args=<unavailable>, nargsf=<unavailable>, kwnames=<unavailable>) at call.c:301:12
    frame #25: 0x00005555557c60b3 python`_PyEval_EvalFrameDefault(tstate=0x0000555555be3ed0, frame=0x00007ffff7fb5080, throwflag=0) at generated_cases.c.h:2965:23
    frame #26: 0x00005555557cb722 python`_PyEval_EvalFrame(tstate=<unavailable>, frame=<unavailable>, throwflag=<unavailable>) at pycore_ceval.h:88:16
    frame #27: 0x00005555557cb84c python`_PyEval_Vector(tstate=0x0000555555be3ed0, func=0x00007ffff7703c50, locals=0x00007ffff7743e00, args=0x0000000000000000, argcount=0, kwnames=0x0000000000000000) at ceval.c:2076:12
    frame #28: 0x00005555557cb8fb python`PyEval_EvalCode(co=0x00007ffff7769ea0, globals=0x00007ffff7743e00, locals=0x00007ffff7743e00) at ceval.c:585:21
    frame #29: 0x00005555558193a4 python`run_eval_code_obj(tstate=0x0000555555be3ed0, co=0x00007ffff7769ea0, globals=0x00007ffff7743e00, locals=0x00007ffff7743e00) at pythonrun.c:1702:9
    frame #30: 0x0000555555819469 python`run_mod(mod=0x0000555555c724b8, filename=<unavailable>, globals=0x00007ffff7743e00, locals=0x00007ffff7743e00, flags=0x00007fffffffded8, arena=0x00007ffff774bb80) at pythonrun.c:1723:19
    frame #31: 0x000055555581c49a python`PyRun_StringFlags(str="import ast;ast.parse('\\na = func(a=[\"test], # Comment\"\\nb=3,\\n)')\n", start=257, globals=0x00007ffff7743e00, locals=0x00007ffff7743e00, flags=0x00007fffffffded8) at pythonrun.c:1592:15
    frame #32: 0x000055555581c4f8 python`PyRun_SimpleStringFlags(command="import ast;ast.parse('\\na = func(a=[\"test], # Comment\"\\nb=3,\\n)')\n", flags=0x00007fffffffded8) at pythonrun.c:486:9
    frame #33: 0x0000555555839e27 python`pymain_run_command(command=<unavailable>) at main.c:255:11
    frame #34: 0x000055555583a90a python`pymain_run_python(exitcode=0x00007fffffffdf7c) at main.c:601:21
    frame #35: 0x000055555583ac22 python`Py_RunMain at main.c:689:5
    frame #36: 0x000055555583ac76 python`pymain_main(args=0x00007fffffffdfc0) at main.c:719:12
    frame #37: 0x000055555583acf6 python`Py_BytesMain(argc=<unavailable>, argv=<unavailable>) at main.c:743:12
    frame #38: 0x000055555564f742 python`main(argc=<unavailable>, argv=<unavailable>) at python.c:15:12
    frame #39: 0x00007ffff7c2920a libc.so.6`__libc_start_call_main(main=(python`main at python.c:14:1), argc=3, argv=0x00007fffffffe108) at libc_start_call_main.h:58:16
    frame #40: 0x00007ffff7c292bc libc.so.6`__libc_start_main_impl(main=(python`main at python.c:14:1), argc=3, argv=0x00007fffffffe108, init=0x00007ffff7ffd020, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007fffffffe0f8) at libc-start.c:389:3
    frame #41: 0x000055555564f671 python`_start + 33

[pablogsal]

Thanks a lot for the report @yilei! :)