Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PyEval_EvalCodeEx() can no longer be called with code which has (CO_NEWLOCALS | CO_OPTIMIZED) flags #87707

vstinner opened this issue Mar 18, 2021 · 4 comments
3.10 only security fixes topic-C-API


Copy link

BPO 43541
Nosy @vstinner, @markshannon, @pablogsal
  • bpo-43541: Fix PyEval_EvalCodeEx() regression #24918
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-03-18.13:52:01.583>
    created_at = <Date 2021-03-18.11:11:13.990>
    labels = ['expert-C-API', '3.10']
    title = 'PyEval_EvalCodeEx() can no longer be called with code which has (CO_NEWLOCALS | CO_OPTIMIZED) flags'
    updated_at = <Date 2021-03-18.13:52:01.583>
    user = '' fields:

    activity = <Date 2021-03-18.13:52:01.583>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-03-18.13:52:01.583>
    closer = 'vstinner'
    components = ['C API']
    creation = <Date 2021-03-18.11:11:13.990>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 43541
    keywords = ['patch']
    message_count = 4.0
    messages = ['389009', '389011', '389013', '389015']
    nosy_count = 3.0
    nosy_names = ['vstinner', 'Mark.Shannon', 'pablogsal']
    pr_nums = ['24918']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = ''
    versions = ['Python 3.10']

    Copy link
    Member Author

    Cython generates a __Pyx_PyFunction_FastCallDict() function which calls PyEval_EvalCodeEx(). With Python 3.9, it worked well. With Python 3.10 in debug mode, it fails with an assertion error:

    python3.10: Python/ceval.c:5148: PyEval_EvalCodeEx: Assertion `(((PyCodeObject *)_co)->co_flags & (CO_NEWLOCALS | CO_OPTIMIZED)) == 0' failed.

    With Python 3.10 in release mode, it does crash.

    Context of the failed assertion:

    • Assertion added recently to CPython 3.10 by 0332e56
    • The code object flags = (CO_NEWLOCALS | CO_OPTIMIZED | CO_NOFREE)
    • Code co_argcount = 2
    • Code co_kwonlyargcount = 0
    • Cython __Pyx_PyFunction_FastCallDict() called with: nargs=1 and kwargs=NULL

    See the Cython issue to a reproducer: cython/cython#4025 (comment)

    In Python 3.9, _PyFunction_Vectorcall() has the following fast-path:

        if (co->co_kwonlyargcount == 0 && nkwargs == 0 &&
            (co->co_flags & ~PyCF_MASK) == (CO_OPTIMIZED | CO_NEWLOCALS | CO_NOFREE))
            if (argdefs == NULL && co->co_argcount == nargs) {
                return function_code_fastcall(tstate, co, stack, nargs, globals);
            else if (nargs == 0 && argdefs != NULL
                     && co->co_argcount == PyTuple_GET_SIZE(argdefs)) {
                /* function called with no arguments, but all parameters have
                   a default value: use default values as arguments .*/
                stack = _PyTuple_ITEMS(argdefs);
                return function_code_fastcall(tstate, co,
                                              stack, PyTuple_GET_SIZE(argdefs),

    When the bug occurs, __Pyx_PyFunction_FastCallDict() doesn't take the fast-path because nargs < co_argcount (1 < 2).

    In Python 3.10, _PyFunction_Vectorcall() is very different:

        if (((PyCodeObject *)f->fc_code)->co_flags & CO_OPTIMIZED) {
            return _PyEval_Vector(tstate, f, NULL, stack, nargs, kwnames);
        else {
            return _PyEval_Vector(tstate, f, f->fc_globals, stack, nargs, kwnames);

    PyEval_EvalCodeEx() must not crash if the code object has (CO_NEWLOCALS | CO_OPTIMIZED | CO_NOFREE) flags.

    @vstinner vstinner added 3.10 only security fixes topic-C-API labels Mar 18, 2021
    Copy link
    Member Author

    If I the comment the following assertion in PyEval_EvalCodeEx():

    assert ((((PyCodeObject *)_co)->co_flags & (CO_NEWLOCALS | CO_OPTIMIZED)) == 0);

    The garbage collector fails while visiting builtins of the function with func_traverse(), at line:


    I guess that the problem is that f->func_builtins doesn't hold a strong reference to builtins.


    Modules/gcmodule.c:113: gc_decref: Assertion "gc_get_refs(g) > 0" failed: refcount is too small
    Enable tracemalloc to get the memory block allocation traceback

    object address : 0x7fffea69d650
    object refcount : 2265
    object type : 0x870e80
    object type name: dict
    object repr : {'name': 'builtins', 'doc': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the nil' object; Ellipsis represents ...' in slices.", 'package': '', 'loader': <class '_frozen_importlib.BuiltinImporter'>, 'spec': ModuleSpec(name='builtins', loader=<class '_frozen_importlib.BuiltinImporter'>, origin='built-in'), 'build_class': , 'import': , 'abs': , 'all': , 'any': , 'ascii': , ...}

    Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
    Python runtime state: initialized

    Current thread 0x00007ffff7c20740 (most recent call first):
    File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/", line 339 in __init__
    File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/", line 1953 in __init__
    File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/", line 2066 in __init__
    File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/", line 1549 in declare_c_class
    File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/", line 4782 in analyse_declarations
    File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/", line 431 in analyse_declarations
    File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/", line 124 in analyse_declarations
    File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/", line 1608 in visit_ModuleNode
    File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/", line 1598 in __call__
    File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/", line 335 in run
    File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/", line 355 in run_pipeline
    File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/", line 515 in run_pipeline
    File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/", line 727 in compile_single
    File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Build/", line 1208 in cythonize_one
    File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Build/", line 1102 in cythonize
    File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Build/", line 97 in cython_compile
    File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Build/", line 223 in main
    File "/home/vstinner/dev/numpy/env/bin/cythonize", line 33 in <module>

    Extension modules: Cython.Plex.Actions, Cython.Plex.Scanners, Cython.Compiler.Scanning, Cython.Tempita._tempita, Cython.Compiler.Visitor, Cython.Compiler.FlowControl (total: 6)

    Program received signal SIGABRT, Aborted.

    Copy link
    Member Author

    In commit 46496f9, I modified _PyEval_BuiltinsFromGlobals() to return a borrowed reference rather than a strong reference. It seems like I forgot to remove a Py_DECREF() in PyEval_EvalCodeEx()!

    I'm working on a fix.

    Copy link
    Member Author

    New changeset fc980e0 by Victor Stinner in branch 'master':
    bpo-43541: Fix PyEval_EvalCodeEx() regression (GH-24918)

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    3.10 only security fixes topic-C-API
    None yet

    No branches or pull requests

    1 participant