Calling "functions" used to implement generators/comps easily cause crash

[mr-nfamous]

BPO	36956
Nosy	@rhettinger, @mr-nfamous, @websurfer5, @iritkatriel

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-05-18.13:53:21.143>
labels = ['3.10', '3.9', 'type-crash', '3.11']
title = 'Calling "functions" used to implement generators/comps easily cause crash'
updated_at = <Date 2021-09-19.21:54:42.882>
user = 'https://github.com/mr-nfamous'

bugs.python.org fields:

activity = <Date 2021-09-19.21:54:42.882>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2019-05-18.13:53:21.143>
creator = 'bup'
dependencies = []
files = []
hgrepos = []
issue_num = 36956
keywords = []
message_count = 5.0
messages = ['342797', '346807', '346808', '346809', '402171']
nosy_count = 4.0
nosy_names = ['rhettinger', 'bup', 'Jeffrey.Kintscher', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue36956'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-81137: deprecate assignment of code object to a function of a mismatched type #111823

[mr-nfamous]

As far as I know, generators, set comprehensions, list comprehensions, and dict comprehensions, (along with their asynchronous variants) are implemented by first calling the GET_(A)ITER opcode and then building and calling a function that acepts the resulting iterator as its sole argument.

Assigning the code object used to make that function (or using it in the types.FunctionType constructor) and then calling it with a non-iterator argument will obviously cause a crash since the FOR_ITER opcode rightly expects that it will never have to deal with non-iterators and calls tp_iternext without checking if it exists.

The 4-liner demonstrates the crash:

if 1:
  fn = lambda: None
  gi = (i for i in ())
  fn.__code__ = gi.gi_code
  [*fn("abc")]

[websurfer5]

What is the preferred behavior? Raise an exception of some kind?

[websurfer5]

Specifically, the crash occurs in Python/ceval.c function _PyEval_EvalFrameDefault() in the TARGET(FOR_ITER) case at line 3198 (master branch):

PyObject *next = (*iter->ob_type->tp_iternext)(iter);

It segfaults because tp_iternext is NULL.

[rhettinger]

ISTM this report is close to that covered by the traditional stance that word code hacks aren't protected from crashes because trying to do can be complicated, can be expensive, and isn't always possible.

I'm not really sure where this falls on the grey scale with "normal, everyday python shouldn't crash" on one end and "with ctypes or code hacks, all bets are off" on the other end.

Offhand, the posted snippet seems closer to "messing with the internals" but perhaps small cheap check can be added when the __code__ variable is updated.

[iritkatriel]

Reproduced on 3.11.