PyIter_Check returns false positive for objects of type instance

[behzadnouri]

BPO	24161
Nosy	@rhettinger
Files	spammodule.c doc_iter_check.diff: Doc patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-05-11.17:24:48.997>
created_at = <Date 2015-05-10.20:55:34.554>
labels = ['type-bug', 'docs']
title = 'PyIter_Check returns false positive for objects of type instance'
updated_at = <Date 2015-05-11.17:24:48.996>
user = 'https://bugs.python.org/behzadnouri'

bugs.python.org fields:

activity = <Date 2015-05-11.17:24:48.996>
actor = 'rhettinger'
assignee = 'docs@python'
closed = True
closed_date = <Date 2015-05-11.17:24:48.997>
closer = 'rhettinger'
components = ['Documentation']
creation = <Date 2015-05-10.20:55:34.554>
creator = 'behzad.nouri'
dependencies = []
files = ['39335', '39338']
hgrepos = []
issue_num = 24161
keywords = ['patch']
message_count = 5.0
messages = ['242866', '242870', '242874', '242878', '242904']
nosy_count = 4.0
nosy_names = ['rhettinger', 'docs@python', 'python-dev', 'behzad.nouri']
pr_nums = []
priority = 'low'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24161'
versions = ['Python 2.7']

[behzadnouri]

• python 2 only, not reproducible on python 3

Attached file makes an extension module which just returns PyIter_Check value on passed object.

Calling the function with an object of type "instance" returns true, even though the object is not iterator:

    >>> import spam
    >>> class Foo:
    ...     pass
    ... 
    >>> foo = Foo()
    >>> type(foo)
    <type 'instance'>
    >>> spam.isiter(foo)  # <<<< ?!
    1
    >>> next(foo)
    TypeError: instance has no next() method

[rhettinger]

The PyIter_Check() macro in Include/abstract.h does a quick test to see whether the tp_iternext slot is null or marked as not implemented. That works for builtin types but not for user defined classes (heap types).

Old-style instances, see Objects/classobject.c::instance_iternext(), all define iternext with code that attempts lookup and call to the next() method, and if not it is not found, raises the TypeError you are seeing.

The conflict is that PyIter_Check() aims to be a fast check of a static slot entry while instance_iternext() aims for a dynamic call-it-and-see-if-it-works check much like PyObject_HasAttr() does.

Since this code is very old (back to Python 2.2) and has been mostly harmless (as far as we know), one resolution would be to just document this as a known limitation of PyIter_Check(). Rather than using PyIter_Check(), extensions should just call next() on object and see whether it succeeds.

[behzadnouri]

That works for builtin types but not for user defined classes

Rather than using PyIter_Check(), extensions should just call next() on object and see whether it succeeds

but then, what would be the use case of PyIter_Check outside of python core?

[rhettinger]

but then, what would be the use case of PyIter_Check
outside of python core?

You could still use it anywhere. It will give a correct result in the cases of extension modules, builtin types, and new-style classes. It will give a false positive in the case of old-style classes. The latter case doesn't seem to be of much consequence (there is a still a TypeError raised when next() is called), so you just find out a bit later than you otherwise would (I believe that is why this is why we haven't gotten a bug report in the 13+ years this code has existed).

The feature is imperfect, incomplete and not as useful as it could be.
But this ship sailed a long time ago. It is far too late for redesign (and risking unintended breakage).

FWIW, PyIter_Check() is used several times in the Python core: sqlite, cPickle, and iter(). In those examples, there seem to be no adverse consequences for the false positive because we still get a TypeError downstream when the actual call is made to next().

[python-dev]

New changeset 0f7795edca65 by Raymond Hettinger in branch '2.7':
Issue bpo-24161: Document that PyIter_Check() returns false positives for old-style instances.
https://hg.python.org/cpython/rev/0f7795edca65