__contains__ method behavior

[jbvsmo]

BPO	13667
Nosy	@birkenfeld, @rhettinger, @pitrou, @benjaminp, @alex, @jbvsmo

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-12-28.22:35:01.001>
created_at = <Date 2011-12-28.09:11:09.960>
labels = ['interpreter-core', 'type-feature']
title = '__contains__ method behavior'
updated_at = <Date 2011-12-28.22:35:01.000>
user = 'https://github.com/jbvsmo'

bugs.python.org fields:

activity = <Date 2011-12-28.22:35:01.000>
actor = 'benjamin.peterson'
assignee = 'none'
closed = True
closed_date = <Date 2011-12-28.22:35:01.001>
closer = 'benjamin.peterson'
components = ['Interpreter Core']
creation = <Date 2011-12-28.09:11:09.960>
creator = 'JBernardo'
dependencies = []
files = []
hgrepos = []
issue_num = 13667
keywords = []
message_count = 12.0
messages = ['150283', '150284', '150285', '150286', '150296', '150303', '150304', '150307', '150309', '150314', '150315', '150321']
nosy_count = 6.0
nosy_names = ['georg.brandl', 'rhettinger', 'pitrou', 'benjamin.peterson', 'alex', 'JBernardo']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue13667'
versions = ['Python 3.3']

[jbvsmo]

Hi, I'm working on a class which implements the __contains__ method but the way I would like it to work is by generating an object that will be evaluated later.

It'll return a custom object instead of True/False

class C:
    def __contains__(self, x):
        return "I will evaluate this thing later... Don't bother now"

but when I do:

>>> 1 in C()
True

It seems to evaluate the answer with bool!

Reading the docs (http://docs.python.org/py3k/reference/expressions.html#membership-test-details) It says:

"x in y is true if and only if y.__contains__(x) is true."

It looks like the docs doesn't match the code and the code is trying to mimic the behavior of lists/tuples where "x in y" is the same as

any(x is e or x == e for e in y)

and always yield True or False.

There is a reason why it is that way?

Thanks!

[birkenfeld]

"an object is true" is a short way of saying "bool(obj) is True". So the docs match the behavior.

Returning the actual object instead of True/False from the "in" operator is a feature request.

[jbvsmo]

@georg Brandl
Oh sorry, now I see... true != True

But still, why is that the default behavior? Shouldn't it use whatever the method returns?

[birkenfeld]

Well, usually what you want *is* a boolean indicating whether the element is in the collection or not.

Being able to overload "in", mostly for metaprogramming purposes, is a request that probably nobody thought of when implementing "in".

[benjaminp]

I think the idea has some merit. I think it should be well vetted on python-ideas, though. One thing that will certianly weigh against it is that implementation would not be trivial.

[jbvsmo]

I see that every other comparison operator (<, >, <=, >=, ==, !=) except for is work the way I expect and is able to return anything.

e.g.

>>> numpy.arange(5) < 3
array([ True,  True,  True, False, False], dtype=bool)

I didn't checked the code (and probably I'm talking nonsense), but seems like the in operator has an extra call to PyObject_IsTrue that maybe could be dropped?

Of course it can break code relying on x in y being True/False but it would only happen on customized classes.

Another option that won't break code is to add a different method to handle these cases. Something like "__contains_non_bool__", but that'd be a big api change.

[benjaminp]

2011/12/28 João Bernardo <report@bugs.python.org>:
>
> João Bernardo <jbvsmo@gmail.com> added the comment:
>
> I see that every other comparison operator (<, >, <=, >=, ==, !=) except for `is` work the way I expect and is able to return anything.
>
> e.g.
>
>>>> numpy.arange(5) < 3
> array([ True,  True,  True, False, False], dtype=bool)
>
> I didn't checked the code (and probably I'm talking nonsense), but seems like the `in` operator has an extra call to `PyObject_IsTrue` that maybe could be dropped?

I'm not sure what you're referring to, but I doubt that would do the job.

Of course it can break code relying on x in y being True/False but it would only happen on customized classes.

Another option that won't break code is to add a different method to handle these cases. Something like "contains_non_bool", but that'd be a big api change.

And completely hideous.

[jbvsmo]

Using my poor grep abilities I found that on Objects/typeobject.c
(I replaced some declarations/error checking from the code with ...)

static int
slot_sq_contains(PyObject *self, PyObject *value) {
    ...
    func = lookup_maybe(self, "__contains__", &contains_str);
    if (func != NULL) {
        ...
        res = PyObject_Call(func, args, NULL);
        ...
        if (res != NULL) {
            result = PyObject_IsTrue(res);
            Py_DECREF(res);
        }
    }
    else if (! PyErr_Occurred()) {
        /* Possible results: -1 and 1 */
        result = (int)_PySequence_IterSearch(self, value,
                                         PY_ITERSEARCH_CONTAINS);
    }
}

I don't know if I'm in the right place, but the function returns int and evaluates the result to 1 or 0 if contains is found.

I also don't know what SQSLOT means, but unlike the other operators (which are defined as TPSLOT), slot_sq_contains is a function returning "int" while slot_tp_richcompare returns "PyObject *".

Why is that defined that way?

[birkenfeld]

It's defined that way because it's a slot returning a bool, so it doesn't need to return anything except for 0 or 1.

Changing this to return a PyObject would mean that every extension module (i.e. module written in C) that defines a custom __contains__ would need to be adapted. That is the non-trivial implementation that Benjamin was talking about.

[rhettinger]

-1 on this proposal. It has everyone paying a price for a questionable feature that would benefit very few.

[alex]

For what it's worth I proposed this on -ideas a while ago, the sticking points were what does not in do (no one had an answer anyone was happy with for this), and do we need a way to override it from the other perspective (e.g. if I want to do SpecialObj() in [1, 2, 3], is there a way to do that?).

[jbvsmo]

The problem with not in is because it must evaluate the result. It's not just another operator like "==" and "!=".

Looks like we're suffering from premature optimization and now it would break a lot of code to make it good.

For my application, I created a different method to generate the object (Not as good as I wanted, but there's no option right now):

MyClass().has(1) instead of 1 in MyClass()

So, if no one comes up with a better idea, this issue should be closed.

Thanks