Document auto __ne__ generation; provide a use case for non-trivial __ne__

[terryjreedy]

BPO	4395
Nosy	@rhettinger, @terryjreedy, @mdickinson, @ncoghlan, @rbtcollins, @merwok, @cjerdonek, @berkerpeksag, @vadmium, @serhiy-storchaka, @antocuni
Files	default-ne-reflected-priority.patch default-ne-reflected-priority.v2.patch default-ne-reflected-priority.v3.patch: Resolve conflict default-ne-reflected-priority.v4.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2015-08-06.22:43:02.223>
created_at = <Date 2008-11-23.18:21:33.030>
labels = ['type-bug', 'docs']
title = 'Document auto __ne__ generation; provide a use case for non-trivial __ne__'
updated_at = <Date 2015-08-06.22:43:02.215>
user = 'https://github.com/terryjreedy'

bugs.python.org fields:

activity = <Date 2015-08-06.22:43:02.215>
actor = 'rbcollins'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2015-08-06.22:43:02.223>
closer = 'rbcollins'
components = ['Documentation']
creation = <Date 2008-11-23.18:21:33.030>
creator = 'terry.reedy'
dependencies = []
files = ['37671', '37859', '39954', '39958']
hgrepos = []
issue_num = 4395
keywords = ['patch']
message_count = 20.0
messages = ['76270', '76374', '89532', '89533', '89553', '89591', '89592', '181630', '233835', '234601', '234605', '234696', '234780', '236012', '246955', '246959', '246968', '246969', '248155', '248156']
nosy_count = 17.0
nosy_names = ['rhettinger', 'terry.reedy', 'mark.dickinson', 'ncoghlan', 'rbcollins', 'eric.araujo', 'Arfrever', 'medwards', 'cvrebert', 'chris.jerdonek', 'docs@python', 'python-dev', 'berker.peksag', 'martin.panter', 'serhiy.storchaka', 'franck', 'antocuni']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue4395'
versions = ['Python 3.4', 'Python 3.5', 'Python 3.6']

[terryjreedy]

3.0c3 doc (Basic customization) says
"There are no implied relationships among the comparison operators. The
truth of x==y does not imply that x!=y is false. Accordingly, when
defining __eq__(), one should also define __ne__() so that the operators
will behave as expected. "

In http://mail.python.org/pipermail/python-ideas/2008-October/002235.html
Guido says
"I should also note that part of George's proposal has already been
implemented: if you define __eq__, you get a complementary __ne__ for
free. However it doesn't work the other way around (defining __ne__
doesn't give you __eq__ for free), and there is no similar
relationship for the ordering operators."

And indeed, as Arnaud Delobelle posted on python-list
class A:
def __init__(self, x):
self.x = x
def __eq__(self, other):
return self.x == other.x

a, b, c = A(1), A(1), A(2)
print(a==b, b==c, c==a) # True, False, False
print(a!=b, b!=c, c!=a) # False, True, True

Suggested revision:
"There is one implied relationship among comparison operators: defining
__eq__ gives an automatic __ne__ (but not the other way). There is no
similar relationship for the order comparisons.

[medwards]

It would be really useful to explain, right in this section, why __ne__
is worth having. Something along these lines (based on the logic from
Python 2.x -- modify as necessary):

<doctext>

The values most commonly returned by the rich comparison methods are
True, False, and NotImplemented (which tells the Python interpreter to
try a different comparison strategy). However, it is quite legal and
often useful to return some other value, usually one which can be
coerced to True/False by bool().

For instance, if equality testing of instances of some class is
computationally expensive, that class's implementation of __eq__ may
return a "comparison object" whose __nonzero__ method calculates and
caches the actual boolean value. Subsequent references to this same
comparison object may be returned for subsequent, logically equivalent
comparisons; the expensive comparison takes place only once, when the
object is first used in a boolean context. This class's implementation
of __ne__ could return, not just "not (self == other)", but an object
whose __nonzero__ method returns "not (self == other)" -- potentially
delaying the expensive operation until its result is really tested for
boolean truth.

Python allows the programmer to define __ne__ separately from __eq__ for
this and similar reasons. It is up to the programmer to ensure that
bool(self != other) == (not bool(self == other)), if this is a desired
property. (One can even imagine situations in which it is appropriate
for neither (self == other) nor (self != other) to be true. For
instance, a mathematical theorem prover might contain values a, b, c,
... that are formally unknown, and raise an exception when a==b is used
in a boolean context, but allow comparison of M = (a==b) against N =
(a!=b).)

</doctext>

Now that I write this, I see a use for magic __logical_or__,
__logical_and__, and __logical_not__ methods, so that one can postpone
or even avoid the evaluation of expensive/indeterminate comparisons.
Consider the expression:
((a==b) and (c==d)) and ((a!=b) and (d==f))
If my class is designed such that a==b and a!=b cannot both be true,
then I can conclude that this expression is false without evaluating any
of the equality/inequality tests.

Is it too late to request these for Python 3.0?

[terryjreedy]

The current paragraph
"There are no implied relationships among the comparison operators. The
truth of x==y does not imply that x!=y is false. Accordingly, when
defining __eq__(), one should also define __ne__() so that the operators
will behave as expected. "
is false.

Please, let us replace it now, for 3.1 release, with the correct
"There is one implied relationship among comparison operators: defining
__eq__ gives an automatic __ne__ (but not the other way). There is no
similar relationship for the order comparisons."
without waiting for a more extensive rewrite.

[rhettinger]

One other thought: The __ne__ method follows automatically from __eq__
only if __ne__ isn't already defined in a superclass. So, if you're
inheriting from a builtin, it's best to override both.

[terryjreedy]

The situation appears to be at least slightly different from what Guido
stated. In 3.x, all classes subclass object, which has .__ne__, so if
that stopped inferred != behavior, it would never happen.

>>> class A:
	def __eq__(s,p): return 1

>>> id(object.__ne__)
10703216
>>> id(A.__ne__)
10703216

No new A.__ne__ added. But

>>> c,d=object(),object()
>>> c==d
False
>>> c!=d
True
>>> a,b = A(),A()
>>> a==b
1
>>> a!=b
False

So it seems that a!=b *is* evaluated as not a==b rather than as
a.__ne__(b). If so, my revised suggested replacement would be:

"There is one implied relationship among comparison operators: defining
__eq__ causes '!=' to be evaluated as 'not ==' (but not the other way).
There is no similar relationship for the order comparisons."

I am a bit puzzled though. In
ttp://svn.python.org/view/python/branches/py3k/Python/ceval.c?revision=73066&view=markup
I traced compare_op to cmp_outcome to (in object.c) PyOjbect_RichCompare
to do_richcompare to class specific tp_richcompare and I do not see the
special casing of eq. However, I am newbie at codebase.

[medwards]

The implementation you are looking for is in object_richcompare, in
http://svn.python.org/projects/python/branches/py3k/Objects/typeobject.c
. It would be most accurate to say something like:

The "object" base class, from which all user-defined classes

inherit, provides a single "rich comparison" method to which all of the
comparison operators (eq, __ne__, __lt__, __le__, __ge__, __gt__)
map. This method returns a non-trivial value (i. e., something other
than NotImplemented) in only two cases:

• When called as __eq__, if the two objects are identical, this method
  returns True. (If they are not identical, it returns NotImplemented so
  that the other object's implementation of __eq__ gets a chance to return
  True.)
• When called as __ne__, it calls the equivalent of "self == other";
  if this returns a non-trivial value X, then it returns !X (which is
  always either True or False).

[medwards]

It would also be useful to point out that there is a shortcut in the
interpreter itself (PyObject_RichCompareBool, in object.c) which checks
the equivalent of id(a) == id(b) and bypasses __eq__/ne if so.
Since not every call to __eq__ passes through this function, it's fairly
important that implementations of __eq__ return either True or
NotImplemented when id(a) == id(b). Ditto for extension modules;
anything that installs its own tp_richcompare should handle object
identity and __ne__ in substantially the same way, so that subclass
authors can rely on the documented behavior when overriding __eq__.

[mdickinson]

Issue bpo-17151 closed as a duplicate of this one.

[vadmium]

Here is a patch that documents the default object.__ne__() implementation. It also documents the subclass priority rules for the reflected comparison methods, which is raised in bpo-22052.

I have made some more tests to verify the relationships exists from __ne__ to __eq__, but no other relationships exist for the other methods. I will include it in my patch in bpo-21408 to avoid the patches conflicting with each other.

[ncoghlan]

While Martin's patch doesn't cover all the vagaries of comparison operations discussed above, it fixes the outright error, and provides an appropriate cross-reference to functools.total_ordering.

[vadmium]

The reference to @functools.total_ordering was actually already there; I just moved it into the paragraph about relationships between the operators. I should also point out that my description of the default __ne__() assumes that bpo-21408 is resolved; the current behaviour is slightly different.

If you think something else could be added to the patch, I’m happy to try and add it. Perhaps the default object.__eq__() behaviour?

[vadmium]

Adding a new patch that just fixes the typo error in the first patch

[serhiy-storchaka]

See also bpo-23326.

[vadmium]

bpo-21408 has been committed to 3.4 and 3.5 branches, so my patch can now be considered to document the newly fixed behaviour.

[vadmium]

Nick seemed to approve of this, so perhaps it is ready to commit? The new patch just resolves a minor conflict with the current code.

[serhiy-storchaka]

Added comments on Rietveld.

[vadmium]

This updated patch adds the clarification about NotImplemented.

[serhiy-storchaka]

LGTM.

[python-dev]

New changeset f5069e6e4229 by Robert Collins in branch '3.4':
Issue bpo-4395: Better testing and documentation of binary operators.
https://hg.python.org/cpython/rev/f5069e6e4229

New changeset b9a0165a3de8 by Robert Collins in branch '3.5':
Issue bpo-4395: Better testing and documentation of binary operators.
https://hg.python.org/cpython/rev/b9a0165a3de8

New changeset e56893df8e76 by Robert Collins in branch 'default':
Issue bpo-4395: Better testing and documentation of binary operators.
https://hg.python.org/cpython/rev/e56893df8e76

[rbtcollins]

Thanks for the patch; applied to 3.4 and up.