Doc: remove errors about mixed-type comparisons.

[terryjreedy]

BPO	12067
Nosy	@rhettinger, @terryjreedy, @mdickinson, @benjaminp, @ezio-melotti, @stevendaprano, @bitdancer, @cjerdonek, @berkerpeksag, @vadmium, @andy-maier, @csabella, @humbdrag
PRs	gh-56276: Add tests to test_compare #3199 bpo-12067: Add tests to test_compare #30624 bpo-12067: Add tests to test_compare #30625 bpo-12067: Add tests to test_compare #30651 bpo-12067: Add tests to test_compare #30667
Dependencies	bpo-29321: Wrong documentation (Language Ref) for unicode and str comparison
Files	issue12067-expressions.diff issue12067-expressions_v2.diff: Minor wording changes. issue12067-expressions_v3.diff: Andy's version of the patch for the 3.5 tip issue12067-expressions_v4.diff: Updates (see message), for v3.5 tip issue12067-expressions-py34_v5.diff: v5 of the patch, targeting 3.4. issue12067-expressions-py34_v6.diff: v6 (same as v5, just now created properly), targeting 3.4. issue12067-expressions-py34_v7.diff: v7 (same as v5, just now created properly), targeting 3.4. issue12067-expressions-py34_v8.diff: v8 (improved doc and added tests), targeting 3.4. try_eq.py: Test program for py34 for equality of collections try_eq.out: Output of try_eq.py with cpython 3.4.1 issue12067-expressions-py34_v9.diff: v9 of the patch, targeting 3.4. issue12067-expressions-py34_v10.diff: v10 of the patch, targeting 3.4 issue12067-expressions-py34_delta-v9-v10.diff: delta between v9 and v10 of the patch issue12067-expressions-py34_v11.diff: v11 of the patch, targeting 3.4 issue12067-expressions-py34_v12.diff: v12 of the patch, targeting 3.4 issue12067-expressions-py3.5_v13.diff issue12067-expressions-py3.5_v14.diff: v14 of the patch, targeting 3.5 (default) issue12067-expressions-py3.6_v15.diff issue12067-expressions-py3.6_v16.diff expressions-py2.7.diff expressions-py2.7_v17.diff expressions-py3.7_v17.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2011-05-12.21:59:33.279>
labels = ['easy', 'type-feature', 'tests', 'docs']
title = 'Doc: remove errors about mixed-type comparisons.'
updated_at = <Date 2022-01-26.02:09:04.532>
user = 'https://github.com/terryjreedy'

bugs.python.org fields:

activity = <Date 2022-01-26.02:09:04.532>
actor = 'terry.reedy'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Tests']
creation = <Date 2011-05-12.21:59:33.279>
creator = 'terry.reedy'
dependencies = ['29321']
files = ['27256', '27442', '35843', '35845', '35850', '35851', '35853', '35924', '35938', '35939', '35971', '36896', '36897', '36920', '36978', '38183', '38303', '39970', '40052', '46372', '46398', '46399']
hgrepos = []
issue_num = 12067
keywords = ['patch', 'easy']
message_count = 53.0
messages = ['135873', '135875', '135890', '148760', '148774', '170887', '170936', '170952', '170953', '170966', '171012', '172148', '183786', '222204', '222209', '222254', '222257', '222270', '222275', '222276', '222278', '222310', '222442', '222747', '222772', '222938', '223217', '226496', '226522', '226529', '228769', '229217', '229229', '229236', '229240', '229245', '229327', '229328', '229721', '236269', '237069', '247079', '247088', '247559', '251405', '251406', '285948', '286128', '286439', '295145', '295222', '300787', '411699']
nosy_count = 17.0
nosy_names = ['rhettinger', 'terry.reedy', 'mark.dickinson', 'benjamin.peterson', 'ezio.melotti', 'steven.daprano', 'r.david.murray', 'cvrebert', 'chris.jerdonek', 'docs@python', 'python-dev', 'berker.peksag', 'martin.panter', 'mikehoy', 'andymaier', 'cheryl.sabella', 'humbdrag']
pr_nums = ['3199', '30624', '30625', '30651', '30667']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue12067'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6']

Linked PRs

• [3.11] gh-56276: Add tests to test_compare (GH-3199) #104694

[terryjreedy]

Current 3.2 doc, 5.9. Comparisons, has this paragraph about mixed-type comparisons.

"The operators <, >, ==, >=, <=, and != compare the values of two objects. The objects need not have the same type. If both are numbers, they are converted to a common type. Otherwise, the == and != operators *always* consider objects of different types to be unequal, while the <, >, >= and <= operators raise a TypeError when comparing objects of different types that do not implement these operators for the given pair of types. You can control comparison behavior of objects of non-built-in types by defining rich comparison methods like __gt__(), described in section Basic customization."

Sentence 3: "If both are numbers, they are converted to a common type." I suspect it would be more true to say 'common internal type' as I would not think it a language requirement to produce Python objects.

In any case, I think it is only true for built-in number types, and I do not see that qualification anywhere previously.

That aside, it does not appear to be true for Decimals and Fractions in 2.7.1.

Sentence 4: first clause is only true for built-in types. That qualification is not obvious to everyone, as evidenced by a current python-list sub thread.

For 2.7, which has a different continuation, I suggest adding 'built-in' before 'objects of'.
For 3.2/3, I suggest deleting '*always*' and adding a comma after 'TypeError' so that the 'when' condition applies to equality comparisons also.

After discussion about same-type comparisons, there is another paragraph about mixed-type comparison:

"Comparison of objects of the differing types depends on whether either of the types provide explicit support for the comparison. Most numeric types can be compared with one another, but comparisons of float and Decimal are not supported to avoid the inevitable confusion arising from representation issues such as float('1.1') being inexactly represented and therefore not exactly equal to Decimal('1.1') which is. When cross-type comparison is not supported, the comparison method returns NotImplemented. This can create the illusion of non-transitivity between supported cross-type comparisons and unsupported comparisons. For example, Decimal(2) == 2 and 2 == float(2) but Decimal(2) != float(2)."

I suggest deleting this entirely. The first sentence and first clause of the second repeat what was said above. The rest is obsolete as float/decimal comparisons *are* implemented in 2.7.1 and 3.2.0.

[ezio-melotti]

Can you provide a patch?

[mdickinson]

[Docs]
"If both are numbers, they are converted to a common type."

[Terry]
"In any case, I think it is only true for built-in number types,"

It's not even true for built-in number types. When comparing an int with a float, it's definitely *not* the case that the int is converted to a float and the floats compared. And that's for good reason: the int -> float conversion is lossy for large integers, so if int <-> float comparisons just converted the int to a float before comparing, we'd have (for example):

>> 10**16 == 1e16 == 10**16 + 1

leading to broken transitivity of equality, and strange dict and set behaviour.

So int <-> float comparisons do a complicated dance under the hood to compare the exact numerical values of the two objects and produce the correct result.

I'm not sure what the intent of the original sentence was, or how to reword it. The key point is simply that it *is* possible to compare an int with a float, and that the result is sensible, based on numeric values.

[ezio-melotti]

Would it be ok to state that:

1. <, >, ==, >=, <=, and != compare the values of two objects;
2. the two objects don't necessarily have to be of the same type;
3. with == and !=, objects of different types compare unequal, unless they define a specific __eq__ and/or __ne__;
4. with <, >, <=, and >=, the comparison of objects of different types raises a TypeError, unless they define specific __lt__, __gt__, __le__, and __ge__;
5. some built-in types define these operations, so it's possible to compare e.g. int and floats;

This should summarize the possible behaviors. There's no reason IMHO to expose implementation details and to special case built-in types (unless their comparison is actually different and doesn't depend on __eq__, __ne__, etc.).

[terryjreedy]

In Python 3, where all classes inherit from object, the default rules are, by experiment, (which someone can verify from the code) simpler than you stated.

3. By default, == and /= compare identities.
4. By default, order comparisons raise TypeError.
   ob <= ob raises even though ob == ob because ob is ob.

I am not sure of the method look-up rules for rich comparisons, but perhaps the following are true:

3. with == and !=, an object is equal to itself and different objects (a is not b) compare unequal, unless the class of the first define a specific __eq__ and __ne__;

4. with <, >, <=, and >=, comparison raises a TypeError, unless the class of the first object defines specific __lt__, __gt__, __le__, and __ge__, or the class of the second defines the reflected method (ge reflects __lt__, etcetera);

What is not clear to me is whether the reflected method is called if the first raises TypeError. The special method names doc (reference 3.3) says "A rich comparison method may return the singleton NotImplemented if it does not implement the operation for a given pair of arguments.
...
There are no swapped-argument versions of these methods (to be used when the left argument does not support the operation but the right argument does); rather, __lt__() and __gt__() are each other’s reflection, __le__() and __ge__() are each other’s reflection, and __eq__() and __ne__() are their own reflection."

Does 'not supported' mean 'raises TypeError', 'returns NotImplemented', or both? If the last, I don't really understand the reason for NotImplemented versus TypeError. That point should be clarified in 3.3 also. And 3.3 should be referenced in the comparisons section.

I think point 5 should say a bit more: builtin numbers compare as expected, even when of different types; builtin sequences compare lexicographically.

[mikehoy]

http://bugs.python.org/issue15997 is this issue related to what Terry has mentioned:

Does 'not supported' mean 'raises TypeError', 'returns NotImplemented', or both? If the last, I don't really understand the reason for NotImplemented versus TypeError. That point should be clarified in 3.3 also. And 3.3 should be referenced in the comparisons section.

I am working on this patch and need confirmation as to whether or not this has to be included in my patch. I'm not clear on it so I may just pass on making a patch if it is required for this issue.

[terryjreedy]

After further thought: This section is about the syntax operators, not the special methods. The syntax operators never evaluate to NotImplemented as they (apparently) interpret its return from a special method the same as a raising of TypeError, and always raise TypeError when neither the op or its reflection is supported. So there should be no mention of NotImplemented here. Just a reference to 3.3. bpo-15997 is related to my 'wonder' but not directly relevant to a patch for this. Please submit a draft patch when you have one.

I determined that 'raise TypeError' and 'return NotImplemented' both result in the call of the reflected method, at least for a couple of cases. (And same seems true for arithmetic ops too.)

class C():
    def __ge__(self, other):
        # print("in C.__ge__", end='')
        return True
    def __add__(self, other):
        return 44
    __radd__ = __add__

class O():
    def __le__(self, other):
        # print ("in O.__le__")
        return NotImplemented
    def __add__(self, other):
        return NotImplemented
    
c = C()
o = O()
ob = object() 
print(c >= o, o <= c, ob <= c)
# True True True
# print(ob <= ob) # raises TypeError
print(c + o, o + c, ob + c)
# 44 44 44
# print(ob + ob)  # raises TypeError
# print(ob >= o)  # with O.__le__ print uncommented
# in O.__le__  # so non-implemented reflected o <= ob *is* called
# TypeError: unorderable types: object() >= O()

[mikehoy]

I've attempted to incorporate both Terry's and Ezio's suggestions. Here is a patch to get started with. There is a section that has been deleted. Patch uploaded.

[cjerdonek]

Some minor comments:

-The operators <, >, ==, >=, <=, and != compare the
+<, >, ==, >=, <=, and != compare the values of two

I think it reads better to start a sentence (and in this case a paragraph) with a word rather than a symbol.

-values of two objects. The objects need not have the same type. If both are
+objects. The two objects don't necessarily have to be of the same type. With

The replacement sentence seems wordier to me.

+(:meth:`__ge__()` reflects :meth:`__lt__()`, etcetera). Builtin numbers compare
+as expected, even when of different types. Builtin sequences compare

"Built-in" is hyphenated in the docs. See, for example, here:

http://docs.python.org/dev/library/functions.html

[mdickinson]

I determined that 'raise TypeError' and 'return NotImplemented' both
result in the call of the reflected method

Are you sure? raise TypeError *should* result in the operation being abandoned, with the reflected operation not tried.

Python 3.3.0rc2+ (default:3504cbb3e1d8, Sep 20 2012, 22:08:44) 
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class A:
...     def __add__(self, other):
...         raise TypeError("Don't know how to add")
...     def __le__(self, other):
...         raise TypeError("Can't compare")
... 
[65945 refs]
>>> class B:
...     def __radd__(self, other):
...         return 42
...     def __ge__(self, other):
...         return False
... 
[66016 refs]
>>> A() <= B()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in __le__
TypeError: Can't compare
[66064 refs]
>>> A() + B()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in __add__
TypeError: Don't know how to add
[66065 refs]

[terryjreedy]

You are right, I did not look deep enough. I was fooled by the conversion of NotImplemented, returned from object.__le__, etc, to TypeError. Sorry for that noise.

For comparison and arithmetic, the actual alternative to defining a function that returns NotImplemented seems to be to not define it at all.

class C():
    def __ge__(self, other):
        return True
    def __add__(self, other):
        return 44
    __radd__ = __add__

class O():
    pass  # removed NotImplemented defs
    
c = C()
o = O()
print(c >= o, o <= c)
# True True
print(c + o, o + c)
# 44 44

(I looked at the codes for binary_op1 in abstract.c and do_richcompare in object.c and do not yet see any effective difference between not defined and a NotImplemented return.)

I'll take a look at the patch later.