Fix complex type to avoid coercion in 2.7.

[mdickinson]

BPO	5211
Nosy	@mdickinson, @meadori
Files	issue-5211-patch: patch against 2.7 trunk unnamed

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/mdickinson'
closed_at = <Date 2010-02-21.12:58:44.750>
created_at = <Date 2009-02-10.23:19:22.161>
labels = ['interpreter-core', 'easy', 'type-feature']
title = 'Fix complex type to avoid coercion in 2.7.'
updated_at = <Date 2010-11-28.22:40:11.928>
user = 'https://github.com/mdickinson'

bugs.python.org fields:

activity = <Date 2010-11-28.22:40:11.928>
actor = 'gumtree'
assignee = 'mark.dickinson'
closed = True
closed_date = <Date 2010-02-21.12:58:44.750>
closer = 'mark.dickinson'
components = ['Interpreter Core']
creation = <Date 2009-02-10.23:19:22.161>
creator = 'mark.dickinson'
dependencies = []
files = ['16148', '19861']
hgrepos = []
issue_num = 5211
keywords = ['easy']
message_count = 25.0
messages = ['81612', '81633', '81694', '98714', '98784', '98798', '98810', '98811', '98828', '98829', '98830', '98941', '98971', '99653', '99691', '99697', '106757', '122432', '122453', '122472', '122499', '122659', '122724', '122727', '122742']
nosy_count = 3.0
nosy_names = ['mark.dickinson', 'gumtree', 'meador.inge']
pr_nums = []
priority = 'low'
resolution = 'accepted'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue5211'
versions = ['Python 2.7']

[mdickinson]

In the 'coercion rules' section of the reference manual, at:

http://docs.python.org/dev/reference/datamodel.html#id5

it says:

"""Over time, the type complex may be fixed to avoid coercion."""

In 3.x, the complex type has (necessarily) been fixed to avoid coercion,
and it ought to be a fairly easy task to backport that fix to 2.7, for
someone who wants to get his or her feet wet with some CPython hacking.

As far as I can see, there's no great benefit in such a change, except
that the presence of coercion for the complex type causes confusion
occasionally: see bpo-3734 for an example of this.

[mdickinson]

Comment by gumtree copied from bpo-3734 discussion:

While Mark Dickinson's patch fixes the documentation, it does not
offer
a solution to the original problem, which was rooted in a need to
provide special behaviour based on the numeric types. I made the
original posting because I hoped that this problem could be resolved.

gumtree, would you be interested in working on a patch for this feature-
request? As mentioned above, the necessary changes are already present
in 3.x, so all that's entailed is figuring out which bits of the
Object/complexobject.c in the py3k source need to be transferred to the
trunk, and making sure that everything's properly tested.

Alternatively, could you explain in a little more detail why this change
is important to you? Subclassing complex doesn't seem like a very
common thing to want to do, so I'm curious about your use case.

[gumtree]

I am happy to collaborate in finding a solution, but I do not have
enough knowledge or skill to work with the code alone.

Simply documenting it does not remove the frustration that a few people
will enounter. The key point being that you can subclass the other
numeric types, but not complex. Worse, you may think that you have
succeeded in subclassing complex too, because it is only when the
__ropn__ binary operators are used that the type reverts to the base
class type.

Would you want to subclass a numeric type? I agree, it is a bit obsure,
but I did a search on this before making the post and there have been
others who found the problem too.

In my case, I think that the motivation may seem a bit obscure. I had
developed some numerical-like types (from scratch -- no subclassing)
and I wanted to be able to write functions taking as possible arguments
these types and Python numerical types indifferently. I realised that I
could not achieve exactly what I wanted, however, by subclassing float,
int, etc I could add a few methods that would allow my generic
functions to work with either my types or the subclassed Python types.
At the same time, the subclassed numerical types could still be used as
numerical quantities (float, int,...). It seemed like a pretty elegant
solution.

If that explanation does not make sense, then I suppose other simpler
motivations could be, eg, to subclass float so that only positive
values are acceptable; to subclass complex so that only values lying
within the unit circle are acceptable, etc. That is, one might like to
define a type that can only take on physically meaningful values (mass
cannot be negative, a complex reflection coeffcieint cannot have a
magnitude greater than unity, ..)

So, my feeling is that this is worth fixing because the work done on
float, int etc, is clearly useful and it appears (to me) that the
complex case is an oversight.

[meadori]

Mark,

Is this still of interest?

I found the relevant changes in py3k, but I am not sure it is the behavior that gumtree is expecting. Since py3k removes coercion completely, the test case from bpo-3734 would just issue:

   Traceback (most recent call last):
     File "test-5211.py", line 34, in <module>
       print type(z + xz)
     File "test-5211.py", line 5, in __coerce__
       t = complex.__coerce__(self,other)
   AttributeError: type object 'complex' has no attribute '__coerce__'

Where as I think gumtree wants the xcomplex case to behave as the xfloat case, e.g.
<class '__main__.xfloat'>
<class '__main__.xcomplex'>
Since coercion is getting axed in py3k, I don't think it makes sense to provide this behavior.

On the other hand, as you mentioned, the removal of coercion from complex could be backported. However, if we are going to do that then we might as well just backport the whole removal of coercion instead of just the bits from 'complexobject.c'. Bascially checkins r51431 and r58226.

[mdickinson]

Yes, I'd certainly be interested in reviewing a patch. Though the current behaviour is at most a minor wart, and since it's gone in Python 3.x the motivation to fix it isn't huge. :)

Where as I think gumtree wants the xcomplex case to behave as
the xfloat case, e.g. ...

Yes, that was what I was proposing. But as you point out, the new behaviour wouldn't even match the behaviour of Python 3.x, so it really wouldn't be a terribly useful change.

However, if we are going to do that then we might as well just
backport the whole removal of coercion.

That's not really an option: it has the potential to break existing code that uses coercion. Removing coercion in Python 2.x would require at least a PEP plus one version's worth of DeprecationWarning. And given that it currently looks as though Python 2.8 isn't likely to happen at all, that would likely be a wasted effort.

[gumtree]

I also agree that this bug was never more than a small wart. However, I'm now curious.

If Python 3 does not support coercion, I think that it will not be possible to write something like my xfloat class, derived from float (i.e., some binary operations between an xfloat and a float would return a float instead of an xfloat).

If I am correct in think that it would seem to be a step backward.

Will Python 3 deal with mixed types in some other way, or has this problem been abandoned altogether? If it is the latter, I think it is a pity.

[meadori]

Mark

Yes, that was what I was proposing.

As you pointed out in bpo-3734, the patch is basically:

===================================================================

--- Objects/complexobject.c	(revision 77909)
+++ Objects/complexobject.c	(working copy)
@@ -1282,7 +1282,8 @@
 	PyObject_GenericGetAttr,		/* tp_getattro */
 	0,					/* tp_setattro */
 	0,					/* tp_as_buffer */
-	Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
+	Py_TPFLAGS_DEFAULT | Py_TPFLAGS_CHECKTYPES 
+                | Py_TPFLAGS_BASETYPE,          /* tp_flags */
 	complex_doc,				/* tp_doc */
 	0,					/* tp_traverse */
 	0,					/* tp_clear */

plus the relevant test cases and documentation changes.

Mark

That's not really an option: it has the potential to break existing
code that uses coercion. Removing coercion in Python 2.x would
require at least a PEP plus one version's worth of DeprecationWarning.

Ah, good point. In that case I am -1 on changing the behavior at all. For the sole reason that cases like Blair originally reported would not work as expected in version < 2.7, then would work in 2.7, and then would break again in > 3.0. I think explaining and documenting the change in behavior would be really confusing.

Blair

If I am correct in think that it would seem to be a step backward.

I disagree. I think type coercion (or implicit conversions) is hard to follow and understand. I say this from many years of C and C++ programming. I see bugs all of the time in these languages due to conversions silently happening that the programmer was not aware of. Yes the feature is convenient, but documenting and explaining these rules to programmers becomes very tedious and requires that most programmers have deep understanding of the languages typing rules. This typically leads to books like "Effective C++" that catalog a list of things *not* to do in a language. The relevant case for C++ being to avoid user defined conversions and use explicit constructers. As such, I am definitely +1 on removing coercion.

[gumtree]

OK. I have gone back to the beginning to refresh my memory and I see a possible point of misunderstanding. I am not sure that we are really talking about the problem that prompted my initial report (msg72169, bpo-3734).

Immediately following my message, Daniel Diniz confirmed the bug and expanded on my code with an xfloat class of his own that uses __coerce__.

In fact, if I had submitted an xfloat class it would have been the following

class xfloat( float ):

    def __new__(cls,*args,**kwargs):
        return float.__new__(cls,*args,**kwargs)
        
    def __add__(self,x):
        return xfloat( float.__add__(self,x) )

    def __radd__(self,x):
        return xfloat( float.__radd__(self,x) )

My xfloat works fine in 2.6.4 and it was my wish, at the time, to write a class for xcomplex that behaved in a similar way. According to the Python manual, that should have been possible, but it wasn't.

So, I guess coercion is not really the problem.

However, there does seem to be something wrong with the complex type.

I have looked at the manual for Python 3 and see that the same rules apply for classes that emulate numeric types, namely:

"If the right operand’s type is a subclass of the left operand’s type and that subclass provides the reflected method for the operation, this method will be called before the left operand’s non-reflected method. This behavior allows subclasses to override their ancestors’ operations."

The question I have then is will the following work in Python 3 (it doesn't in 2.6.4)?

class xcomplex( complex ):

    def __new__(cls,*args,**kwargs):
        return complex.__new__(cls,*args,**kwargs)

##    def __coerce__(self,other):
##        t = complex.__coerce__(self,other)
##        try:
##            return (self,xcomplex(t[1]))
##        except TypeError:
##            return t
        
    def __add__(self,x):
        return xcomplex( complex.__add__(self,x) )
    
    def __radd__(self,x):
        return xcomplex( complex.__radd__(self,x) )

xz = xcomplex(1+2j)
xy = float(10.0)
z = complex(10+1j)

print "would like xcomplex type each time"
print type(xz + z)
print type(xz + xy)
print type(xz + 10)
print type(xy + xz)
print type(10 + xz)
print type(z + xz)

[mdickinson]

Blair: I don't think you'll have any problems getting the behaviour you in Python 3. For example:

Python 3.2a0 (py3k:77952, Feb  4 2010, 10:56:12) 
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class xcomplex(complex):
...     def __add__(self, other): return xcomplex(complex(self) + other)
...     __radd__ = __add__
... 
>>> xz = xcomplex(1+2j)
>>> all(type(xz + y) is type(y + xz) is xcomplex for y in (1, 10.0, 10+1j, xz))
True

So I don't consider that the removal of coerce and the __coerce__ magic method is a step backward: it still allows mixed-type operations, but without coerce the rules for those operations are significantly cleaner.

The real problem case in 2.6.4 seems to be when doing <instance of complex> + <instance of xcomplex>: in this case, Python first calls complex.__coerce__ (which returns its arguments unchanged), then complex.__add__. None of the xcomplex methods even gets a look in, so there's no opportunity to force the return type to be xcomplex.

If you look at the source (see particularly the binary_op1 function in Objects/abstract.c ) you can see where this behaviour is coming from. The complex type (along with its subclasses) is classified as an 'old-style' number, while ints, longs and floats are 'new-style' (note that this has nothing to do with the distinction between old-style and new-style classes). Operations between new-style numbers use the scheme described in the documentation, but where old-style numbers are involved there's an extra coercion step. In particular, when adding a complex to an xcomplex, the rule you quoted (about the case when the right operand is an instance of a subclass of the class of the left operand) isn't applied.

It's too risky to change the general behaviour for old-style numbers: I don't think there are any old-style numbers besides complex in the Python code, but there may well be some in third party extensions. Certainly documenting it better would be an option, and making complex a new-style number for Python 2.7 seems like a reasonable thing to consider.

[mdickinson]

As you pointed out in bpo-3734, the patch is basically:
<snipped patch that adds Py_TPFLAGS_CHECKTYPES>

Yes, that's the essence of it. In addition, each of the functions implementing a complex special method would need to do its own argument conversion. (Compare the implementation of complex_add in py3k with that in trunk.)

[mdickinson]

Yes, that was what I was proposing. But as you point out, the new
behaviour wouldn't even match the behaviour of Python 3.x, so it really
wouldn't be a terribly useful change.

Hmm. I take this back: if complex were made 'new-style' in 2.7, then it *would* be possible to write fairly obvious code (not using coerce or __coerce__) that operated in the same way in both 2.7 and 3.2. So I still think it's worth considering.

[meadori]

if complex were made 'new-style' in 2.7, then it *would* be possible to > write fairly obvious code (not using coerce or __coerce__) that operated > in the same way in both 2.7 and 3.2. So I still think it's worth
considering.

Agreed. I have attached a patch with src, test, and doc changes.