Fix complex type to avoid coercion in 2.7. #49461
Comments
|
In the 'coercion rules' section of the reference manual, at: http://docs.python.org/dev/reference/datamodel.html#id5 it says: """Over time, the type complex may be fixed to avoid coercion.""" In 3.x, the complex type has (necessarily) been fixed to avoid coercion, As far as I can see, there's no great benefit in such a change, except |
mdickinson
added
interpreter-core
|
Comment by gumtree copied from bpo-3734 discussion:
gumtree, would you be interested in working on a patch for this feature- Alternatively, could you explain in a little more detail why this change |
|
I am happy to collaborate in finding a solution, but I do not have Simply documenting it does not remove the frustration that a few people Would you want to subclass a numeric type? I agree, it is a bit obsure, In my case, I think that the motivation may seem a bit obscure. I had If that explanation does not make sense, then I suppose other simpler So, my feeling is that this is worth fixing because the work done on |
|
Mark, Is this still of interest? I found the relevant changes in py3k, but I am not sure it is the behavior that gumtree is expecting. Since py3k removes coercion completely, the test case from bpo-3734 would just issue: Traceback (most recent call last):
File "test-5211.py", line 34, in <module>
print type(z + xz)
File "test-5211.py", line 5, in __coerce__
t = complex.__coerce__(self,other)
AttributeError: type object 'complex' has no attribute '__coerce__'Where as I think gumtree wants the xcomplex case to behave as the xfloat case, e.g. On the other hand, as you mentioned, the removal of coercion from complex could be backported. However, if we are going to do that then we might as well just backport the whole removal of coercion instead of just the bits from 'complexobject.c'. Bascially checkins r51431 and r58226. |
|
Yes, I'd certainly be interested in reviewing a patch. Though the current behaviour is at most a minor wart, and since it's gone in Python 3.x the motivation to fix it isn't huge. :)
Yes, that was what I was proposing. But as you point out, the new behaviour wouldn't even match the behaviour of Python 3.x, so it really wouldn't be a terribly useful change.
That's not really an option: it has the potential to break existing code that uses coercion. Removing coercion in Python 2.x would require at least a PEP plus one version's worth of DeprecationWarning. And given that it currently looks as though Python 2.8 isn't likely to happen at all, that would likely be a wasted effort. |
|
I also agree that this bug was never more than a small wart. However, I'm now curious. If Python 3 does not support coercion, I think that it will not be possible to write something like my xfloat class, derived from float (i.e., some binary operations between an xfloat and a float would return a float instead of an xfloat). If I am correct in think that it would seem to be a step backward. Will Python 3 deal with mixed types in some other way, or has this problem been abandoned altogether? If it is the latter, I think it is a pity. |
As you pointed out in bpo-3734, the patch is basically: =================================================================== --- Objects/complexobject.c (revision 77909)
+++ Objects/complexobject.c (working copy)
@@ -1282,7 +1282,8 @@
PyObject_GenericGetAttr, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
- Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
+ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_CHECKTYPES
+ | Py_TPFLAGS_BASETYPE, /* tp_flags */
complex_doc, /* tp_doc */
0, /* tp_traverse */
0, /* tp_clear */plus the relevant test cases and documentation changes.
Ah, good point. In that case I am -1 on changing the behavior at all. For the sole reason that cases like Blair originally reported would not work as expected in version < 2.7, then would work in 2.7, and then would break again in > 3.0. I think explaining and documenting the change in behavior would be really confusing.
I disagree. I think type coercion (or implicit conversions) is hard to follow and understand. I say this from many years of C and C++ programming. I see bugs all of the time in these languages due to conversions silently happening that the programmer was not aware of. Yes the feature is convenient, but documenting and explaining these rules to programmers becomes very tedious and requires that most programmers have deep understanding of the languages typing rules. This typically leads to books like "Effective C++" that catalog a list of things *not* to do in a language. The relevant case for C++ being to avoid user defined conversions and use explicit constructers. As such, I am definitely +1 on removing coercion. |
|
OK. I have gone back to the beginning to refresh my memory and I see a possible point of misunderstanding. I am not sure that we are really talking about the problem that prompted my initial report (msg72169, bpo-3734). Immediately following my message, Daniel Diniz confirmed the bug and expanded on my code with an xfloat class of his own that uses __coerce__. In fact, if I had submitted an xfloat class it would have been the following class xfloat( float ):
def __new__(cls,*args,**kwargs):
return float.__new__(cls,*args,**kwargs)
def __add__(self,x):
return xfloat( float.__add__(self,x) )
def __radd__(self,x):
return xfloat( float.__radd__(self,x) )My xfloat works fine in 2.6.4 and it was my wish, at the time, to write a class for xcomplex that behaved in a similar way. According to the Python manual, that should have been possible, but it wasn't. So, I guess coercion is not really the problem. However, there does seem to be something wrong with the complex type. I have looked at the manual for Python 3 and see that the same rules apply for classes that emulate numeric types, namely: "If the right operand’s type is a subclass of the left operand’s type and that subclass provides the reflected method for the operation, this method will be called before the left operand’s non-reflected method. This behavior allows subclasses to override their ancestors’ operations." The question I have then is will the following work in Python 3 (it doesn't in 2.6.4)? class xcomplex( complex ):
def __new__(cls,*args,**kwargs):
return complex.__new__(cls,*args,**kwargs)
## def __coerce__(self,other):
## t = complex.__coerce__(self,other)
## try:
## return (self,xcomplex(t[1]))
## except TypeError:
## return t
def __add__(self,x):
return xcomplex( complex.__add__(self,x) )
def __radd__(self,x):
return xcomplex( complex.__radd__(self,x) )
xz = xcomplex(1+2j)
xy = float(10.0)
z = complex(10+1j)print "would like xcomplex type each time" |
|
Blair: I don't think you'll have any problems getting the behaviour you in Python 3. For example: Python 3.2a0 (py3k:77952, Feb 4 2010, 10:56:12)
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class xcomplex(complex):
... def __add__(self, other): return xcomplex(complex(self) + other)
... __radd__ = __add__
...
>>> xz = xcomplex(1+2j)
>>> all(type(xz + y) is type(y + xz) is xcomplex for y in (1, 10.0, 10+1j, xz))
TrueSo I don't consider that the removal of coerce and the __coerce__ magic method is a step backward: it still allows mixed-type operations, but without coerce the rules for those operations are significantly cleaner. The real problem case in 2.6.4 seems to be when doing <instance of complex> + <instance of xcomplex>: in this case, Python first calls complex.__coerce__ (which returns its arguments unchanged), then complex.__add__. None of the xcomplex methods even gets a look in, so there's no opportunity to force the return type to be xcomplex. If you look at the source (see particularly the binary_op1 function in Objects/abstract.c ) you can see where this behaviour is coming from. The complex type (along with its subclasses) is classified as an 'old-style' number, while ints, longs and floats are 'new-style' (note that this has nothing to do with the distinction between old-style and new-style classes). Operations between new-style numbers use the scheme described in the documentation, but where old-style numbers are involved there's an extra coercion step. In particular, when adding a complex to an xcomplex, the rule you quoted (about the case when the right operand is an instance of a subclass of the class of the left operand) isn't applied. It's too risky to change the general behaviour for old-style numbers: I don't think there are any old-style numbers besides complex in the Python code, but there may well be some in third party extensions. Certainly documenting it better would be an option, and making complex a new-style number for Python 2.7 seems like a reasonable thing to consider. |
Yes, that's the essence of it. In addition, each of the functions implementing a complex special method would need to do its own argument conversion. (Compare the implementation of complex_add in py3k with that in trunk.) |
Hmm. I take this back: if complex were made 'new-style' in 2.7, then it *would* be possible to write fairly obvious code (not using coerce or __coerce__) that operated in the same way in both 2.7 and 3.2. So I still think it's worth considering. |
Agreed. I have attached a patch with src, test, and doc changes. |
|
Thanks. I'll try to find time to look at this tomorrow. |
|
Apologies for the delay; tomorrow was a long time coming... The patch looks great---thank you! I added a ".. versionchanged" note to the documentation, and fixed a couple of whitespace issues; apart from that I didn't change anything. Applied in r78280. |
|
I must have been operating on autopilot; not only did I forget to put the issue number in the checkin message, but I forgot to acknowledge Meador Inge for the patch. Fixed now, with apologies to Meador. |
Thanks. I checked out the changes you made so that I will know what to do next time :).
No worries. Thanks for applying the patch! |
|
r78280 didn't remove the implicit coercion for rich comparisons; that's now been done in r81606. |
|
Hi Mark, I thought that this had all been fixed, but it seems not. Consider the following: class xcomplex( complex ):
def __new__(cls,*args,**kwargs):
return complex.__new__(cls,*args,**kwargs)
def __add__(self,x):
return xcomplex( complex.__add__(self,x) )
def __radd__(self,x):
print "larg: ", type(x),"returning: ",
return xcomplex( complex.__radd__(self,x) )
class xfloat(float):
def __new__(cls,*args,**kwargs):
return float.__new__(cls,*args,**kwargs)
def __add__(self,x):
return xfloat( float.__add__(self,x) )
def __radd__(self,x):
print "larg: ", type(x),"returning: ",
return xfloat( float.__radd__(self,x) )
z = 1j
xz = xcomplex(1j)
f = 1.0
xf = xfloat(1.0)print When this runs, the first three conversions are fine, the last is not: there ----------- The last line shows that no call to __radd__ occurred. Is there anything that can be done now about this now, or is it just too Regards Blair At 01:13 a.m. 31/05/2010, you wrote: Mark Dickinson <dickinsm@gmail.com> added the comment: r78280 didn't remove the implicit coercion for rich comparisons; that's now ---------- Python tracker <report@bugs.python.org> |
|
I think that's expected behaviour. Note that int vs float behaves in the same way as float vs complex: >>> class xint(int):
... def __radd__(self, other):
... print "__radd__"
... return 42
...
>>> 3 + xint(5)
__radd__
42
>>> 3.0 + xint(5) # xint.__radd__ not called.
8.0As with your example, the float.__add__ method is happy to deal with an int or an instance of any subclass of int. |
|
I see your point Mark, however it does not seem to be the right way to do Are you aware that Python has formally specified this behaviour somewhere? I The problem that has been fixed is covered in the documentation: (3.4.8. Emulating numeric types: Note This rule is needed so that mixed-type arithmetic operations do not revert xi = xint(3)
3 + xi # is an xint(6)
3.0 + xi # is float(6)This is the same problem as the one that has been fixed from a practical It seems to me that xint.__radd__ would need to be called if the left Am I missing something? Mark Dickinson <dickinsm@gmail.com> added the comment: I think that's expected behaviour. Note that int vs float behaves in the ... def __radd__(self, other):
... print "__radd__"
... return 42
...
>>> 3 + xint(5)
__radd__
42
>>> 3.0 + xint(5) # xint.__radd__ not called.
8.0As with your example, the float.__add__ method is happy to deal with an int ---------- Python tracker <report@bugs.python.org> |
|
I'd like to add a few more observations to the mix. I have run the following in both 2.6.6 and in 2.7 class xfloat(float):
def __new__(cls,x):
return float.__new__(cls,x)
def __radd__(self,lhs):
print "__radd__ got: %s" % type(lhs)
if isinstance(lhs,(int,float)):
return xfloat( float(self) + lhs )
else:
return NotImplemented
xf = xfloat(9.0)
cases = dict(int=1,float=1.0,complex=1.0+1j)
for k,v in cases.items():
y = v + xf
print "%s + xfloat" % k
print type(y)
print yIn 2.7 this gives: __radd__ got: <type 'int'> In 2.6.6 I get: __radd__ got: <type 'int'> They are the same except for the last case. My feeling is that the behaviour of 2.6.6 (for subclassing float) is The behaviour of 2.6.6 is needed to enable you to implement the commutative I have also tried the following class xint(int):
def __new__(cls,x):
return int.__new__(cls,x)
def __radd__(self,lhs):
print "__radd__ got: %s" % type(lhs)
if isinstance(lhs,(int,)):
return xint( float(self) + lhs )
else:
return NotImplementedprint cases = dict(int=1,float=1.0,complex=1.0+1j)
for k,v in cases.items():
y = v + xf
print "%s + xint" % k
print type(y)
print yIn 2.6.6 I get __radd__ got: <type 'int'> and in 2.7 ------------------- In my opinion, 2.6.6 was faulty in the float + xint case, for the same |
Well, I disagree: Python is behaving as designed and documented in these cases. If you want to argue that the *design* decisions are the wrong ones, then I'd suggest opening a discussion on the python-ideas mailing list, where more people are likely to get involved---this tracker isn't really the right place for that sort of discussion. Leaving complex out of the mix for the moment, it sounds to me as though you'd like, e.g., <float> + <subclass of int> to call the int subclass's __radd__ method (if it exists) before calling the float's __add__ method. Is that correct? Or are you suggesting that float's __add__ method shouldn't accept instances of subclasses of int at all? (i.e., that float.__add__ should return NotImplemented when given an instance of xint). In the first case, you need to come up with general semantics that would give you the behaviour you want for float and xint---e.g., when given numeric objects x and y, what general rule should Python apply to decide whether to call x.__add__ or y.__radd__ first? In the second case, I'd argue that you're going against the whole idea of object-oriented programming; by making xint a subclass of int, you're declaring that its instances *are* 'ints' in a very real sense, so it's entirely reasonable for float's __add__ method to accept them. In either case, note that Python 2.x is not open for behaviour changes, only for bugfixes. Since this isn't a bug (IMO), such changes could only happen in 3.x. Please take further discussion to the python-ideas mailing list. |
|
Just to keep this discussion as clear as possible Mark, it was your first When that is done (as it was for a subclass of float in 2.6.6) it is On Mon, Nov 29, 2010 at 5:04 AM, Mark Dickinson <report@bugs.python.org>wrote:
|
Okay, so you want <float instance> + <xint instance> to try xint.__radd__ before float.__add__. How do you propose this be achieved? Can you explain exactly what semantics you'd like to see? You've indicated what you want to happen for float and xint, but how should Python behave in general here? In particular, when evaluating 'x + y' for general Python objects x and y, what rule or rules should Python use to decide whether to try x.__add__(y) or y.__radd__(x) first? It seems you want some general mechanism that results in xint being 'preferred' to float, in the sense that xint.__radd__ and xint.__add__ will be tried in preference to float.__add__ and float.__radd__ (respectively). But it's not clear to me what criterion Python would use to determine which out of two types (neither one inheriting from the other) should be 'preferred' in this sense. |
|
I am not really the person (I don't know how Python is implemented) to However, for what its worth, I think that the 'correct behaviour' was I would say that the semantics do not need to apply to arbitrary Python In that case: For 'x opn y' any binary operator (like +,*, etc), if (and only if) 'x' is If that were done, then subclasses of number types can implement commutative I see this as 'special' behaviour required of the int, long, float and If both 'x' and 'y' are subclasses of built in number types the required And should this apply to non-number types? I think not. Numbers deserve I hope this helps. On Mon, Nov 29, 2010 at 9:48 AM, Mark Dickinson <report@bugs.python.org>wrote:
|
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: