uint64 converted silently to float64 when adding an int

[nasailja]

This code

import numpy
a = numpy.zeros(1, dtype = numpy.uint64)[0]
print(type(a))
i = 1
print(type(i))
a += i
print(type(a))

prints

<class 'numpy.uint64'>
<class 'int'>
<class 'numpy.float64'>

which was a big surprise for me. Why would adding an integer to uint64 result in a floating point value?

[charris]

Because integer is signed and uint64 is not, so the common type needs more precision than uint64. Using float64 is admittedly a compromise since some precision is still lost. The += operator here is also between two (immutable) scalars, so it is not actually done inplace, rather, Python adds the two numbers and assigns the result to a.

[nasailja]

Hmm 0x123...def is only about 2**61 which seems to fit into a regular int:

>>> 0x1234567890abcdef
1311768467294899695
>>> 2**60
1152921504606846976
>>> 2**61
2305843009213693952
>>> 2**62
4611686018427387904
>>> 2**63
9223372036854775808L

so 64 bits should be enough whether the number is signed or not. At least for me a regular integer is able to represent all positive values that are supposed to fit into it (on x86_64):

>>> 0x7fffffffffffffff
9223372036854775807
>>> 0x8000000000000000
9223372036854775808L

This behavior also seems odd:

>>> numpy.uint64(0x7ffffffffffffffe) - 2**63
-2L
>>> numpy.uint64(0x7ffffffffffffffe) - 2**62
4.6116860184273879e+18
>>> numpy.uint64(0x7ffffffffffffffe) - 2.0**63
0.0

I guess 0x7f...f rounds to the same value as 2**63 when converted to floating point, so the case of adding to a standard floating point value makes sense but what is happening with 2**62 and how can that be considered not a bug?

[nasailja]

Even better demonstration that numpy is doing something strange:

>>> 2**62+(2**62-1)
9223372036854775807
>>> numpy.uint64(2**62)+(2**62-1)
9.2233720368547758e+18

[nasailja]

Looks like a bug in uint64 as int64 works correctly:

>>> numpy.int64(2**62)+(2**62-1)
9223372036854775807
>>> numpy.uint64(2**62)+(2**62-1)
9.2233720368547758e+18

[nasailja]

With respect to converting to floating point, I'd have preferred int64's behavior:

>>> numpy.int64(2**62)+(2**62)
__main__:1: RuntimeWarning: overflow encountered in long_scalars
-9223372036854775808
>>> numpy.uint64(2**62)+(2**62)
9.2233720368547758e+18

even though it shouldn't be needed for uint64 above. It seems to always want to convert to floating point:

>>> numpy.uint64(1) + 1
2.0

I'm using Python 3.4.3 (default, Mar 10 2015, 14:38:54) [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin and numpy.version.version '1.9.2' from macports.

[charris]

The difference is that in the first example both numbers are signed. For mixed signed and unsigned types the result is the smallest signed type that can hold both. For example

In [1]: type(uint32(1) + 1)
Out[1]: numpy.int64

If you need to have uint64 type result, do

In [5]: type(uint64(2**62)+ uint64((2**62-1)))
Out[5]: numpy.uint64

One can argue that numpy should check the actual values, but then the result becomes less predictable.

[nasailja]

Kind of makes sense but it still doesn't feel consistent that uint64 "increases" the size of the variable even when there's no need but int64 doesn't even when there is. Especially as uint64's current behavior can lead to subtle bugs that might be difficult to find: #5746 If I recall correctly signed integer overflow is undefined in C but unsigned integer overflow is defined and widely used which makes numpy's behavior even more confusing.

[charris]

Which is not to say I don't agree with you that a float result is weird, I was never happy with that myself. ISTR that at some point the result of adding arrays of uint64 and int64 was of object type, but I may be mistaken there.

[njsmith]

None of numpy's conversion rules look at the actual values, only at their
type. This can be confusing in these kinds of micro tests but is an
important property to make sure that code works consistently and that what
you get when trying stuff at the prompt like this matches what you'll get
with larger days later where you aren't checking every value.

Silently casting uint64 to float64 is pretty weird though -- it breaks all
the usual rules for which casts are allowed. (It changes kinds and loses
precision, so by the rules we use everywhere else it's actually worse
than e.g. silently casting float64 to float32.) Bleh.

I guess we could stretch the scalar kind rule to say that positive python
ints are ambiguous between unsigned and signed, sort of meld two weird
special cases into one? ints are already very weird since in py3 they're
now always infinite precision, but we still treat them as int32 or int64
depending on platform. This wouldn't fix all the weirdness -- int64 +
uint64 would still be weird -- but at least uint64 + int would be less
surprising?

Kind of makes sense but it still doesn't feel consistent that uint64
"increases" the size of the variable even when there's no need but int64
doesn't even when there is. Especially as uint64's current behavior can
lead to subtle bugs that might be difficult to find: #5746
#5746 If I recall correctly signed
integer overflow is undefined in C but unsigned integer overflow is defined
and widely used which makes numpy's behavior even more confusing.

—
Reply to this email directly or view it on GitHub
#5745 (comment).

[153957]

We had a similar problem in our code, which we work around this by converting the value to a Python int, adding the extra number and converting back to uint64.

ts = np.uint64(1400000002000000600)
print ts + 1
# 1.4000000020000005e+18
print np.uint64(int(ts) + 1)
# 1400000002000000601

[zed]

@njsmith :

None of numpy's conversion rules look at the actual values, only at their
type.

Here's an example of the NumPy type promotion rules:

>>> np.result_type(np.int32(0), 0) 
dtype('int64') # np.int_
>>> np.result_type(np.int32(0), 1<<62)
dtype('int64')
>>> np.result_type(np.int32(0), 1<<63)
dtype('float64')
>>> np.result_type(np.int32(0), 1<<64)
dtype('O') # Python object

The input types are the same in all these cases (np.int32, int) but the result types are different.

[shoyer]

The input types are the same in all these cases (np.int32, int) but the result types are different.

@njsmith's statement is mostly but not entirely correct: scalar types are treated specially. But it certainly is true that for arrays (excluding 0-dimensional arrays), NumPy does not look at actual values.