Universal function at indices must have fewer than 2**31 elements or else they are ignored

[erykoff]

When running an accumulator (for example with np.add.at) then the length of the indices must be fewer than 2**32 elements. Otherwise, the accumulator only uses the first np.int32(indices.size) elements in the accumulator. This fails on all numpys up to and including 1.16.2.

Reproducing code example:

Note that the following code needs to be run on a computer with a lot of memory, >~ 64Gb.

This code does not work. Expectation is that inds.size is the same as accum.sum(), which is not true. You might notice that accum is mostly 0s.

import numpy as np

print(np.__version__)
np.random.seed(12345)

arr = np.arange(1000)
inds = np.random.choice(arr, size=2**32 - 1)
print(inds.size)

accum = np.zeros(arr.size, dtype=np.int64)
np.add.at(accum, inds, 1)
np.testing.assert_equal(accum.sum(), inds.size)

The following code does work, and seems to be the largest size that does:

import numpy as np

print(np.__version__)
np.random.seed(12345)

arr = np.arange(1000)
inds = np.random.choice(arr, size=2**31 - 1)
print(inds.size)

accum = np.zeros(arr.size, dtype=np.int64)
np.add.at(accum, inds, 1)

np.testing.assert_equal(accum.sum(), inds.size)

Error message:

There is no error logged from numpy , the answer is simply wrong. In the test code:

AssertionError: 
Items are not equal:
 ACTUAL: 0
 DESIRED: 4294967295

Numpy/Python version information:

1.16.2 3.7.2 (default, Dec 29 2018, 06:19:36)
[GCC 7.3.0]

[seberg]

A simpler reproducer:

arr = np.ones(0)
indices = np.zeros(1, dtype=np.intp)
indices = np.broadcast_to(indices, 2**32)

np.add.at(arr, indices, 1)  # should take very long, finishes quickly with wrong result
print(arr)
assert arr[0] == 2**32

EDIT: Actually meant to write np.ones(1), fixed the "test"