Avoid np.prod in make_shared_array

[h-mayorquin]

np.prod produces a numpy scalars (not integers or floats!).

Numpy scalars behave like numpy values and can overflow. Example:

In [1]: a_billion_int = 1_000_000_000

In [2]: np.prod((a_billion_int, a_billion_int, a_billion_int, a_billion_int))
Out[2]: -5527149226598858752

Python integers on the other hand do not overflow.

This PR changes this so overflow errors do not happen.

This error appear to a user here:
#1871 (comment)

[zm711]

This still needs to be wrapped as an int no?

[zm711]

Related to Windows failing.

[h-mayorquin]

itemsize is int, shape is int, that's unecessary. Am I wrong?

[zm711]

When I check they are coming up as ints, but based on the fact the tests are failing just for Windows and all you did was remove the int over the operation maybe something was being converted.... Not sure.

[h-mayorquin]

The thing that converts is the np.prod. because of an overflow. Let me show you:

In [1]: a_billion_int = 1_000_000_000

In [2]: np.prod((a_billion_int, a_billion_int, a_billion_int, a_billion_int))
Out[2]: -5527149226598858752

[h-mayorquin]

Yeah, the only thing that I can't think off is that fork (the multiprocessing mode) does not exist on windows which implies some re-instantiation and some of the attributes estimated instead of accessed... But I am out of luck.

Where else is a possible point of overflow?

[zm711]

Windows and linux numpy handle overflow differently. See this. Based on the discussion the behavior will likely be changed in Numpy 2.0, but in short

np.full((2, 2), np.iinfo(np.int32).max, dtype=np.int32).sum(axis=0)
shows similar behavior: array([4294967294, 4294967294]) on ubuntu and array([-2, -2]) on windows.

However, this function returns an array instead of a scalar, so we can inspect the dtype. On Ubuntu the dtype is int64, but on Windows the dtype is int32.

So maybe this was due to the difference in how certain cases of overflow occur.

[h-mayorquin]

Thanks a bunch for sharing the issue!

[h-mayorquin]

So numpy 2.0 should align this.

[zm711]

Anything that makes the debugging easier among the operating systems. Much less surface area if one fix works for all of them. :) (although the final message wasn't completely confident sounding...)

[zm711]

Suggested change

	shape = (int(x) for x in shape) # We need to be sure that shape comes in int and not number scalars
	shape = tuple(int(x) for x in shape) # We need to be sure that shape comes in int and not numy scalars

[zm711]

Cool you saw it :)

[h-mayorquin]

Yes, thanks, I realized it just now.

You know this generator notation has fucked me up a couple of times, check this out:

is_one_equal_to_0 = np.all((1 == 0 for _ in range(10)))
if is_one_equal_to_0:
    print("This is great, 1 is equal to 0")

Syntactic suggar is nice until it is not.

[zm711]

That's a cool example. The tuple/generator for () and (even sometimes I forget) {} for set in addition to dictionary. Those mess with me. But yeah the accident generator is way too often.

[samuelgarcia]

bien joué!