Can't create large multidimensional carray

[josephdunn]

With Numpy I can do something like this:

foo = np.zeros([ 2 ] * 20)

And get an ndarray with the corresponding shape. I can then:

ac = ca.carray(foo)

To get a carray object. Great. But I'm playing around with carray because I want to use array sizes that are larger than could otherwise fit in memory, and [2] * 20 is an easy shape for Numpy to handle, so for me it's a baseline of sorts.

Looking to explore the capabilities of carray, I try to create the object directly, without the intermediate Numpy step:

ac = ca.zeros([2] * 20)

But I get an error:

~/python/lib/python2.7/site-packages/carray/toplevel.pyc in zeros(shape, dtype, **kwargs)
    291     """
    292     dtype = np.dtype(dtype)
--> 293     return fill(shape=shape, dflt=np.zeros((), dtype), dtype=dtype, **kwargs)
    294
    295 def ones(shape, dtype=np.float, **kwargs):

~/python/lib/python2.7/site-packages/carray/toplevel.pyc in fill(shape, dflt, dtype, **kwargs)
    256     # Then fill it
    257     # We need an array for the defaults so as to keep the atom info
--> 258     dflt = np.array(obj.dflt, dtype=dtype)
    259     # Making strides=(0,) below is a trick to create the array fast and
    260     # without memory consumption

Which leads me to wonder: is something like this possible using carray?

[FrancescAlted]

Yes, this is a bug. The reason of the failure is that carray tries to create a default value that is too large to fit in memory, and this is why this fails. This must be fixed.

BTW, carray is not maintained anymore as such and its currently encarnated as the BLZ component of Blaze:

https://github.com/ContinuumIO/blaze

I filed this ticket there instead (blaze/blaze#75). Thanks for the report.