Skip to content
This repository has been archived by the owner on Dec 11, 2023. It is now read-only.

Can't create large multidimensional carray #25

Closed
josephdunn opened this issue Sep 6, 2013 · 1 comment
Closed

Can't create large multidimensional carray #25

josephdunn opened this issue Sep 6, 2013 · 1 comment

Comments

@josephdunn
Copy link

With Numpy I can do something like this:

foo = np.zeros([ 2 ] * 20)

And get an ndarray with the corresponding shape. I can then:

ac = ca.carray(foo)

To get a carray object. Great. But I'm playing around with carray because I want to use array sizes that are larger than could otherwise fit in memory, and [2] * 20 is an easy shape for Numpy to handle, so for me it's a baseline of sorts.

Looking to explore the capabilities of carray, I try to create the object directly, without the intermediate Numpy step:

ac = ca.zeros([2] * 20)

But I get an error:

~/python/lib/python2.7/site-packages/carray/toplevel.pyc in zeros(shape, dtype, **kwargs)
    291     """
    292     dtype = np.dtype(dtype)
--> 293     return fill(shape=shape, dflt=np.zeros((), dtype), dtype=dtype, **kwargs)
    294
    295 def ones(shape, dtype=np.float, **kwargs):

~/python/lib/python2.7/site-packages/carray/toplevel.pyc in fill(shape, dflt, dtype, **kwargs)
    256     # Then fill it
    257     # We need an array for the defaults so as to keep the atom info
--> 258     dflt = np.array(obj.dflt, dtype=dtype)
    259     # Making strides=(0,) below is a trick to create the array fast and
    260     # without memory consumption

Which leads me to wonder: is something like this possible using carray?

@FrancescAlted
Copy link
Member

Yes, this is a bug. The reason of the failure is that carray tries to create a default value that is too large to fit in memory, and this is why this fails. This must be fixed.

BTW, carray is not maintained anymore as such and its currently encarnated as the BLZ component of Blaze:

https://github.com/ContinuumIO/blaze

I filed this ticket there instead (blaze/blaze#75). Thanks for the report.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants