Skip to content

MemoryError creating recarrays from medium sized python objects (Trac #479) #1077

Closed
numpy-gitbot opened this Issue Oct 19, 2012 · 3 comments

1 participant

@numpy-gitbot

Original ticket http://projects.scipy.org/numpy/ticket/479 on 2007-03-26 by @FrancescAlted, assigned to unknown.

The next exposes the (weird) behaviour:

In [207]:N=8670
In [208]:numpy.rec.array([[range(N)]], dtype=[('var1', 'f8', (N,))])
Out[208]:
recarray([ (array([  0.00000000e+00,   1.00000000e+00,   2.00000000e+00, ...,
         8.66700000e+03,   8.66800000e+03,   8.66900000e+03]),)],
      dtype=[('var1', '<f8', (8670,))])
In [209]:N=8680
In [210]:numpy.rec.array([[range(N)]], dtype=[('var1', 'f8', (N,))])
---------------------------------------------------------------------------
exceptions.MemoryError                               Traceback (most recent call last)

/home/faltet/<ipython console>

/usr/lib/python2.4/site-packages/numpy/core/records.py in array(obj, dtype, shape, offset, strides, formats, names, titles, aligned, byteorder, copy)
    531     elif isinstance(obj, (list, tuple)):
    532         if isinstance(obj[0], (tuple, list)):
--> 533             return fromrecords(obj, dtype=dtype, shape=shape, **kwds)
    534         else:
    535             return fromarrays(obj, dtype=dtype, shape=shape, **kwds)

/usr/lib/python2.4/site-packages/numpy/core/records.py in fromrecords(recList, dtype, shape, formats, names, titles, aligned, byteorder)
    384
    385     try:
--> 386         retval = sb.array(recList, dtype = descr)
    387     except TypeError:  # list of lists instead of list of tuples
    388         if (shape is None or shape == 0):

MemoryError:

The memory of my machine is 512 MB, and the recarray + python object should fit in far less than 1 MB.

Curiously enough, you can achieve the desired recarray by using a regular numpy array

In [211]:numpy.rec.array(numpy.arange(N*1.), dtype=[('var1', 'f8', (N,))])
Out[211]:
recarray([ (array([  0.00000000e+00,   1.00000000e+00,   2.00000000e+00, ...,
         8.67700000e+03,   8.67800000e+03,   8.67900000e+03]),)],
      dtype=[('var1', '<f8', (8680,))])

But note the strange nesting of the resulting recarray (see ticket #1076).

@numpy-gitbot

@teoliphant wrote on 2007-03-29

rec arrays use various memory copies to get the data into the right format.

If memory is a concern, then use standard arrays.

@numpy-gitbot

@alberts wrote on 2007-05-12

If there is no bug here, this issue can probably be resolved?

@numpy-gitbot

Milestone changed to 1.0.3 Release by @alberts on 2007-05-12

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.