coo.todense() segfaults (Trac #1654)

[scipy-gitbot]

Original ticket http://projects.scipy.org/scipy/ticket/1654 on 2012-05-03 by trac user rlm, assigned to unknown.

Zipped pickle (~7MB) of a COO matrix can be found here:

http://rlmiller.org/badM.obj.zip

To reproduce:

import cPickle
M = cPickle.load(open('badM.obj'))
D = M.todense()
Segmentation fault: 11

The setup:

$ uname -a
Darwin dev-rmiller.local 11.3.0 Darwin Kernel Version 11.3.0: Thu Jan 12 18:47:41 PST 2012; root:xnu-1699.24.23~1/RELEASE_X86_64 x86_64

scipy-0.10.0-py2.7-macosx-10.7-intel

Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin

[scipy-gitbot]

@stefanv wrote on 2012-05-03

With 0.11.0.dev I get a MemoryError instead.

It looks like the row and column coordinates are within bounds:

np.max(M.row * M.shape[1] + M.col) > np.prod(M.shape)

But the output array would have a prohibitively large memory footprint:

np.dtype(float).itemsize * np.prod(M.shape) / 1024.**3

[willbuckner]

This is very much still an issue.

While the memory footprint may have been "prohibitively large" on rlm's machine, mine has 250GB of ram. This matrix is approximately 19GB.

In [4]: import numpy as np

In [5]: np.dtype(float).itemsize * np.prod(prM.shape) / 1024.**3
Out[5]: 19.75651279091835

In [6]: badM = cPickle.load(open('/var/xxx/badM.obj', 'rb'))

In [7]: badM.todense()
Segmentation fault

xxx@xxx:~/xxx$ free -m -g
total used free shared buffers cached
Mem: 230 59 171 0 0 51
-/+ buffers/cache: 7 223
Swap: 0 0 0

xxx@xxx:~/xxx$ ipython
Python 2.7.3 (default, Aug 1 2012, 05:14:39)

What gives?

[rgommers]

OK, reopening. This is with scipy 0.12.0 or master I assume?

[willbuckner]

@rgommers: Thanks! Yes, this is SciPy 0.12.0.

This is Ubuntu 12.04LTS, 230GB ram, 18 2.2Ghz Intel Xeon Virtual Cores. Python 2.7.3 from apt.

What other information can I provide you? Here's my show_config():

In [11]: sp.show_config()
amd_info:
libraries = ['amd']
library_dirs = ['/usr/lib']
define_macros = [('SCIPY_AMD_H', None)]
swig_opts = ['-I/usr/include/suitesparse']
include_dirs = ['/usr/include/suitesparse']
umfpack_info:
libraries = ['umfpack', 'amd']
library_dirs = ['/usr/lib']
define_macros = [('SCIPY_UMFPACK_H', None), ('SCIPY_AMD_H', None)]
swig_opts = ['-I/usr/include/suitesparse', '-I/usr/include/suitesparse']
include_dirs = ['/usr/include/suitesparse']
atlas_threads_info:
libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
define_macros = [('ATLAS_INFO', '""3.8.4""')]
language = f77
include_dirs = ['/usr/include/atlas']
blas_opt_info:
libraries = ['ptf77blas', 'ptcblas', 'atlas']
library_dirs = ['/usr/lib/atlas-base']
define_macros = [('ATLAS_INFO', '""3.8.4""')]
language = c
include_dirs = ['/usr/include/atlas']
atlas_blas_threads_info:
libraries = ['ptf77blas', 'ptcblas', 'atlas']
library_dirs = ['/usr/lib/atlas-base']
define_macros = [('ATLAS_INFO', '""3.8.4""')]
language = c
include_dirs = ['/usr/include/atlas']
lapack_opt_info:
libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
define_macros = [('ATLAS_INFO', '""3.8.4""')]
language = f77
include_dirs = ['/usr/include/atlas']
lapack_mkl_info:
NOT AVAILABLE
blas_mkl_info:
NOT AVAILABLE
mkl_info:
NOT AVAILABLE

In [12]: sp.test()
Running unit tests for scipy
NumPy version 1.7.1
NumPy is installed in /usr/local/lib/python2.7/dist-packages/numpy
SciPy version 0.12.0
SciPy is installed in /usr/local/lib/python2.7/dist-packages/scipy
Python version 2.7.3 (default, Aug 1 2012, 05:14:39) [GCC 4.6.3]
nose version 1.3.0
/usr/local/lib/python2.7/dist-packages/numpy/lib/utils.py:139: DeprecationWarning: scipy.lib.blas is deprecated, use scipy.linalg.blas instead!
warnings.warn(depdoc, DeprecationWarning)
/usr/local/lib/python2.7/dist-packages/numpy/lib/utils.py:139: DeprecationWarning: scipy.lib.lapack is deprecated, use scipy.linalg.lapack instead!
warnings.warn(depdoc, DeprecationWarning)

..............................................................................................................................................................................................................................K..............................................................................................................K....................................................................K..K............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................SSSSSS......SSSSSS......SSSS.......................................................................................................K..............................................................................................................................................................................................................................................................................................K..........................................................................................................................................................................................................................K.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K....................................................................K......................................................................................................................................................................KK..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K.K..........................K................................................................................................................................................................................................................................................................................................................................................................................K........K......................SSSSSSSS.........................................................................................................................................................................................S...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Ran 6136 tests in 75.327s

OK (KNOWNFAIL=16, SKIP=25)
Out[12]: <nose.result.TextTestResult run=6136 errors=0 failures=0>

[rgommers]

That should be enough info. It'll just be hard to reproduce, given that very few people have that much RAM. I just get a ValueError with current master:

D = M.todense()

/home/rgommers/Code/scipy/scipy/sparse/base.pyc in todense(self, order, out)
    513             `numpy.matrix` object that shares the same memory.
    514         """
--> 515         return np.asmatrix(self.toarray(order=order, out=out))
    516 
    517     def toarray(self, order=None, out=None):

/home/rgommers/Code/scipy/scipy/sparse/coo.pyc in toarray(self, order, out)
    236     def toarray(self, order=None, out=None):
    237         """See the docstring for `spmatrix.toarray`."""
--> 238         B = self._process_toarray_args(order, out)
    239         fortran = int(B.flags.f_contiguous)
    240         if not fortran and not B.flags.c_contiguous:

/home/rgommers/Code/scipy/scipy/sparse/base.pyc in _process_toarray_args(self, order, out)
    640             return out
    641         else:
--> 642             return np.zeros(self.shape, dtype=self.dtype, order=order)
    643 
    644 

ValueError: array is too big.

If you can produce a segfault with an array that would fit into 8Gb, that would help a lot.

[willbuckner]

I will experiment and see if I can make it happen with 8GB. If not, I'm
sure I could set you up with an account on one of my large-ram machines at
work, as we're very interested in getting this fixed. I've also got a bit
of C debugging and development experience, so I can definitely run some
tests for you/help debug if you point me in the right direction/etc. I was
able to reproduce this on a machine with 68GB of ram, and am currently
experiencing it on my 250GB machine.

We're using SciPy for some big data ML applications and working with dense
200GB matrices in RAM is something we want to do :). Thanks for looking
into this!

On Wed, Jun 5, 2013 at 12:07 PM, Ralf Gommers notifications@github.comwrote:

That should be enough info. It'll just be hard to reproduce, given that
very few people have that much RAM. I just get a ValueError with current
master:

D = M.todense()

/home/rgommers/Code/scipy/scipy/sparse/base.pyc in todense(self, order, out)
513 numpy.matrix object that shares the same memory.
514 """
--> 515 return np.asmatrix(self.toarray(order=order, out=out))
516
517 def toarray(self, order=None, out=None):

/home/rgommers/Code/scipy/scipy/sparse/coo.pyc in toarray(self, order, out)
236 def toarray(self, order=None, out=None):
237 """See the docstring for spmatrix.toarray."""
--> 238 B = self._process_toarray_args(order, out)
239 fortran = int(B.flags.f_contiguous)
240 if not fortran and not B.flags.c_contiguous:

/home/rgommers/Code/scipy/scipy/sparse/base.pyc in _process_toarray_args(self, order, out)
640 return out
641 else:
--> 642 return np.zeros(self.shape, dtype=self.dtype, order=order)
643
644

ValueError: array is too big.

If you can produce a segfault with an array that would fit into 8Gb, that
would help a lot.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2179#issuecomment-19001135
.

[rgommers]

Could you try to get a backtrace with gdb? http://www.scipy.org/scipylib/bug-report.html describes how to do that.

[rgommers]

Out of interest, what company/application (if you're allowed to share that)?

[pv]

(Have now a machine with enough memory to test this:) This is a 32-bit integer overflow issue, and fixed by PR gh-3224

[jnothman]

Is this fixed?

[rgommers]

gh-3224 was merged, so yes.