Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

coo.todense() segfaults (Trac #1654) #2179

Closed
scipy-gitbot opened this Issue Apr 25, 2013 · 11 comments

Comments

Projects
None yet
5 participants

Original ticket http://projects.scipy.org/scipy/ticket/1654 on 2012-05-03 by trac user rlm, assigned to unknown.

Zipped pickle (~7MB) of a COO matrix can be found here:

http://rlmiller.org/badM.obj.zip

To reproduce:

import cPickle
M = cPickle.load(open('badM.obj'))
D = M.todense()
Segmentation fault: 11

The setup:

$ uname -a
Darwin dev-rmiller.local 11.3.0 Darwin Kernel Version 11.3.0: Thu Jan 12 18:47:41 PST 2012; root:xnu-1699.24.23~1/RELEASE_X86_64 x86_64

scipy-0.10.0-py2.7-macosx-10.7-intel

Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin

@stefanv wrote on 2012-05-03

With 0.11.0.dev I get a MemoryError instead.

It looks like the row and column coordinates are within bounds:

np.max(M.row * M.shape[1] + M.col) > np.prod(M.shape)

But the output array would have a prohibitively large memory footprint:

np.dtype(float).itemsize * np.prod(M.shape) / 1024.**3

This is very much still an issue.

While the memory footprint may have been "prohibitively large" on rlm's machine, mine has 250GB of ram. This matrix is approximately 19GB.

In [4]: import numpy as np

In [5]: np.dtype(float).itemsize * np.prod(prM.shape) / 1024.**3
Out[5]: 19.75651279091835

In [6]: badM = cPickle.load(open('/var/xxx/badM.obj', 'rb'))

In [7]: badM.todense()
Segmentation fault

xxx@xxx:~/xxx$ free -m -g
total used free shared buffers cached
Mem: 230 59 171 0 0 51
-/+ buffers/cache: 7 223
Swap: 0 0 0

xxx@xxx:~/xxx$ ipython
Python 2.7.3 (default, Aug 1 2012, 05:14:39)

What gives?

@rgommers rgommers reopened this Jun 1, 2013

Owner

rgommers commented Jun 1, 2013

OK, reopening. This is with scipy 0.12.0 or master I assume?

@rgommers: Thanks! Yes, this is SciPy 0.12.0.

This is Ubuntu 12.04LTS, 230GB ram, 18 2.2Ghz Intel Xeon Virtual Cores. Python 2.7.3 from apt.

What other information can I provide you? Here's my show_config():

In [11]: sp.show_config()
amd_info:
libraries = ['amd']
library_dirs = ['/usr/lib']
define_macros = [('SCIPY_AMD_H', None)]
swig_opts = ['-I/usr/include/suitesparse']
include_dirs = ['/usr/include/suitesparse']
umfpack_info:
libraries = ['umfpack', 'amd']
library_dirs = ['/usr/lib']
define_macros = [('SCIPY_UMFPACK_H', None), ('SCIPY_AMD_H', None)]
swig_opts = ['-I/usr/include/suitesparse', '-I/usr/include/suitesparse']
include_dirs = ['/usr/include/suitesparse']
atlas_threads_info:
libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
define_macros = [('ATLAS_INFO', '""3.8.4""')]
language = f77
include_dirs = ['/usr/include/atlas']
blas_opt_info:
libraries = ['ptf77blas', 'ptcblas', 'atlas']
library_dirs = ['/usr/lib/atlas-base']
define_macros = [('ATLAS_INFO', '""3.8.4""')]
language = c
include_dirs = ['/usr/include/atlas']
atlas_blas_threads_info:
libraries = ['ptf77blas', 'ptcblas', 'atlas']
library_dirs = ['/usr/lib/atlas-base']
define_macros = [('ATLAS_INFO', '""3.8.4""')]
language = c
include_dirs = ['/usr/include/atlas']
lapack_opt_info:
libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
define_macros = [('ATLAS_INFO', '""3.8.4""')]
language = f77
include_dirs = ['/usr/include/atlas']
lapack_mkl_info:
NOT AVAILABLE
blas_mkl_info:
NOT AVAILABLE
mkl_info:
NOT AVAILABLE

In [12]: sp.test()
Running unit tests for scipy
NumPy version 1.7.1
NumPy is installed in /usr/local/lib/python2.7/dist-packages/numpy
SciPy version 0.12.0
SciPy is installed in /usr/local/lib/python2.7/dist-packages/scipy
Python version 2.7.3 (default, Aug 1 2012, 05:14:39) [GCC 4.6.3]
nose version 1.3.0
/usr/local/lib/python2.7/dist-packages/numpy/lib/utils.py:139: DeprecationWarning: scipy.lib.blas is deprecated, use scipy.linalg.blas instead!
warnings.warn(depdoc, DeprecationWarning)
/usr/local/lib/python2.7/dist-packages/numpy/lib/utils.py:139: DeprecationWarning: scipy.lib.lapack is deprecated, use scipy.linalg.lapack instead!
warnings.warn(depdoc, DeprecationWarning)

..............................................................................................................................................................................................................................K..............................................................................................................K....................................................................K..K............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................SSSSSS......SSSSSS......SSSS.......................................................................................................K..............................................................................................................................................................................................................................................................................................K..........................................................................................................................................................................................................................K.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K....................................................................K......................................................................................................................................................................KK..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K.K..........................K................................................................................................................................................................................................................................................................................................................................................................................K........K......................SSSSSSSS.........................................................................................................................................................................................S...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Ran 6136 tests in 75.327s

OK (KNOWNFAIL=16, SKIP=25)
Out[12]: <nose.result.TextTestResult run=6136 errors=0 failures=0>

Owner

rgommers commented Jun 5, 2013

That should be enough info. It'll just be hard to reproduce, given that very few people have that much RAM. I just get a ValueError with current master:

D = M.todense()

/home/rgommers/Code/scipy/scipy/sparse/base.pyc in todense(self, order, out)
    513             `numpy.matrix` object that shares the same memory.
    514         """
--> 515         return np.asmatrix(self.toarray(order=order, out=out))
    516 
    517     def toarray(self, order=None, out=None):

/home/rgommers/Code/scipy/scipy/sparse/coo.pyc in toarray(self, order, out)
    236     def toarray(self, order=None, out=None):
    237         """See the docstring for `spmatrix.toarray`."""
--> 238         B = self._process_toarray_args(order, out)
    239         fortran = int(B.flags.f_contiguous)
    240         if not fortran and not B.flags.c_contiguous:

/home/rgommers/Code/scipy/scipy/sparse/base.pyc in _process_toarray_args(self, order, out)
    640             return out
    641         else:
--> 642             return np.zeros(self.shape, dtype=self.dtype, order=order)
    643 
    644 

ValueError: array is too big.

If you can produce a segfault with an array that would fit into 8Gb, that would help a lot.

I will experiment and see if I can make it happen with 8GB. If not, I'm
sure I could set you up with an account on one of my large-ram machines at
work, as we're very interested in getting this fixed. I've also got a bit
of C debugging and development experience, so I can definitely run some
tests for you/help debug if you point me in the right direction/etc. I was
able to reproduce this on a machine with 68GB of ram, and am currently
experiencing it on my 250GB machine.

We're using SciPy for some big data ML applications and working with dense
200GB matrices in RAM is something we want to do :). Thanks for looking
into this!

On Wed, Jun 5, 2013 at 12:07 PM, Ralf Gommers notifications@github.comwrote:

That should be enough info. It'll just be hard to reproduce, given that
very few people have that much RAM. I just get a ValueError with current
master:

D = M.todense()

/home/rgommers/Code/scipy/scipy/sparse/base.pyc in todense(self, order, out)
513 numpy.matrix object that shares the same memory.
514 """
--> 515 return np.asmatrix(self.toarray(order=order, out=out))
516
517 def toarray(self, order=None, out=None):

/home/rgommers/Code/scipy/scipy/sparse/coo.pyc in toarray(self, order, out)
236 def toarray(self, order=None, out=None):
237 """See the docstring for spmatrix.toarray."""
--> 238 B = self._process_toarray_args(order, out)
239 fortran = int(B.flags.f_contiguous)
240 if not fortran and not B.flags.c_contiguous:

/home/rgommers/Code/scipy/scipy/sparse/base.pyc in _process_toarray_args(self, order, out)
640 return out
641 else:
--> 642 return np.zeros(self.shape, dtype=self.dtype, order=order)
643
644

ValueError: array is too big.

If you can produce a segfault with an array that would fit into 8Gb, that
would help a lot.


Reply to this email directly or view it on GitHubhttps://github.com/scipy/scipy/issues/2179#issuecomment-19001135
.

Owner

rgommers commented Jun 9, 2013

Could you try to get a backtrace with gdb? http://www.scipy.org/scipylib/bug-report.html describes how to do that.

Owner

rgommers commented Jun 9, 2013

Out of interest, what company/application (if you're allowed to share that)?

Owner

pv commented Jan 19, 2014

(Have now a machine with enough memory to test this:) This is a 32-bit integer overflow issue, and fixed by PR gh-3224

Contributor

jnothman commented Feb 16, 2014

Is this fixed?

Owner

rgommers commented Feb 16, 2014

gh-3224 was merged, so yes.

@rgommers rgommers closed this Feb 16, 2014

@rgommers rgommers added this to the 0.14.0 milestone Feb 16, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment