ValueError: On entry to SGESDD parameter number 12 had an illegal value

[ppalasek]

Hello,

I'm getting a ValueError from scipy/linalg/decomp_svd.py, line 104 when tring to run sklearn's pca.fit() function.

Traceback:
File "train.py", line 269, in
pca.fit(X)
File "/homes/pp305/.local/lib/python2.7/site-packages/sklearn/decomposition/pca.py", line 221, in fit
self._fit(X)
File "/homes/pp305/.local/lib/python2.7/site-packages/sklearn/decomposition/pca.py", line 271, in _fit
U, S, V = linalg.svd(X, full_matrices=False)
File "/homes/pp305/.local/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 104, in svd
full_matrices=full_matrices, overwrite_a=overwrite_a)
ValueError: On entry to SGESDD parameter number 12 had an illegal value

Versions I'm using:
Python 2.7.5
scipy 0.16.0
sklearn 0.16.1
numpy 1.9.2

print scipy.config.show()
atlas_3_10_blas_threads_info:
libraries = ['tatlas']
library_dirs = ['/usr/lib64/atlas']
define_macros = [('HAVE_CBLAS', None), ('ATLAS_INFO', '""3.10.1""')]
language = c
include_dirs = ['/usr/include']
lapack_opt_info:
libraries = ['tatlas', 'tatlas', 'tatlas']
library_dirs = ['/usr/lib64/atlas']
define_macros = [('ATLAS_INFO', '""3.10.1""')]
language = f77
include_dirs = ['/usr/include']
blas_opt_info:
libraries = ['tatlas']
library_dirs = ['/usr/lib64/atlas']
define_macros = [('HAVE_CBLAS', None), ('ATLAS_INFO', '""3.10.1""')]
language = c
include_dirs = ['/usr/include']
openblas_info:
NOT AVAILABLE
openblas_lapack_info:
NOT AVAILABLE
atlas_3_10_threads_info:
libraries = ['tatlas', 'tatlas', 'tatlas']
library_dirs = ['/usr/lib64/atlas']
define_macros = [('ATLAS_INFO', '""3.10.1""')]
language = f77
include_dirs = ['/usr/include']
lapack_mkl_info:
NOT AVAILABLE
blas_mkl_info:
NOT AVAILABLE
mkl_info:
NOT AVAILABLE
None

I found an issue like this here already:
#3895

Could you let me know how I can go about getting this to work?

Thank you!

[pv]

(i) What is your LAPACK version? If older than 3.2.2, GESDD work space query was known to have bugs. (ii) What is the size of the matrix?

[ppalasek]

Hello,

LAPACK version is 3.4.2 and the size of the matrix was 9537x46080.

Now I tried using matrices with other dimensions and I'm getting the same error when I try with matrices of sizes 9537x20000, 9537x10000, 9537x9538, 9537x9537.

If I use a matrix of size 9537x9536 it's working fine.

[pv]

LAPACK uses 32-bit integers and will start failing if the work space size becomes larger than INT_MAX. You can maybe check this via

import numpy as np
from scipy.linalg import lapack
intmax = np.iinfo(np.intc).max
n, m = 9537, 9537
lwork = lapack.sgesdd_lwork(n, n, True, True)[0]
print(lwork / intmax) # should be < 1

# Try to decompose:
a = np.random.rand(n,n)
lapack.sgesdd(a, True, True, lwork=int(lwork))

n=9537 is however not yet so big it should be problematic, at least with lapack 3.5.0, but it is not very far from the limit either. Maybe it is different on 3.4.2?

[ppalasek]

OK, I ran the code you wrote and got the same error:

python test.py
0.127092868149
Traceback (most recent call last):
File "test.py", line 10, in
lapack.sgesdd(a, True, True, lwork=int(lwork))
ValueError: On entry to SGESDD parameter number 12 had an illegal value

with n = 9536 it works.

What do you suggest? Should I just try upgrading lapack, or is there something else I can try?

Thank you!

[ppalasek]

Just to update, I tried your code again with n=9538, n=9600 and with n=10000 and it worked each time. I also tried running with n=20000 and n=40000 and it seems to be working both times (I didn't wait until it finished).

n=9537 still resulting in error (). Is there something special about 9537 or lwork being 272929856?

[pv]

These numbers come directly from LAPACK. What I think is happening is the following: LAPACK sgesdd lwork query returns the optimal lwork as a single-precision float value. However, the next possible float32 number bigger than 272929856 is actually 272929888. If the value of lwork LAPACK computed falls between these two, it gets rounded down when returned in a float32 variable and may end up being too small.

So I think:

• This is a LAPACK bug (should be reproduced with a small fortran program and reported to LAPACK authors).
• Scipy should work around it, by taking the next largest floating point value (lwork = int(np.nextafter(lwork, np.float32(np.inf))) instead of lwork = int(lwork)).

In the meantime, you can work around it by using the low-level lapack.*gesdd* routines instead of scipy.linalg.svd.

[ppalasek]

Thank you very much for your replies! I will try to use your workaround.

[pv]

LAPACK bug report: http://icl.cs.utk.edu/lapack-forum/viewtopic.php?f=13&t=4836
Workaround in gh-5413

[pv]

Appears to be fixed in the next lapack version http://www.netlib.org/lapack/Errata/index2.html#_strong_span_class_green_bug120_span_strong_workspace_query_return_too_small_value_for_xgesdd