missing cblas dependency for utils/arrayfuncs.so on FreeBSD #860

Closed
ndwns opened this Issue May 16, 2012 · 23 comments

Comments

Projects
None yet
6 participants
@ndwns

ndwns commented May 16, 2012

pip install installs ok but there's a missing runtime dependency on cblas

Python 2.7.3 (default, May  3 2012, 05:58:58) 
[GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd9
Type "help", "copyright", "credits" or "license" for more information.
>>> from sklearn.decomposition import NMF
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/sklearn/decomposition/__init__.py", line 10, in <module>
    from .sparse_pca import SparsePCA, MiniBatchSparsePCA
  File "/usr/local/lib/python2.7/site-packages/sklearn/decomposition/sparse_pca.py", line 8, in <module>
    from ..linear_model import ridge_regression
  File "/usr/local/lib/python2.7/site-packages/sklearn/linear_model/__init__.py", line 15, in <module>
    from .least_angle import Lars, LassoLars, lars_path, LARS, LassoLARS, \
  File "/usr/local/lib/python2.7/site-packages/sklearn/linear_model/least_angle.py", line 20, in <module>
    from ..utils import array2d, arrayfuncs, deprecated
ImportError: /usr/local/lib/python2.7/site-packages/sklearn/utils/arrayfuncs.so: Undefined symbol "cblas_srotg"

cblas_srotg is in:

[root@ /]# nm /usr/local/lib/libcblas.so | grep cblas_srotg
0000000000005480 T cblas_srotg

but arrayfuncs.so doesn't have libcblas as a dependency:

[root@ /]# ldd /usr/local/lib/python2.7/site-packages/sklearn/utils/arrayfuncs.so 
/usr/local/lib/python2.7/site-packages/sklearn/utils/arrayfuncs.so:
        liblapack.so.4 => /usr/local/lib/liblapack.so.4 (0x801208000)
        libblas.so.2 => /usr/local/lib/libblas.so.2 (0x801a0d000)
        libm.so.5 => /lib/libm.so.5 (0x801c60000)
        libthr.so.3 => /lib/libthr.so.3 (0x801e81000)
        libc.so.7 => /lib/libc.so.7 (0x80084a000)
        libgfortran.so.3 => /usr/local/lib/gcc46/libgfortran.so.3 (0x8020a4000)
        libgcc_s.so.1 => /usr/local/lib/gcc46/libgcc_s.so.1 (0x8023b8000)
        libquadmath.so.0 => /usr/local/lib/gcc46/libquadmath.so.0 (0x8025cd000)

FreeBSD 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Jan 3 07:46:30 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
`

@fabianp

This comment has been minimized.

Show comment Hide comment
@fabianp

fabianp May 16, 2012

Member

Hi Ian,

could you post the output of from numpy.distutils.system_info import get_info; print get_info('blas_opt', 0)

Thanks,

Fabian

Member

fabianp commented May 16, 2012

Hi Ian,

could you post the output of from numpy.distutils.system_info import get_info; print get_info('blas_opt', 0)

Thanks,

Fabian

@ndwns

This comment has been minimized.

Show comment Hide comment
@ndwns

ndwns May 16, 2012

Fabian,

output is below - thanks

>>> from numpy.distutils.system_info import get_info; print get_info('blas_opt', 0)
Found executable /usr/local/bin/gfortran46
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
{'libraries': ['lapack', 'blas'], 'library_dirs': ['/usr/local/lib'], 'define_macros': [('NO_ATLAS_INFO', -1)], 'language': 'c', 'include_dirs': ['/usr/local/include']}

ndwns commented May 16, 2012

Fabian,

output is below - thanks

>>> from numpy.distutils.system_info import get_info; print get_info('blas_opt', 0)
Found executable /usr/local/bin/gfortran46
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
{'libraries': ['lapack', 'blas'], 'library_dirs': ['/usr/local/lib'], 'define_macros': [('NO_ATLAS_INFO', -1)], 'language': 'c', 'include_dirs': ['/usr/local/include']}
@juliesls-dev

This comment has been minimized.

Show comment Hide comment
@juliesls-dev

juliesls-dev Jun 6, 2012

Hi!

Any news on this issue? I am exactly in the same situation...
Thanks :)

Julie

Hi!

Any news on this issue? I am exactly in the same situation...
Thanks :)

Julie

@ndwns

This comment has been minimized.

Show comment Hide comment
@ndwns

ndwns Jun 6, 2012

Not really unfortunately. We worked around it by not loading the sparse_pca component, but this surely breaks things (that didn't affect our usecase):

sed -i '' '/.sparse_pca/,+5d' /usr/local/lib/python2.7/site-packages/sklearn/decomposition/__init__.py

ndwns commented Jun 6, 2012

Not really unfortunately. We worked around it by not loading the sparse_pca component, but this surely breaks things (that didn't affect our usecase):

sed -i '' '/.sparse_pca/,+5d' /usr/local/lib/python2.7/site-packages/sklearn/decomposition/__init__.py
@juliesls-dev

This comment has been minimized.

Show comment Hide comment
@juliesls-dev

juliesls-dev Jun 7, 2012

Ok thanks anyway. I'll try to figure something out...

Ok thanks anyway. I'll try to figure something out...

@amueller

This comment has been minimized.

Show comment Hide comment
@amueller

amueller Jun 13, 2012

Member

@nwns @juliesoulas I'm not so familiar with this but maybe it would help if you provided the build command that is used on your box with make 2>/dev/null | grep arrayfuncs. On my box, arrayfuncs is linked against blas....

Member

amueller commented Jun 13, 2012

@nwns @juliesoulas I'm not so familiar with this but maybe it would help if you provided the build command that is used on your box with make 2>/dev/null | grep arrayfuncs. On my box, arrayfuncs is linked against blas....

@shishirpy

This comment has been minimized.

Show comment Hide comment
@shishirpy

shishirpy Jul 10, 2012

Hey,
I want to use NMF on windows. Having the same trouble. Is there a solution to it?

Hey,
I want to use NMF on windows. Having the same trouble. Is there a solution to it?

@jfrelinger

This comment has been minimized.

Show comment Hide comment
@jfrelinger

jfrelinger Jul 16, 2012

I ran into this problem in the middle of the scikit-learn tutorial at scipy. I managed to fix it by building for source
adding the the build command 'python setup.py build_ext -lgslcblas -L/usr/local/lib'. You'll probably want to swap cblas for gslcblas if you have atlas installed. Seems to work around the issue for me now.

I ran into this problem in the middle of the scikit-learn tutorial at scipy. I managed to fix it by building for source
adding the the build command 'python setup.py build_ext -lgslcblas -L/usr/local/lib'. You'll probably want to swap cblas for gslcblas if you have atlas installed. Seems to work around the issue for me now.

@amueller

This comment has been minimized.

Show comment Hide comment
@amueller

amueller Jul 16, 2012

Member

@jfrelinger could you give us the linker flags you had before? The setup.py should set the linker flags for blas.

The return of from numpy.distutils.system_info import get_info; print get_info('blas_opt', 0) is used for linking blas.

Can you run that?

To fix this, we need to understand what went wrong.

Member

amueller commented Jul 16, 2012

@jfrelinger could you give us the linker flags you had before? The setup.py should set the linker flags for blas.

The return of from numpy.distutils.system_info import get_info; print get_info('blas_opt', 0) is used for linking blas.

Can you run that?

To fix this, we need to understand what went wrong.

@jfrelinger

This comment has been minimized.

Show comment Hide comment
@jfrelinger

jfrelinger Jul 16, 2012

[jolly@moku Matplotlib Tutorial]$ python
Python 2.7.3 (default, Jun  3 2012, 20:27:23) 
[GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd9
Type "help", "copyright", "credits" or "license" for more information.
>>> from numpy.distutils.system_info import get_info; print get_info('blas_opt', 0)
/usr/local/lib/python2.7/site-packages/numpy/distutils/system_info.py:528: UserWarning: Specified path /usr/local/include/suitesparse is invalid.
  warnings.warn('Specified path %s is invalid.' % d)
Found executable /usr/local/bin/gfortran46
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
_configtest.o: In function `main':
_configtest.c:(.text+0x5): undefined reference to `ATL_buildinfo'
_configtest.o: In function `main':
_configtest.c:(.text+0x5): undefined reference to `ATL_buildinfo'
{'libraries': ['lapack', 'blas'], 'library_dirs': ['/usr/local/lib'], 'define_macros': [('ATLAS_INFO', '"\\"None\\""')], 'language': 'c'}
[jolly@moku Matplotlib Tutorial]$ python
Python 2.7.3 (default, Jun  3 2012, 20:27:23) 
[GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd9
Type "help", "copyright", "credits" or "license" for more information.
>>> from numpy.distutils.system_info import get_info; print get_info('blas_opt', 0)
/usr/local/lib/python2.7/site-packages/numpy/distutils/system_info.py:528: UserWarning: Specified path /usr/local/include/suitesparse is invalid.
  warnings.warn('Specified path %s is invalid.' % d)
Found executable /usr/local/bin/gfortran46
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
_configtest.o: In function `main':
_configtest.c:(.text+0x5): undefined reference to `ATL_buildinfo'
_configtest.o: In function `main':
_configtest.c:(.text+0x5): undefined reference to `ATL_buildinfo'
{'libraries': ['lapack', 'blas'], 'library_dirs': ['/usr/local/lib'], 'define_macros': [('ATLAS_INFO', '"\\"None\\""')], 'language': 'c'}
@amueller

This comment has been minimized.

Show comment Hide comment
@amueller

amueller Jul 16, 2012

Member

Ok, I think the problem is define_macros': [('ATLAS_INFO', '"\\"None\\""')]

You don't have atlas installed, right? The script doesn't detect correctly that this is the case.

Member

amueller commented Jul 16, 2012

Ok, I think the problem is define_macros': [('ATLAS_INFO', '"\\"None\\""')]

You don't have atlas installed, right? The script doesn't detect correctly that this is the case.

@jfrelinger

This comment has been minimized.

Show comment Hide comment
@jfrelinger

jfrelinger Jul 16, 2012

yeah, atlas isn't installed. The freebsd port for numpy has an option to link against atlas, but defaults to linking against lapack and blas if atlas isn't installed.

yeah, atlas isn't installed. The freebsd port for numpy has an option to link against atlas, but defaults to linking against lapack and blas if atlas isn't installed.

@amueller

This comment has been minimized.

Show comment Hide comment
@amueller

amueller Jul 16, 2012

Member

@jfrelinger Are you working on current master? I have a branch that might fix your issue ready in around 5 minutes.

Member

amueller commented Jul 16, 2012

@jfrelinger Are you working on current master? I have a branch that might fix your issue ready in around 5 minutes.

@amueller

This comment has been minimized.

Show comment Hide comment
@amueller

amueller Jul 16, 2012

Member
@jfrelinger

This comment has been minimized.

Show comment Hide comment
@jfrelinger

jfrelinger Jul 16, 2012

no I'm not on master, I downloaded scikit-learn-0.11.tar.gz. I'll give the new branch a try later tonight.

no I'm not on master, I downloaded scikit-learn-0.11.tar.gz. I'll give the new branch a try later tonight.

@amueller

This comment has been minimized.

Show comment Hide comment
@amueller

amueller Jul 16, 2012

Member

@ndwns If you wouldn't mind, it would be great if you could have a try, too.

Member

amueller commented Jul 16, 2012

@ndwns If you wouldn't mind, it would be great if you could have a try, too.

@amueller

This comment has been minimized.

Show comment Hide comment
@amueller

amueller Jul 16, 2012

Member

@jfrelinger : cool, thanks. It doesn't link against lapack atm but that should be fixable. Just trying to figure out if I'm going in the right direction.

Member

amueller commented Jul 16, 2012

@jfrelinger : cool, thanks. It doesn't link against lapack atm but that should be fixable. Just trying to figure out if I'm going in the right direction.

@jfrelinger

This comment has been minimized.

Show comment Hide comment
@jfrelinger

jfrelinger Jul 17, 2012

branch builds fine, but since linking hasn't been fixed it still errors out with missing symbol:

ImportError: /usr/local/lib/python2.7/site-packages/scikit_learn-0.12_git-py2.7-freebsd-9.0-STABLE-amd64.egg/sklearn/linear_model/cd_fast.so: Undefined symbol "cblas_ddot"

it's not trying to link against cblas either

[jolly@moku ~]$ ldd /usr/local/lib/python2.7/site-packages/scikit_learn-0.12_git-py2.7-freebsd-9.0-STABLE-amd64.egg/sklearn/linear_model/cd_fast.so
/usr/local/lib/python2.7/site-packages/scikit_learn-0.12_git-py2.7-freebsd-9.0-STABLE-amd64.egg/sklearn/linear_model/cd_fast.so:
    liblapack.so.4 => /usr/local/lib/liblapack.so.4 (0x801221000)
    libblas.so.2 => /usr/local/lib/libblas.so.2 (0x801a26000)
    libm.so.5 => /lib/libm.so.5 (0x801c79000)
    libthr.so.3 => /lib/libthr.so.3 (0x801e9a000)
    libc.so.7 => /lib/libc.so.7 (0x80081b000)
    libgfortran.so.3 => /usr/local/lib/gcc46/libgfortran.so.3 (0x8020bd000)
    libgcc_s.so.1 => /usr/local/lib/gcc46/libgcc_s.so.1 (0x8023d1000)
    libquadmath.so.0 => /usr/local/lib/gcc46/libquadmath.so.0 (0x8025e6000)

branch builds fine, but since linking hasn't been fixed it still errors out with missing symbol:

ImportError: /usr/local/lib/python2.7/site-packages/scikit_learn-0.12_git-py2.7-freebsd-9.0-STABLE-amd64.egg/sklearn/linear_model/cd_fast.so: Undefined symbol "cblas_ddot"

it's not trying to link against cblas either

[jolly@moku ~]$ ldd /usr/local/lib/python2.7/site-packages/scikit_learn-0.12_git-py2.7-freebsd-9.0-STABLE-amd64.egg/sklearn/linear_model/cd_fast.so
/usr/local/lib/python2.7/site-packages/scikit_learn-0.12_git-py2.7-freebsd-9.0-STABLE-amd64.egg/sklearn/linear_model/cd_fast.so:
    liblapack.so.4 => /usr/local/lib/liblapack.so.4 (0x801221000)
    libblas.so.2 => /usr/local/lib/libblas.so.2 (0x801a26000)
    libm.so.5 => /lib/libm.so.5 (0x801c79000)
    libthr.so.3 => /lib/libthr.so.3 (0x801e9a000)
    libc.so.7 => /lib/libc.so.7 (0x80081b000)
    libgfortran.so.3 => /usr/local/lib/gcc46/libgfortran.so.3 (0x8020bd000)
    libgcc_s.so.1 => /usr/local/lib/gcc46/libgcc_s.so.1 (0x8023d1000)
    libquadmath.so.0 => /usr/local/lib/gcc46/libquadmath.so.0 (0x8025e6000)
@amueller

This comment has been minimized.

Show comment Hide comment
@amueller

amueller Jul 20, 2012

Member

@jfrelinger I updated my branch. In the last version, it was only fixed for arrayfunc.so, now it should be fixed for all libraries.

Member

amueller commented Jul 20, 2012

@jfrelinger I updated my branch. In the last version, it was only fixed for arrayfunc.so, now it should be fixed for all libraries.

@jfrelinger

This comment has been minimized.

Show comment Hide comment
@jfrelinger

jfrelinger Jul 23, 2012

just updated to the latest https://github.com/amueller/scikit-learn/tree/linking_arrayfuncs. Still having the same error as above.

ImportError: /usr/local/lib/python2.7/site-packages/scikit_learn-0.12_git-py2.7-freebsd-9.0-STABLE-amd64.egg/sklearn/linear_model/cd_fast.so: Undefined symbol "cblas_ddot"

just updated to the latest https://github.com/amueller/scikit-learn/tree/linking_arrayfuncs. Still having the same error as above.

ImportError: /usr/local/lib/python2.7/site-packages/scikit_learn-0.12_git-py2.7-freebsd-9.0-STABLE-amd64.egg/sklearn/linear_model/cd_fast.so: Undefined symbol "cblas_ddot"
@amueller

This comment has been minimized.

Show comment Hide comment
@amueller

amueller Jul 23, 2012

Member

@jfrelinger Thanks for the feedback. Sorry that it still doesn't work. It would be great if you could try again.

Member

amueller commented Jul 23, 2012

@jfrelinger Thanks for the feedback. Sorry that it still doesn't work. It would be great if you could try again.

@amueller

This comment has been minimized.

Show comment Hide comment
@amueller

amueller Sep 3, 2012

Member

Could any one with BSD give this another try on master?

Member

amueller commented Sep 3, 2012

Could any one with BSD give this another try on master?

@amueller

This comment has been minimized.

Show comment Hide comment
@amueller

amueller Sep 18, 2012

Member

Closing as this should be fixed now.

Member

amueller commented Sep 18, 2012

Closing as this should be fixed now.

@amueller amueller closed this Sep 18, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment