Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ImportError: undefined symbol: cblas_daxpy #220

Closed
Fhechene opened this issue Apr 10, 2017 · 31 comments · Fixed by #249
Closed

ImportError: undefined symbol: cblas_daxpy #220

Fhechene opened this issue Apr 10, 2017 · 31 comments · Fixed by #249
Assignees

Comments

@Fhechene
Copy link

Hi!
I've managed to install HORTON, but I have an issue.
When I try to import it to python I get the following error
ImportError: /usr/local/lib/python2.7/dist-packages/horton/gbasis/cext.so:
undefined symbol: cblas_daxpy

Any idea why could this be?

@tovrstra
Copy link
Member

There could be several reasons, depending on the operating system you are using and other factors. Which OS are you working on? Which steps did you follow to install HORTON, i.e. which commands did you use on the command line, including options given to ./setup.py?

In essence, the problem is that the file /usr/local/lib/python2.7/dist-packages/horton/gbasis/cext.so was dynamically linked against a BLAS library at compile time but that, at the time you import horton, the BLAS library can no longer be found. Most of the workarounds are explained here: http://theochem.github.io/horton/2.1.0b1/user_download_and_install_setup_cfg.html

@tovrstra tovrstra self-assigned this Apr 10, 2017
@Fhechene
Copy link
Author

I have Linux Mint 64
It seems that I screwed up even more. I installed horton a few weeks ago, so I can't remember what exactly did I do. Because of that, I ran again the setup as ./setup.py install --user. Then, I remembered I had to edit some files, to correct the libint2 path and finally, after a few minutes, I imported horton and got a No module named cext error.

@tovrstra
Copy link
Member

Oops. Linux Mint is similar to Ubuntu, so the Ubuntu instructions should work reasonably well, not sure which Ubuntu version though. The file to be edited is setup.cfg. There are a few examples in data/setup_cfgs to get you started. If you run one of the distributions in that directory, the right file is picked up automatically but we don't cover all distributions.

@Fhechene
Copy link
Author

But now cext isn't even recognized. Anyway, LM uses the Ubuntu core, so there should be no problem, but one never knows...

@kimt33
Copy link
Contributor

kimt33 commented Apr 15, 2017

@Fhechene Could you try to install it again and post the first few messages that pop up when you install it? It should look something like the following

MACHINE=Linux-Fedora-25-x86_64
LIBXC Configuration
  From setup.cfg
      -
  From environment variables
      -
  Static lib config in ${QAWORKDIR}
      extra_objects: ['qaworkdir/cached/libxc-2.2.2/lib/libxc.a']
      include_dirs: ['qaworkdir/cached/libxc-2.2.2/include']
  Final
      extra_objects: ['qaworkdir/cached/libxc-2.2.2/lib/libxc.a']
      include_dirs: ['qaworkdir/cached/libxc-2.2.2/include']
LIBINT2 Configuration
  From setup.cfg
      include_dirs: ['/usr/include/libint2']
      libraries: ['int2']
  From environment variables
      -
  Final
      include_dirs: ['/usr/include/libint2']
      libraries: ['int2']
BLAS Configuration
  From setup.cfg
      include_dirs: ['/usr/include/openblas/']
      libraries: ['openblas']
      library_dirs: ['/lib64/']
  From environment variables
      -
  Final
      include_dirs: ['/usr/include/openblas/']
      libraries: ['openblas']
      library_dirs: ['/lib64/']
BLAS precompiler directive: -DBLAS_OPENBLAS
Version of dependencies:
            breathe: -- not found --
            coverage: /bin/sh: coverage: command not found
            cppcheck: -- not found --
                curl: curl 7.50.3 (x86_64-redhat-linux-gnu) libcurl/7.50.3 NSS/3.26 zlib/1.2.8 libidn/1.33 libpsl/0.14.0 (+libidn/1.33) libssh2/1.8.0 nghttp2/1.13.0
              cython: Cython version 0.25.2
            docutils: -- not found --
            doxygen: -- not found --
                gcc: gcc (GCC) 6.2.1 20160916 (Red Hat 6.2.1-2)
                git: git version 2.9.3
          gitpython: -- not found --
                h5py: 2.7.0
          matplotlib: 2.0.0
                nose: nosetests version 1.3.7
              numpy: 1.12.1
                pip: pip 9.0.1 from /usr/lib/python2.7/site-packages (python 2.7)
        pycodestyle: -- not found --
          pydocstyle: -- not found --
              pylint: 
              python: Python 2.7.13
              scipy: 0.19.0
              sphinx: -- not found --
    sphinx_rtd_theme: -- not found --
              sympy: 1.0

@Fhechene
Copy link
Author

I did create a LinuxMint-18-x86_64 cfg file. Copied the ubuntu 16.04 86_64 file and changed the name.
MACHINE=Linux-LinuxMint-18-x86_64
LIBXC Configuration
File setup.cfg not found. Skipping.
From environment variables
-
Static lib not found in ${QAWORKDIR}
extra_objects: ['qaworkdir/cached/libxc-2.2.2/lib/libxc.a']
include_dirs: ['qaworkdir/cached/libxc-2.2.2/include']
pkg-config failed.
File data/setup_cfgs/setup.Linux-LinuxMint-18-x86_64.cfg not found. Skipping.
Last resort fallback plan
libraries: ['xc']
Final
libraries: ['xc']
LIBINT2 Configuration
File setup.cfg not found. Skipping.
From environment variables
-
Static lib not found in ${QAWORKDIR}
extra_objects: ['qaworkdir/cached/libint-2.0.3/lib/libint2.a']
include_dirs: ['qaworkdir/cached/libint-2.0.3/include/libint2']
pkg-config failed.
File data/setup_cfgs/setup.Linux-LinuxMint-18-x86_64.cfg not found. Skipping.
Last resort fallback plan
libraries: ['int2']
Final
libraries: ['int2']
BLAS Configuration
File setup.cfg not found. Skipping.
From environment variables
-
No static config available for this library
pkg-config failed.
File data/setup_cfgs/setup.Linux-LinuxMint-18-x86_64.cfg not found. Skipping.
Last resort fallback plan
libraries: ['atlas']
Final
libraries: ['atlas']
BLAS precompiler directive: -DBLAS_ATLAS
Version of dependencies:
breathe: -- not found --
coverage: /bin/sh: 1: coverage: not found
cppcheck: -- not found --
curl: curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3
cython: Cython version 0.23.4
docutils: -- not found --
doxygen: -- not found --
gcc: gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
git: git version 2.7.4
gitpython: -- not found --
h5py: 2.6.0
matplotlib: 1.5.1
nose: nosetests version 1.3.7
numpy: 1.11.0
pip: pip 9.0.1 from /usr/local/lib/python2.7/dist-packages (python 2.7)
pycodestyle: -- not found --
pydocstyle: -- not found --
pylint:
python: Python 2.7.12
scipy: 0.17.0
sphinx: -- not found --
sphinx_rtd_theme: -- not found --
sympy: 0.7.6.1

@kimt33
Copy link
Contributor

kimt33 commented Apr 18, 2017

@Fhechene I'm not too sure about Mint Linux, but atlas normally has its own include directory (especially if it doesn't ship with the OS). Could you double check that you have the header files for atlas somewhere? It should be somewhere like /usr/include/atlas. Once you find it, add it to the blas section of your cfg file (e.g. include_dirs = /usr/include/atlas).
You might also need a fresh installation (delete ~/.local/lib/python2.7/site-packages/horton, ~/.local/share/horton, ~/.local/bin/horton* or something like this), just in case.

@Fhechene
Copy link
Author

Fhechene commented Apr 19, 2017

Managed to install again HORTON. Turns out the cfg file had a cfg.cfg file. I guess I should be more careful. Any way, I checked the atlas headers folder. It's filled with .h files, so that should be all right. Added it to the blas section in the cfg file.

Now again, I arrived to
ImportError: /usr/local/lib/python2.7/dist-packages/horton/gbasis/cext.so:
undefined symbol: cblas_daxpy

Edit: I found a CBLAS installation folder, which I downloaded a few years ago. Tried to install it. Installed HORTON again. Returned to
No module named cext error.

@kimt33
Copy link
Contributor

kimt33 commented Apr 20, 2017

Could you double check that when you install HORTON, the configuration points to the appropriate directories: libxc should have include_dirs = qaworkdir/cached/libxc-2.2.2/include, libraries = xc;
libint should have include_dirs = /usr/include/libint2 or include_dirs = /qaworkdir/cached/libint-2.0.3/include/libint2 and libraries = int2; and blas should have include_dirs = atlas_directory, library_dirs = library_directory and libraries = atlas. You should double check that your blas include directory contains a file called cblas.h (what your horton cannot seem to find).
You can't just reference a cblas library b/c you need the rest of the blas library (I think) to build horton.

If this fails, you can always use some other blas package (Atlas seems to give a lot of headache to non-Fedora users), for example, OpenBlas. You need to find OpenBlas, e.g. ldconfig -p | grep openblas (or some other method of finding libraries), and link it up like Atlas. See http://theochem.github.io/horton/2.1.0b1/user_download_and_install_setup_cfg.html for more details.

If that also fails, you can use conda to install horton. First, download and install conda for Python2.7 (https://conda.io/docs/get-started.html). Create an environment for horton and install it, conda create --name horton-env -c pyqc horton. To activate the environment, source activate horton-env.

@Fhechene
Copy link
Author

Fhechene commented Apr 21, 2017

I had to recruit my computing engineer cousin. We managed to pinpont the error to having two versions of numpy installed: one from anaconda and the other from pip. Now, I have the following error
undefined symbol: PyFPE_jbuf
No matter if I install or uninstall numpy or python or cython, nothing happens. I keep getting that error.

@Fhechene
Copy link
Author

Fhechene commented Apr 23, 2017

Trying to install the different packages made my Linux distro to not start. Had to format my notebook. I've downloaded again the horton-2.0.1.tar.gz file. I've found that there is no /tools/qa/install_libxc-2.2.2.sh file. I couldn't find libint.sh either, so I downloaded the horton-master. Let's see what happens.
Edit 1: used those both files, but when I tried to install horton from the master folder, couldn't find some .h files. Tried to instal from the 2.0.1 folder and there is no json file.
Edit 2: also, there are many erros like warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++

@kimt33
Copy link
Contributor

kimt33 commented Apr 24, 2017

When you are installing things, you should only give sudo to your main package manager (apt-get in your case I think). Otherwise, you really should install things locally (for pip, pip install --user package) or within virtual environments (this is quite convenient with conda). If something goes wrong, you don't have to reinstall everything and you can just delete the directory (~/.local/lib/python2.7/site-packages or your virtual environment). Also, do not to give conda sudo, because it wreaks havoc on all your paths and whatnot.

If you suspect that you have the same module installed multiple times and you don't know which you are using, enter python interpreter, import module and module.__file__.

@matt-chan @tovrstra There isn't install_libxc or install_libint in the link provided in the horton 2.0.1 documentation (https://github.com/theochem/horton/releases/download/2.0.1/horton-2.0.1.tar.gz). It is also missing the dependencies.json.

To download and install libint,

mkdir qaworkdir
cd qaworkdir
curl -OL 'http://downloads.sourceforge.net/project/libint/libint-for-mpqc/libint-2.0.3-stable.tgz'
tar -xzf libint-2.0.3-stable.tgz
cd libint-2.0.3-stable
CONFIG_SHELL=$(which sh) CFLAGS='-fPIC' CPPFLAGS='-fPIC' ./configure --with-cxx-optflags='-O1' --prefix=../cached/libint-2.0.3-stable/
make install

To download and install libxc,

mkdir -p qaworkdir
cd qaworkdir
curl -OL 'http://www.tddft.org/programs/octopus/down.php?file=libxc/libxc-2.2.2.tar.gz'
tar -xzf libxc-2.2.2.tar.gz
cd libxc-2.2.2
CFLAGS='-fPIC' CPPFLAGS='-fPIC' FCCPP=-ffreestanding ./configure --prefix=../cached/libxc-2.2.2/
make install

@kimt33
Copy link
Contributor

kimt33 commented Apr 24, 2017

@Fhechene It seems that there are some files missing in the https://github.com/theochem/horton/releases/download/2.0.1/horton-2.0.1.tar.gz, so for now just work off of horton master (https://github.com/theochem/horton/archive/master.zip).
If you are missing header files, you need to check that your setup.cfg is set up properly, CPATH variable is properly set, and that the dependencies are installed. Normally, these things are set up right away, but you can get some crazy behaviours with multiple package managers (and other stuff). Have you tried installing horton using conda?
I'll try to use Mint to see if I can reproduce your errors at some point.

@Fhechene
Copy link
Author

Fhechene commented Apr 24, 2017

I managed to find an old horton-2.0.1 tar.gz, which has all of the above files (.h's, dependencies.json, libxc and libint install files) and didn't give me warnings about c++. Now, I'm back to step one: ImportError: No module named cext

Edit: I tried to use the master folder, but it gave lot's of .h missing files and warning: command line option ‘stuffs’ is valid for C/ObjC but not for C++ warnings, which the file I've just found doesn't give me.

@tczorro
Copy link

tczorro commented Apr 24, 2017

This problem import error problem happened to centOS before because the new version of Atlas(3.10.1) on CentOS does not contain libcblas.so.3 while the old one(3.8.4) does.
Here are some old tickets
jovigb/nwatechsummit-2015#1
fommil/netlib-java#66

I also met this problem when I tried to install HORTON on my raspberry pi(Raspbian). I replaced atlas with openblas and the problem solved since I couldn't find the old version.

To double check, @Fhechene could you run ldconfig -p | grep blas and paste the output here

@Fhechene
Copy link
Author

@tczorro
So, after make, what do I do?

@tczorro
Copy link

tczorro commented Apr 24, 2017

@Fhechene Could you please do ldconfig -p | grep blas and paste the result. I am curious about does mint have the same problem with centOS

@Fhechene
Copy link
Author

libf77blas.so.3 (libc6,x86-64) => /usr/lib/libf77blas.so.3 libf77blas.so (libc6,x86-64) => /usr/lib/libf77blas.so libcblas.so.3 (libc6,x86-64) => /usr/lib/libcblas.so.3 libcblas.so (libc6,x86-64) => /usr/lib/libcblas.so libblas.so.3 (libc6,x86-64) => /usr/lib/libblas.so.3 libblas.so (libc6,x86-64) => /usr/lib/libblas.so By the way, I installed OpenBLAS with make PATH=/stuff and then changed the LinuxMint Config file. As I executed setup.py, I got
LAS Configuration File setup.cfg not found. Skipping. From environment variables - No static config available for this library pkg-config failed. From data/setup_cfgs/setup.Linux-LinuxMint-18.1-x86_64.cfg include_dirs: ['/home/federico/OpenBLAS-0.2.19'] libraries: ['blas'] Final include_dirs: ['/home/federico/OpenBLAS-0.2.19'] libraries: ['blas'] Unknown BLAS implementation. Assuming Netlib-compatible headers.

@tczorro
Copy link

tczorro commented Apr 24, 2017

@Fhechene Ah OK, you have installed openblas already. libcblas.so.3 is there. That's is good. Could you change the blas in your config into openblas and run it again

@Fhechene
Copy link
Author

Done. No luck.

@tczorro
Copy link

tczorro commented Apr 24, 2017

@Fhechene If it works, could you run the nosetests, just make sure anything function properly.

@Fhechene
Copy link
Author

Fhechene commented Apr 24, 2017

`Failure: ImportError (/home/federico/.local/lib/python2.7/site-packages/horton/gbasis/cext.so: undefined symbol: cblas_daxpy) ... ERROR

======================================================================
ERROR: Failure: ImportError (/home/federico/.local/lib/python2.7/site-packages/horton/gbasis/cext.so: undefined symbol: cblas_daxpy)

Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nose/loader.py", line 407, in loadTestsFromName
module = resolve_name(addr.module)
File "/usr/lib/python2.7/dist-packages/nose/util.py", line 312, in resolve_name
module = import('.'.join(parts_copy))
File "/home/federico/.local/lib/python2.7/site-packages/horton/init.py", line 33, in
from horton.part import *
File "/home/federico/.local/lib/python2.7/site-packages/horton/part/init.py", line 31, in
from horton.part.mulliken import *
File "/home/federico/.local/lib/python2.7/site-packages/horton/part/mulliken.py", line 26, in
from horton.gbasis.cext import get_shell_nbasis
File "/home/federico/.local/lib/python2.7/site-packages/horton/gbasis/init.py", line 24, in
from horton.gbasis.cext import *
ImportError: /home/federico/.local/lib/python2.7/site-packages/horton/gbasis/cext.so: undefined symbol: cblas_daxpy


Ran 1 test in 0.001s

FAILED (errors=1)`
I suspect that cython has something to do with it, but I'm not sure.

@tczorro
Copy link

tczorro commented Apr 24, 2017

@Fhechene Hmm, not sure why this keep happening. David and I will try to solve it.
Some suggestion might help,
clean/remove installed openblas or atlas
use apt-get install libopenblas-dev
change config file libraries=openblas
then run setup.py
by the way, try build_ext -i other than install

@Fhechene
Copy link
Author

With build_ext -i I get }
/horton/gbasis/libint2.h:31:50: fatal error: /usr/include/libint2/intrinsic_types.h: File or folder doesn't exist compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
besides a bunch of warnings like
warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++

I installed libopenblas-dev, and nothing

@tczorro
Copy link

tczorro commented Apr 24, 2017

Execute /tools/qa/install_libxc-2.2.2.sh and /tools/qa/install_libint-2.0.3.sh
Forget about build_ext. Just use install.

@Fhechene
Copy link
Author

Fhechene commented Apr 24, 2017

Input: ./tools/qa/install_libxc-2.2.2.sh
Output: Using Cached libxc-2.2.2

Input: ./tools/qa/install_libint-2.0.3.sh
Output: Using Cached libint-2.0.3

After building and installing again, I still get undefined symbol: cblas_daxpy

@tczorro
Copy link

tczorro commented Apr 24, 2017

@Fhechene change config file libraries=openblas and run .setup install again. If failed, please paste the configuration info at the beginning of installation.

@Fhechene
Copy link
Author

MACHINE=Linux-LinuxMint-18.1-x86_64
LIBXC Configuration
File setup.cfg not found. Skipping.
From environment variables
-
Static lib config in ${QAWORKDIR}
extra_objects: ['qaworkdir/cached/libxc-2.2.2/lib/libxc.a']
include_dirs: ['qaworkdir/cached/libxc-2.2.2/include']
Final
extra_objects: ['qaworkdir/cached/libxc-2.2.2/lib/libxc.a']
include_dirs: ['qaworkdir/cached/libxc-2.2.2/include']
LIBINT2 Configuration
File setup.cfg not found. Skipping.
From environment variables
-
Static lib config in ${QAWORKDIR}
extra_objects: ['qaworkdir/cached/libint-2.0.3/lib/libint2.a']
include_dirs: ['qaworkdir/cached/libint-2.0.3/include/libint2']
Final
extra_objects: ['qaworkdir/cached/libint-2.0.3/lib/libint2.a']
include_dirs: ['qaworkdir/cached/libint-2.0.3/include/libint2']
BLAS Configuration
File setup.cfg not found. Skipping.
From environment variables
-
No static config available for this library
pkg-config failed.
From data/setup_cfgs/setup.Linux-LinuxMint-18.1-x86_64.cfg
include_dirs: ['/home/federico/OpenBLAS-0.2.19']
libraries: ['openblas']
Final
include_dirs: ['/home/federico/OpenBLAS-0.2.19']
libraries: ['openblas']
BLAS precompiler directive: -DBLAS_OPENBLAS
/usr/lib/python2.7/distutils/extension.py:133: UserWarning: Unknown Extension options: 'cython_directives'
warnings.warn(msg)
running install

@tczorro
Copy link

tczorro commented Apr 24, 2017

@Fhechene Could you please change include_dirs: ['/home/federico/OpenBLAS-0.2.19'] from your cfg. to /usr/lib/openblas-baseand
then, run ./setup.py install again.
If it install properly, open a new tab and run python, then import horton

@Fhechene
Copy link
Author

@tczorro I changed it and still get the same error.
Now, I'll add something that may be of use.
If I execute python setup.py build or ./setup.py build I get the same messages from the terminal, BUT,
there is a difference between python setup.py install and ./setup.py install. The later gives me the following error
error: [Errno 1] Operation not permitted: '/usr/local/bin/horton-esp-test.py'

@tovrstra
Copy link
Member

I'm going to close this issue because the problem is user configuration of dynamic link libraries, not a specific problem in HORTON. Some of the problems mentioned above have been solved in the development branch and are being backported to 2.0.1. See #232

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants