New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shogun 5.0.0 compile/linking error PythonModular #3752

Closed
ghost opened this Issue Mar 28, 2017 · 11 comments

Comments

Projects
None yet
1 participant
@ghost

ghost commented Mar 28, 2017

Hi,
I am trying to install Shogun 5.0.0 from source on a Debian 8.7 server / kernel 4.4.41.1.amd64 inside a dedicated Conda environment (env packages in this gist )
The following cmake command

cmake
-DPythonModular=ON
-DBUILD_META_EXAMPLES=OFF
-DCMAKE_PREFIX_PATH=/TL/epigenetics2/work/pebert/conda/envs/comikl
-DCMAKE_INSTALL_PREFIX=/TL/epigenetics2/work/pebert/tools/shogun_5.0.0
-DSWIG_EXECUTABLE=/TL/epigenetics2/work/pebert/conda/envs/comikl/bin/swig
-DGENERATE_EXAMPLES=OFF

results in this cmake output - looks fine to me...
However, make does not finish w/o errors - see make output.

[ 78%] Linking CXX executable variational_approx_example
../../../src/shogun/libshogun.so.17.2: undefined reference to `dsyev_'
../../../src/shogun/libshogun.so.17.2: undefined reference to `dorgqr_'
../../../src/shogun/libshogun.so.17.2: undefined reference to `dstemr_'
../../../src/shogun/libshogun.so.17.2: undefined reference to `dsyevr_'
../../../src/shogun/libshogun.so.17.2: undefined reference to `dsygvx_'
../../../src/shogun/libshogun.so.17.2: undefined reference to `dgesvd_'

[...]

collect2: error: ld returned 1 exit status
examples/undocumented/libshogun/CMakeFiles/transfer_multitaskleastsquaresregression.dir/build.make:125: recipe for target 'examples/undocumented/libshogun/transfer_multitaskleastsquaresregression' failed
make[2]: *** [examples/undocumented/libshogun/transfer_multitaskleastsquaresregression] Error 1

[and so on]

Maybe related to #3213 ?

Thanks for your help.
Best,
Peter

@vigsterkr

This comment has been minimized.

Show comment
Hide comment
@vigsterkr

vigsterkr Mar 28, 2017

Member

hi @mpi-pebert thnx for reporting this. in case you just wanna use shogun on your jessie i'd suggest using the apt repository with the nightly packages of shogun: https://github.com/shogun-toolbox/docs/blob/master/INSTALL.md#debian-

the python interface is precompiled as well: simply install the python-shogun package.
i'm successfully using these binaries with a jupyter notebook server on http://cloud.shogun.ml (which i suggest you to try in case you just want to use shogun in a python project); here's the docker image for it:
https://hub.docker.com/r/shogun/cloud/

and here's the Dockerfile:
https://github.com/shogun-toolbox/shogun-cloud/blob/master/docker/cloud/Dockerfile

Member

vigsterkr commented Mar 28, 2017

hi @mpi-pebert thnx for reporting this. in case you just wanna use shogun on your jessie i'd suggest using the apt repository with the nightly packages of shogun: https://github.com/shogun-toolbox/docs/blob/master/INSTALL.md#debian-

the python interface is precompiled as well: simply install the python-shogun package.
i'm successfully using these binaries with a jupyter notebook server on http://cloud.shogun.ml (which i suggest you to try in case you just want to use shogun in a python project); here's the docker image for it:
https://hub.docker.com/r/shogun/cloud/

and here's the Dockerfile:
https://github.com/shogun-toolbox/shogun-cloud/blob/master/docker/cloud/Dockerfile

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Mar 28, 2017

Hi Viktor,
thanks for the quick reply - sorry, I see that I have left out essential information:
I have no admin privileges on the server (also, no Docker available), and I need to install Shogun with Python interface for a third-party tool for a somewhat large analysis task, so a (public) cloud setup is unfortunately not an option.

+Peter

ghost commented Mar 28, 2017

Hi Viktor,
thanks for the quick reply - sorry, I see that I have left out essential information:
I have no admin privileges on the server (also, no Docker available), and I need to install Shogun with Python interface for a third-party tool for a somewhat large analysis task, so a (public) cloud setup is unfortunately not an option.

+Peter

@vigsterkr

This comment has been minimized.

Show comment
Hide comment
@vigsterkr

vigsterkr Mar 28, 2017

Member

@mpi-pebert mmm okey... in that case you have still some options:

  1. do you have the dependencies (not the build but the runtime dependencies) for the libshogun17 and python-shogun package on the system? because in that case you can just simply unarchive the deb packages and use the shared libraries inside :)
  2. i suppose you are missing some lapack dependencies... so in that case i'll have to investigate a bit. or probably the lapack detection needs some love in shogun
Member

vigsterkr commented Mar 28, 2017

@mpi-pebert mmm okey... in that case you have still some options:

  1. do you have the dependencies (not the build but the runtime dependencies) for the libshogun17 and python-shogun package on the system? because in that case you can just simply unarchive the deb packages and use the shared libraries inside :)
  2. i suppose you are missing some lapack dependencies... so in that case i'll have to investigate a bit. or probably the lapack detection needs some love in shogun
@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Mar 28, 2017

  1. well, it's worth a shot - I found libshogun17_5.0.0_amd64.deb in the Trusty PPA, but I don't seem to find python-shogun, where can I get the correct deb package?
  2. any hint on what might be missing is of course appreciated

ghost commented Mar 28, 2017

  1. well, it's worth a shot - I found libshogun17_5.0.0_amd64.deb in the Trusty PPA, but I don't seem to find python-shogun, where can I get the correct deb package?
  2. any hint on what might be missing is of course appreciated
@vigsterkr

This comment has been minimized.

Show comment
Hide comment
@vigsterkr

vigsterkr Mar 29, 2017

Member

@mpi-pebert if i were you i'd use the packages in the nightly ppa... trust me it's really that much better :) + stable does not have python-shogun package yet

regarding 2.) would it be possible that you send me the output of your dpkg -l on your system? drop it via mail and i can replicate things in a docker container and maybe even fix the cmake

Member

vigsterkr commented Mar 29, 2017

@mpi-pebert if i were you i'd use the packages in the nightly ppa... trust me it's really that much better :) + stable does not have python-shogun package yet

regarding 2.) would it be possible that you send me the output of your dpkg -l on your system? drop it via mail and i can replicate things in a docker container and maybe even fix the cmake

@vigsterkr

This comment has been minimized.

Show comment
Hide comment
@vigsterkr

vigsterkr Mar 29, 2017

Member

@mpi-pebert btw just realised, you mentioned Trusty? why trusty? isn't it debian jessie 8.7 that you are using atm? if that's the case (debian jessie) use the apt.shogun.ml repository

Member

vigsterkr commented Mar 29, 2017

@mpi-pebert btw just realised, you mentioned Trusty? why trusty? isn't it debian jessie 8.7 that you are using atm? if that's the case (debian jessie) use the apt.shogun.ml repository

@vigsterkr

This comment has been minimized.

Show comment
Hide comment
@vigsterkr

vigsterkr Mar 29, 2017

Member

@mpi-pebert and another btw: have you actually tried to compile develop branch of shogun? if not could you please try it and let us know if it causes the same bug?

Member

vigsterkr commented Mar 29, 2017

@mpi-pebert and another btw: have you actually tried to compile develop branch of shogun? if not could you please try it and let us know if it causes the same bug?

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Mar 29, 2017

I managed to get my hands on libshogun18_5.0.0+1SNAPSHOT201703060902-0_amd64.deb and python-shogun_5.0.0+1SNAPSHOT201703060902-0_amd64.deb using my laptop (where I am root, otherwise same Debian in-house installation). I extracted the contents and linked the libs in my environment (is that the correct procedure? I have never done it that way before...). I just tried the Python interface, which gave me this:

>>> import modshogun
_mod = imp.load_module('_modshogun', fp, pathname, description)
ImportError: [...]../lib/libshogun.so.18: undefined symbol: cblas_dgemm

Also, I cloned the repo and tried to build the dev branch of Shogun - this is the CMake call

cmake
-DPythonModular=ON
-DBUILD_META_EXAMPLES=OFF
-DCMAKE_PREFIX_PATH=/TL/epigenetics2/work/pebert/conda/envs/comikl
-DCMAKE_INSTALL_PREFIX=/TL/epigenetics2/work/pebert/conda/envs/comikl
-DSWIG_EXECUTABLE=/TL/epigenetics2/work/pebert/conda/envs/comikl/bin/swig
-DBUILD_EXAMPLES=OFF

Again, did not result in obvious problems, yet the make resulted in the following:

SGBase.i:311: Error: Unable to find 'swig_typemaps.i'

src/interfaces/python_modular/CMakeFiles/_python_modular.dir/build.make:120: recipe for target 'src/interfaces/python_modular/modshogunPYTHON_wrap.cxx' failed

make[2]: *** [src/interfaces/python_modular/modshogunPYTHON_wrap.cxx] Error 1

CMakeFiles/Makefile2:325: recipe for target 'src/interfaces/python_modular/CMakeFiles/_python_modular.dir/all' failed

I put the full log here: make dev error

I'll send you the dpkg -l output via mail as requested.

ghost commented Mar 29, 2017

I managed to get my hands on libshogun18_5.0.0+1SNAPSHOT201703060902-0_amd64.deb and python-shogun_5.0.0+1SNAPSHOT201703060902-0_amd64.deb using my laptop (where I am root, otherwise same Debian in-house installation). I extracted the contents and linked the libs in my environment (is that the correct procedure? I have never done it that way before...). I just tried the Python interface, which gave me this:

>>> import modshogun
_mod = imp.load_module('_modshogun', fp, pathname, description)
ImportError: [...]../lib/libshogun.so.18: undefined symbol: cblas_dgemm

Also, I cloned the repo and tried to build the dev branch of Shogun - this is the CMake call

cmake
-DPythonModular=ON
-DBUILD_META_EXAMPLES=OFF
-DCMAKE_PREFIX_PATH=/TL/epigenetics2/work/pebert/conda/envs/comikl
-DCMAKE_INSTALL_PREFIX=/TL/epigenetics2/work/pebert/conda/envs/comikl
-DSWIG_EXECUTABLE=/TL/epigenetics2/work/pebert/conda/envs/comikl/bin/swig
-DBUILD_EXAMPLES=OFF

Again, did not result in obvious problems, yet the make resulted in the following:

SGBase.i:311: Error: Unable to find 'swig_typemaps.i'

src/interfaces/python_modular/CMakeFiles/_python_modular.dir/build.make:120: recipe for target 'src/interfaces/python_modular/modshogunPYTHON_wrap.cxx' failed

make[2]: *** [src/interfaces/python_modular/modshogunPYTHON_wrap.cxx] Error 1

CMakeFiles/Makefile2:325: recipe for target 'src/interfaces/python_modular/CMakeFiles/_python_modular.dir/all' failed

I put the full log here: make dev error

I'll send you the dpkg -l output via mail as requested.

@vigsterkr

This comment has been minimized.

Show comment
Hide comment
@vigsterkr

vigsterkr Mar 29, 2017

Member

@mpi-pebert thnx. i'll look into the dpkg list but the

SGBase.i:311: Error: Unable to find 'swig_typemaps.i'

is very alarming because that's basically saying that you are missing a file that is in the repository :(

could you actually check whether the file exists in src/interfaces/python_modular/swig_typemaps.i

Member

vigsterkr commented Mar 29, 2017

@mpi-pebert thnx. i'll look into the dpkg list but the

SGBase.i:311: Error: Unable to find 'swig_typemaps.i'

is very alarming because that's basically saying that you are missing a file that is in the repository :(

could you actually check whether the file exists in src/interfaces/python_modular/swig_typemaps.i

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Mar 29, 2017

Ok, I have identified a problem - I started again from scratch and confirmed that swig_typemaps.i exists; cloned the repo and compiled again; this time, it worked. That was a bit unexpected, so I played around and found that I can reproduce the Unable to find 'swig_typemaps.i' error by doing a make clean and then trying to rebuild the whole thing (i.e., swig_typemaps.i is now indeed missing from the folder, I presume that is expected behavior?)

So, technically, to get a running Shogun w/ Python interface:

  1. set up Conda environment as specified in gist
  2. clone repo
  3. build Shogun from source, set CMake paths as appropriate for environment

In case something needs to be changed

  1. delete everything
  2. clone repo again
  3. build again

The error about the 5.0.0 release version is still unsolved.
In principle, this could be closed. But if you want to track down the source of the release version error, I am happy to run some more tests/provide some more info if you need.

ghost commented Mar 29, 2017

Ok, I have identified a problem - I started again from scratch and confirmed that swig_typemaps.i exists; cloned the repo and compiled again; this time, it worked. That was a bit unexpected, so I played around and found that I can reproduce the Unable to find 'swig_typemaps.i' error by doing a make clean and then trying to rebuild the whole thing (i.e., swig_typemaps.i is now indeed missing from the folder, I presume that is expected behavior?)

So, technically, to get a running Shogun w/ Python interface:

  1. set up Conda environment as specified in gist
  2. clone repo
  3. build Shogun from source, set CMake paths as appropriate for environment

In case something needs to be changed

  1. delete everything
  2. clone repo again
  3. build again

The error about the 5.0.0 release version is still unsolved.
In principle, this could be closed. But if you want to track down the source of the release version error, I am happy to run some more tests/provide some more info if you need.

@vigsterkr

This comment has been minimized.

Show comment
Hide comment
@vigsterkr

vigsterkr Apr 1, 2017

Member

mmm yeah the lapack/cblas problem with release 5.0.0 are actually known... i was just hoping that we got it right finally in current develop. the current develop is inches away to be the next 6.0.0 release so i'm closing this now.

but i'll open a bug regarding the make clean story. thnx

Member

vigsterkr commented Apr 1, 2017

mmm yeah the lapack/cblas problem with release 5.0.0 are actually known... i was just hoping that we got it right finally in current develop. the current develop is inches away to be the next 6.0.0 release so i'm closing this now.

but i'll open a bug regarding the make clean story. thnx

@vigsterkr vigsterkr closed this Apr 1, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment