Error Installing on CentOS and RHEL platforms

[BrianMiner]

I followed all the install directions. Running make for the C parts seemed to finish without error but running python setup.py install creates the following error. I am on 64 bit Linux (mint). Any ideas?

python setup.py install
running install
running build
running build_py
running build_ext
skipping 'fastFM/ffm.c' Cython extension (up-to-date)
building 'ffm' extension
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -IfastFM/ -IfastFM-core/include/ -IfastFM-core/externals/CXSparse/Include/ -I/usr/include/ -I/home/anaconda/lib/python2.7/site-packages/numpy/core/include -I/home/anaconda/include/python2.7 -c fastFM/ffm.c -o build/temp.linux-x86_64-2.7/fastFM/ffm.o
In file included from /home/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1781:0,
from /home/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
from /home/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from fastFM/ffm.c:252:
/home/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by "
^
In file included from /home/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:27:0,
from /home/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from fastFM/ffm.c:252:
/home/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/__multiarray_api.h:1634:1: warning: ‘_import_array’ defined but not used [-Wunused-function]
_import_array(void)
^
gcc -pthread -shared -L/home/anaconda/lib -Wl,-rpath=/home/anaconda/lib,--no-as-needed build/temp.linux-x86_64-2.7/fastFM/ffm.o -LfastFM/ -LfastFM-core/bin/ -LfastFM-core/externals/CXSparse/Lib/ -L/usr/lib/ -L/usr/lib/atlas-base/ -L/home/anaconda/lib -lm -lfastfm -lcxsparse -lcblas -lpython2.7 -o build/lib.linux-x86_64-2.7/ffm.so
/usr/bin/ld: cannot find -lfastfm
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

[BrianMiner]

I ended up re-running everything and it "seemed" to run OK. but I get this error trying to import

from fastFM import sgd

ImportError: /home/anaconda/lib/python2.7/site-packages/fastFM/ffm.so: undefined symbol: g_array_new

[ibayer]

g_array_new is a function from the c library glib.
Please run (README was missing a dependency):
sudo apt-get install libglib2.0-dev python-dev libatlas-base-dev before
recompiling and installing fastFM.
Please let us know if that helps.

[BrianMiner]

I had run it. When I do this again I receive:

sudo apt-get install libglib2.0-dev python-dev libatlas-base-dev

Reading package lists... Done
Building dependency tree
Reading state information... Done
python-dev is already the newest version.
libatlas-base-dev is already the newest version.
libglib2.0-dev is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 440 not upgraded.

[ibayer]

You need the following steps:

1. sudo apt-get install libglib2.0-dev python-dev libatlas-base-dev
2. cd fastFM/; make
3. pip install -r /fastFM/requirements.txt
4. python setup.py install or pip install -e fastFM/

Are you missing step 2 ? Please start with a clean clone.

[BrianMiner]

I got a fresh clone and ran through all the steps. Same error sadly. Not sure why?

/home/anaconda/lib/python2.7/site-packages/fastFM/fastFM/base.py in ()
3
4 import numpy as np
----> 5 import ffm
6 import scipy.sparse as sp
7 from scipy.stats import norm

ImportError: /home/anaconda/lib/python2.7/site-packages/fastFM/ffm.so: undefined symbol: g_array_new

[ibayer]

I need the command and output for every single step starting with git clone --recursive.

[BrianMiner]

OK I will start over again and supply

[BrianMiner]

Attached. Thanks for your help!

ERRORS_fastFM.TXT

[ibayer]

Thanks for reporting the error, I can reproduce it locally. I'll comment here as soon as it's fixed!

[ibayer]

I have committed a bugfix.
Travis is currently only testing the OSX build. I'll add linux asap to avoid issues like this.

[BrianMiner]

Seemed to do the trick! thanks very much!

[BrianMiner]

On a side note, I was curious how FM generally would work on a typical structured data problem (not a click through one with massively high cardinality of IDs) - so I tried fastFM on the current Kaggle dataset from the Prudential challenge. After 1 hot encoding the categorical variables, there is around 850 variables (some numerics, some discrete ordinal ones and the dummy variables). The performance I got so far was very poor. I was curious if you have found that this algorithm can be competitive on problems like this (I noted you mentioned it should be a regular "toolbox" algorithm).

[ibayer]

FM's can be regarded as second order polynomial regression with factorized coefficients for the variable interaction. The factorization is indeed most interesting for categorical variables with high cardinality.

I'm not surprised that FM's don't work miracles in the Prudential challenge but they should perform at least as good as logistic / linear regression with properly tuned hyper parameter.

I think loads of dummy variables show up more and more these days, not only in CRP. That's why I think it's good to have a tool for this kind of data in you toolbox. :-)

[BrianMiner]

It would be interesting to see what an expert can do with it. I was unable to get even decent results (worse than libfm) but I assume it was my newness to tuning.

[zerowgravity]

Just curious to know if this has been merged into master? I'm facing the same issue on Linux 4.1.17-22.30.amzn1.x86_64 #1 SMP Fri Feb 5 23:44:22 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux. The issue is that blas is not found while running make.

[ibayer]

@zerowgravity
Yes, this has been merged. Make sure you have blas installed.
sudo apt-get install python-dev libatlas-base-dev
You need to provide more information: install source / binary, full error output, etc.

[zerowgravity]

I'm running an Amazon Linux AMI, x86_64 x86_64 x86_64 GNU/Linux and installed fastFM from source.

Here is the full error output:

fastFM]$ make
( cd fastFM-core ; make lib )
make[1]: Entering directory /mnt1/anaconda2/lib/python2.7/site-packages/fastFM/fastFM-core' ( cd src ; make lib ) make[2]: Entering directory/mnt1/anaconda2/lib/python2.7/site-packages/fastFM/fastFM-core/src'
( cd ../externals/CXSparse ; make library )
make[3]: Entering directory /mnt1/anaconda2/lib/python2.7/site-packages/fastFM/fastFM-core/externals/CXSparse' ( cd Lib ; make ) make[4]: Entering directory/mnt1/anaconda2/lib/python2.7/site-packages/fastFM/fastFM-core/externals/CXSparse/Lib'
make[4]: Nothing to be done for default'. make[4]: Leaving directory/mnt1/anaconda2/lib/python2.7/site-packages/fastFM/fastFM-core/externals/CXSparse/Lib'
make[3]: Leaving directory /mnt1/anaconda2/lib/python2.7/site-packages/fastFM/fastFM-core/externals/CXSparse' mkdir -p ../bin/ ar rcs ../bin/libfastfm.a kmath.o ffm_random.o ffm_als_mcmc.o ffm_utils.o ffm_sgd.o ffm.o make[2]: Leaving directory/mnt1/anaconda2/lib/python2.7/site-packages/fastFM/fastFM-core/src'
make[1]: Leaving directory `/mnt1/anaconda2/lib/python2.7/site-packages/fastFM/fastFM-core'
python setup.py build_ext --inplace
running build_ext
skipping 'fastFM/ffm.c' Cython extension (up-to-date)
building 'ffm' extension
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/fastFM
gcc -pthread -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -IfastFM/ -IfastFM-core/include/ -IfastFM-core/externals/CXSparse/Include/ -I/usr/include/ -I/mnt1/anaconda2/envs/p3/lib/python3.5/site-packages/numpy/core/include -I/mnt1/anaconda2/envs/p3/include/python3.5m -c fastFM/ffm.c -o build/temp.linux-x86_64-3.5/fastFM/ffm.o
In file included from /mnt1/anaconda2/envs/p3/lib/python3.5/site-packages/numpy/core/include/numpy/ndarraytypes.h:1781:0,
from /mnt1/anaconda2/envs/p3/lib/python3.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
from /mnt1/anaconda2/envs/p3/lib/python3.5/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from fastFM/ffm.c:244:
/mnt1/anaconda2/envs/p3/lib/python3.5/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by "
^
In file included from /mnt1/anaconda2/envs/p3/lib/python3.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:27:0,
from /mnt1/anaconda2/envs/p3/lib/python3.5/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from fastFM/ffm.c:244:
/mnt1/anaconda2/envs/p3/lib/python3.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:1634:1: warning: ‘_import_array’ defined but not used [-Wunused-function]
_import_array(void)
^
gcc -pthread -shared -L/mnt1/anaconda2/envs/p3/lib -Wl,-rpath=/mnt1/anaconda2/envs/p3/lib,--no-as-needed build/temp.linux-x86_64-3.5/fastFM/ffm.o -LfastFM/ -LfastFM-core/bin/ -LfastFM-core/externals/CXSparse/Lib/ -L/usr/lib/ -L/usr/lib/atlas-base/ -L/mnt1/anaconda2/envs/p3/lib -lm -lfastfm -lcxsparse -lcblas -lpython3.5m -o /mnt1/anaconda2/lib/python2.7/site-packages/fastFM/ffm.cpython-35m-x86_64-linux-gnu.so
/usr/bin/ld: cannot find -lcblas
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1
make: *** [all] Error 1

[zerowgravity]

This ended up being a symlink issue. From research, I think it's specific to CentOS and RHEL platforms. The symlink seems to be created otherwise.

[ibayer]

@zerowgravity
Thanks for the heads up. Did you come across a page that describes the symlink issue? Could be useful to document it here, in case someone else runs into it.

[zerowgravity]

I found this http://stackoverflow.com/questions/6789368/how-to-make-sure-the-numpy-blas-libraries-are-available-as-dynamically-loadable and tried to emulate it for fastFM.

[Data-drone]

I didn't manage to successfully follow that stackoverflow should I just find the libblas.so in /usr/lib64 and paste it into the fastFM directory in python2.7/dist-packages?

From what I can see /usr/lib64/libblas.so exists so it isn't an issue with it being called .3gf