cmake/catkin scripts not detecting python & numpy C/C++ header include dirs correctly

[ahundt]

The build scripts incorrectly assume the location of numpy.

I suggest using the python and numpy cmake scripts in the folder linked below to solve the problem. I recently used them in a similar manner to fix the bullet3 configuration:
https://github.com/bulletphysics/bullet3/tree/master/build3/cmake

I followed the setup instructions, then ran the following to debug the build:

catkin build --verbose --make-args VERBOSE=1

The problem is the numpy location is assumed, rather than correctly detected by having cmake run a python script with something like:

numpy.get_include()

explanation of the above:
https://stackoverflow.com/questions/14657375/cython-fatal-error-numpy-arrayobject-h-no-such-file-or-directory

Here is my case:

In [1]: import numpy

In [2]: numpy.get_include()
Out[2]: '/usr/local/lib/python2.7/dist-packages/numpy/core/include'

I got the following error and as you can see the incorrect directory is included -I/usr/include/python2.7:

[ 59%] Building CXX object CMakeFiles/numpy_eigen.dir/src/autogen_module/import_1_D_int.cpp.o
/usr/bin/c++   -D__STRICT_ANSI__ -Dnumpy_eigen_EXPORTS -I/home/ahundt/src/kalibr_ws/src/kalibr/S
chweizer-Messer/numpy_eigen/include -I/usr/include/eigen3 -I/home/ahundt/src/kalibr_ws/src/kalib
r/Schweizer-Messer/numpy_eigen/include/numpy_eigen -I/usr/include/python2.7  -fPIC   -std=c++0x
-o CMakeFiles/numpy_eigen.dir/src/autogen_module/import_1_D_int.cpp.o -c /home/ahundt/src/kalibr
_ws/src/kalibr/Schweizer-Messer/numpy_eigen/src/autogen_module/import_1_D_int.cpp
In file included from /home/ahundt/src/kalibr_ws/src/kalibr/Schweizer-Messer/numpy_eigen/src/aut
ogen_module/import_1_3_float.cpp:4:0:
/home/ahundt/src/kalibr_ws/src/kalibr/Schweizer-Messer/numpy_eigen/include/numpy_eigen/NumpyEige
nConverter.hpp:20:32: fatal error: numpy/arrayobject.h: No such file or directory
compilation terminated.
CMakeFiles/numpy_eigen.dir/build.make:350: recipe for target 'CMakeFiles/numpy_eigen.dir/src/aut
ogen_module/import_1_3_float.cpp.o' failed
make[2]: *** [CMakeFiles/numpy_eigen.dir/src/autogen_module/import_1_3_float.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /home/ahundt/src/kalibr_ws/src/kalibr/Schweizer-Messer/numpy_eigen/src/aut
ogen_module/import_1_5_uchar.cpp:4:0:

in case others come across this my quick fix was the following, which does not solve the issue correctly since it could break other packages when they build or when using other python versions:

ln -s /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy /usr/local/include/numpy

[pxlong]

Hi @ahundt, have you solved this problem?
I am facing the same issue. Thanks.

[ahundt]

nope, just used the hack I mention above.

[NikolausDemmel]

Thanks @ahundt for the pointers. Just had the same issue and the workaround works for now.

You suggested cmake scripts above. Any particular reason why one would need the custom FindPythonLibs? For Kalibr adding FindNumpy.cmake might be enough?

[ahundt]

No unfortunately the python + numpy scripts in CMake itself are broken to my knowledge, they make no effort to coordinate python and numpy, and thus could find python 2.7 and the numpy files for a totally different python version.

Here is the bug report I submitted to CMake itself but there hasn't been any action:
https://gitlab.kitware.com/cmake/cmake/issues/17546

@NikolausDemmel perhaps you can confirm that you were able to reproduce the problem at the linked page? Often that leads to a priority bump. You can also get the files I used there.

[mpdmanash]

The build scripts incorrectly assume the location of numpy.

I suggest using the python and numpy cmake scripts in the folder linked below to solve the problem. I recently used them in a similar manner to fix the bullet3 configuration:
https://github.com/bulletphysics/bullet3/tree/master/build3/cmake

I followed the setup instructions, then ran the following to debug the build:

catkin build --verbose --make-args VERBOSE=1

The problem is the numpy location is assumed, rather than correctly detected by having cmake run a python script with something like:

numpy.get_include()

explanation of the above:
https://stackoverflow.com/questions/14657375/cython-fatal-error-numpy-arrayobject-h-no-such-file-or-directory

Here is my case:

In [1]: import numpy

In [2]: numpy.get_include()
Out[2]: '/usr/local/lib/python2.7/dist-packages/numpy/core/include'

I got the following error and as you can see the incorrect directory is included -I/usr/include/python2.7:

[ 59%] Building CXX object CMakeFiles/numpy_eigen.dir/src/autogen_module/import_1_D_int.cpp.o
/usr/bin/c++   -D__STRICT_ANSI__ -Dnumpy_eigen_EXPORTS -I/home/ahundt/src/kalibr_ws/src/kalibr/S
chweizer-Messer/numpy_eigen/include -I/usr/include/eigen3 -I/home/ahundt/src/kalibr_ws/src/kalib
r/Schweizer-Messer/numpy_eigen/include/numpy_eigen -I/usr/include/python2.7  -fPIC   -std=c++0x
-o CMakeFiles/numpy_eigen.dir/src/autogen_module/import_1_D_int.cpp.o -c /home/ahundt/src/kalibr
_ws/src/kalibr/Schweizer-Messer/numpy_eigen/src/autogen_module/import_1_D_int.cpp
In file included from /home/ahundt/src/kalibr_ws/src/kalibr/Schweizer-Messer/numpy_eigen/src/aut
ogen_module/import_1_3_float.cpp:4:0:
/home/ahundt/src/kalibr_ws/src/kalibr/Schweizer-Messer/numpy_eigen/include/numpy_eigen/NumpyEige
nConverter.hpp:20:32: fatal error: numpy/arrayobject.h: No such file or directory
compilation terminated.
CMakeFiles/numpy_eigen.dir/build.make:350: recipe for target 'CMakeFiles/numpy_eigen.dir/src/aut
ogen_module/import_1_3_float.cpp.o' failed
make[2]: *** [CMakeFiles/numpy_eigen.dir/src/autogen_module/import_1_3_float.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /home/ahundt/src/kalibr_ws/src/kalibr/Schweizer-Messer/numpy_eigen/src/aut
ogen_module/import_1_5_uchar.cpp:4:0:

in case others come across this my quick fix was the following, which does not solve the issue correctly since it could break other packages when they build or when using other python versions:

ln -s /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy /usr/local/include/numpy

Thanks. The hack works for now. I am using Ubuntu 16.04.

[thirdlastletter]

The build scripts incorrectly assume the location of numpy.

I suggest using the python and numpy cmake scripts in the folder linked below to solve the problem. I recently used them in a similar manner to fix the bullet3 configuration:
https://github.com/bulletphysics/bullet3/tree/master/build3/cmake

I followed the setup instructions, then ran the following to debug the build:

catkin build --verbose --make-args VERBOSE=1

The problem is the numpy location is assumed, rather than correctly detected by having cmake run a python script with something like:

numpy.get_include()

explanation of the above:
https://stackoverflow.com/questions/14657375/cython-fatal-error-numpy-arrayobject-h-no-such-file-or-directory

Here is my case:

In [1]: import numpy

In [2]: numpy.get_include()
Out[2]: '/usr/local/lib/python2.7/dist-packages/numpy/core/include'

I got the following error and as you can see the incorrect directory is included -I/usr/include/python2.7:

[ 59%] Building CXX object CMakeFiles/numpy_eigen.dir/src/autogen_module/import_1_D_int.cpp.o
/usr/bin/c++   -D__STRICT_ANSI__ -Dnumpy_eigen_EXPORTS -I/home/ahundt/src/kalibr_ws/src/kalibr/S
chweizer-Messer/numpy_eigen/include -I/usr/include/eigen3 -I/home/ahundt/src/kalibr_ws/src/kalib
r/Schweizer-Messer/numpy_eigen/include/numpy_eigen -I/usr/include/python2.7  -fPIC   -std=c++0x
-o CMakeFiles/numpy_eigen.dir/src/autogen_module/import_1_D_int.cpp.o -c /home/ahundt/src/kalibr
_ws/src/kalibr/Schweizer-Messer/numpy_eigen/src/autogen_module/import_1_D_int.cpp
In file included from /home/ahundt/src/kalibr_ws/src/kalibr/Schweizer-Messer/numpy_eigen/src/aut
ogen_module/import_1_3_float.cpp:4:0:
/home/ahundt/src/kalibr_ws/src/kalibr/Schweizer-Messer/numpy_eigen/include/numpy_eigen/NumpyEige
nConverter.hpp:20:32: fatal error: numpy/arrayobject.h: No such file or directory
compilation terminated.
CMakeFiles/numpy_eigen.dir/build.make:350: recipe for target 'CMakeFiles/numpy_eigen.dir/src/aut
ogen_module/import_1_3_float.cpp.o' failed
make[2]: *** [CMakeFiles/numpy_eigen.dir/src/autogen_module/import_1_3_float.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /home/ahundt/src/kalibr_ws/src/kalibr/Schweizer-Messer/numpy_eigen/src/aut
ogen_module/import_1_5_uchar.cpp:4:0:

in case others come across this my quick fix was the following, which does not solve the issue correctly since it could break other packages when they build or when using other python versions:

ln -s /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy /usr/local/include/numpy

What do I do with the Cmake files in the link? (FindNumPy.cmake...)

[jlm345]

I faced the same problem and solved it in this way!Thank you!