Make issues for Python module (macOS)

[cedavidyang]

I am unable to use Python module on macOS (I did manage to use Python module on an Ubuntu machine following the instructions on README.md)

Among the dependencies, I used homebrew to install cmake, boost, boost-build, boost-python, and eigen3; lp_solve is downloaded from sourceforge.

Since I use Python installed by homebrew, I used -DPYTHON_LIBRARY and -DPYTHON_INCLUDE_DIR to direct cmake to homebrew versions of libpython2.7.dylib and Headers.

To expose Python to CMakeList, I also changed find_package(Boost ${BOOST_VERSION_REQUIRED} COMPONENTS python REQUIRED) to find_package(Boost ${BOOST_VERSION_REQUIRED} COMPONENTS python27 REQUIRED) and set(BOOST_PYTHON_LIBRARY_NAME "Boost_PYTHON_LIBRARY") to set(BOOST_PYTHON_LIBRARY_NAME "Boost_PYTHON27_LIBRARY")

By using g++ shipped with macOS (clang-1000.10.44.4), I can cmake && make. C++ examples can run without a problem. However, Python examples are not working. Specifically, I cannot "import AIToolbox" even through I have AIToolbox.dylib compiled successfully.

Ways I have tried so far to solve this problem:

• Install gcc-7 and gmake using homebrew and use -DCMAKE_C_COMPILER=/usr/local/bin/gcc-7 -DCMAKE_CXX_COMPILER=/usr/local/bin/g++-7 to force cmake use g++-7
• Install gmake using homebrew and use export PATH="/usr/local/opt/make/libexec/gnubin:$PATH to force cmake use gmake
• Following answers to issue Make Errors #22 , I added export LD_LIBRARY_PATH=/usr/local/Cellar/gcc\@7/7.4.0/lib/gcc/7/:/usr/local/lib/:/usr/lib/:$LD_LIBRARY_PATH to direct python to the gcc-7 libstdc++

Overall, my script for build is as follows:

#!/bin/bash
export LD_LIBRARY_PATH=/usr/local/Cellar/gcc\@7/7.4.0/lib/gcc/7/:/usr/local/lib/:/usr/lib/:$LD_LIBRARY_PATH

mkdir build
cd build

PYTHON_LIBRARY=/usr/local/Cellar/python\@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib
PYTHON_INCLUDE_DIR=/usr/local/Cellar/python\@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/Headers/
cmake -DCMAKE_C_COMPILER=/usr/local/bin/gcc-7 -DCMAKE_CXX_COMPILER=/usr/local/bin/g++-7 -DPYTHON_LIBRARY=$PYTHON_LIBRARY -DPYTHON_INCLUDE_DIR=$PYTHON_INCLUDE_DIR -MAKE_LIB=1 -MAKE_PYTHON=1 ..

By doing so, I am able to use g++-7 as required. However, I came across linking errors both during cmake and make (stopped at 47%). The details are as follows:

Upon using the script (i.e. cmake), the output is

-- The CXX compiler identification is GNU 7.4.0
-- Checking whether CXX compiler has -isysroot
-- Checking whether CXX compiler has -isysroot - yes
-- Checking whether CXX compiler supports OSX deployment target flag
-- Checking whether CXX compiler supports OSX deployment target flag - yes
-- Check for working CXX compiler: /usr/local/bin/g++-7
-- Check for working CXX compiler: /usr/local/bin/g++-7 -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- IPO / LTO not supported: <Change Dir: /Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/bin

Run Build Command:"/usr/local/bin/gmake"
/usr/local/Cellar/cmake/3.13.3/bin/cmake -S/Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/src -B/Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/bin --check-build-system CMakeFiles/Makefile.cmake 0
/usr/local/Cellar/cmake/3.13.3/bin/cmake -E cmake_progress_start /Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/bin/CMakeFiles /Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/bin/CMakeFiles/progress.marks
/usr/local/bin/gmake -f CMakeFiles/Makefile2 all
gmake[1]: Entering directory '/Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/bin'
/usr/local/bin/gmake -f CMakeFiles/foo.dir/build.make CMakeFiles/foo.dir/depend
gmake[2]: Entering directory '/Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/bin'
cd /Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/bin && /usr/local/Cellar/cmake/3.13.3/bin/cmake -E cmake_depends "Unix Makefiles" /Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/src /Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/src /Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/bin /Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/bin /Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/bin/CMakeFiles/foo.dir/DependInfo.cmake
Scanning dependencies of target foo
gmake[2]: Leaving directory '/Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/bin'
/usr/local/bin/gmake -f CMakeFiles/foo.dir/build.make CMakeFiles/foo.dir/build
gmake[2]: Entering directory '/Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/bin'
[ 25%] Building CXX object CMakeFiles/foo.dir/foo.cpp.o
/usr/local/bin/g++-7    -flto -fno-fat-lto-objects   -o CMakeFiles/foo.dir/foo.cpp.o -c /Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/src/foo.cpp
cc1plus: error: -fno-fat-lto-objects are supported only with linker plugin
gmake[2]: *** [CMakeFiles/foo.dir/build.make:66: CMakeFiles/foo.dir/foo.cpp.o] Error 1
gmake[2]: Leaving directory '/Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/bin'
gmake[1]: *** [CMakeFiles/Makefile2:113: CMakeFiles/foo.dir/all] Error 2
gmake[1]: Leaving directory '/Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build/CMakeFiles/_CMakeLTOTest-CXX/bin'
gmake: *** [Makefile:87: all] Error 2
>

Build type: Release
Logging is DISABLED
# Building MDP
# Building Factored MDP
# Building POMDP
# Building Python bindings
# Building Tests
# Building Examples

-- Boost version: 1.68.0
-- Found Eigen3: /usr/local/include/eigen3 (Required is at least version "3.2.92")
-- Performing Test LPSOLVE_LINKS_ALONE
-- Performing Test LPSOLVE_LINKS_ALONE - Success
-- Found LpSolve: /usr/local/lib/liblpsolve55.dylib
-- Found PythonInterp: /usr/local/bin/python (found version "2.7.15")
-- Found PythonLibs: /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib (found suitable exact version "2.7.15")
-- Boost version: 1.68.0
-- Found the following Boost libraries:
--   python27
-- Boost version: 1.68.0
-- Found the following Boost libraries:
--   unit_test_framework
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/davidyang/Desktop/RBIcomparison/AI-Toolbox/build

Running make in build folder results in error at Linking CXX shared library ../AIToolbox.dylib:

[ 47%] Linking CXX shared library ../AIToolbox.dylib
Undefined symbols for architecture x86_64:
  "boost::python::objects::function_object(boost::python::objects::py_function const&, std::pair<boost::python::detail::keyword const*, boost::python::detail::keyword const*> const&)", referenced from:
      boost::python::api::object boost::python::detail::make_function_aux<void (*)(_object*), boost::python::default_call_policies, boost::mpl::vector2<void, _object*>, mpl_::int_<0> >(void (*)(_object*), boost::python::default_call_policies const&, boost::mpl::vector2<void, _object*> const&, std::pair<boost::python::detail::keyword const*, boost::python::detail::keyword const*> const&, mpl_::int_<0>) [clone .isra.972] in Types.cpp.o
       void boost::python::indexing_suite<std::vector<Eigen::Matrix<double, -1, 1, 0, -1, 1>, std::allocator<Eigen::Matrix<double, -1, 1, 0, -1, 1> > >, boost::python::detail::final_vector_derived_policies<std::vector<Eigen::Matrix<double, -1, 1, 0, -1, 1>, std::allocator<Eigen::Matrix<double, -1, 1, 0, -1, 1> > >, false>, false, false, Eigen::Matrix<double, -1, 1, 0, -1, 1>, unsigned long, Eigen::Matrix<double, -1, 1, 0, -1, 1> >::visit<boost::python::class_<std::vector<Eigen::Matrix<double, -1, 1, 0, -1, 1>, std::allocator<Eigen::Matrix<double, -1, 1, 0, -1, 1> > >, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified> >(boost::python::class_<std::vector<Eigen::Matrix<double, -1, 1, 0, -1, 1>, std::allocator<Eigen::Matrix<double, -1, 1, 0, -1, 1> > >, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified>&) const in Types.cpp.o
      exportTypes()     in Types.cpp.o
      exportBanditRollingAverage()     in RollingAverage.cpp.o
      exportBanditPolicyInterface()     in PolicyInterface.cpp.o
      ...
  "boost::python::objects::register_dynamic_id_aux(boost::python::type_info, std::pair<void*, boost::python::type_info> (*)(void*))", referenced from:
      exportTypes()     in Types.cpp.o
      boost::python::api::object boost::python::objects::detail::demand_iterator_class<__gnu_cxx::__normal_iterator<Eigen::Matrix<double, -1, 1, 0, -1, 1>*, std::vector<Eigen::Matrix<double, -1, 1, 0, -1, 1>, std::allocator<Eigen::Matrix<double, -1, 1, 0, -1, 1> > > >, boost::python::return_internal_reference<1ul, boost::python::default_call_policies> >(char const*, __gnu_cxx::__normal_iterator<Eigen::Matrix<double, -1, 1, 0, -1, 1>*, std::vector<Eigen::Matrix<double, -1, 1, 0, -1, 1>, std::allocator<Eigen::Matrix<double, -1, 1, 0, -1, 1> > > >*, boost::python::return_internal_reference<1ul, boost::python::default_call_policies> const&) in Types.cpp.o
      boost::python::api::object boost::python::objects::detail::demand_iterator_class<__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, boost::python::return_value_policy<boost::python::return_by_value, boost::python::default_call_policies> >(char const*, __gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >*, boost::python::return_value_policy<boost::python::return_by_value, boost::python::default_call_policies> const&) in Types.cpp.o
      boost::python::api::object boost::python::objects::detail::demand_iterator_class<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, boost::python::return_value_policy<boost::python::return_by_value, boost::python::default_call_policies> >(char const*, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >*, boost::python::return_value_policy<boost::python::return_by_value, boost::python::default_call_policies> const&) in Types.cpp.o
      exportBanditRollingAverage()     in RollingAverage.cpp.o
      exportBanditPolicyInterface()     in PolicyInterface.cpp.o
      exportBanditEpsilonPolicy()     in EpsilonPolicy.cpp.o
      ...
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[2]: *** [src/CMakeFiles/AIToolbox.dir/build.make:975: AIToolbox.dylib] Error 1
make[1]: *** [CMakeFiles/Makefile2:995: src/CMakeFiles/AIToolbox.dir/all] Error 2
make: *** [Makefile:95: all] Error 2

[Svalorzen]

Thanks for the well reviewed question.

Some notes:

• In the cmake command, -MAKE_LIB=1 -MAKE_PYTHON=1 should be -DMAKE_LIB=1 -DMAKE_PYTHON=1 or they are not going to be read. In this case it doesn't matter though, it's just for clarification. You may also want to set -DPYTHON_VERSION=2, but it's also probably not necessary.
• I have no idea why make (gmake) is being called during the cmake call. In particular, there's some mentions in there of a foo dependency (Scanning dependencies of target foo), and I have no idea what those are supposed to be.
• It's also very weird that you end up with a AIToolbox.dylib file if the linking process fails.. usually nothing should be produced at all. I don't know how this could have happened.

An important thing is that you mention changing set(BOOST_PYTHON_LIBRARY_NAME "Boost_PYTHON27_LIBRARY"). I'm not sure whether this is 100% correct: please make sure that it is. You can test this by simply adding a:

message(${Boost_PYTHON27_LIBRARY}) 

and checking that the found library is printed. If nothing is printed, it means that the library won't be linked anymore, which can result in the errors you are seeing. If that's the case, please revert to Boost_PYTHON_LIBRARY (and again, check with message that the library filename is in there) - if not, ignore this step.

I have found by Googling some mentions of a similar problem: here, here, and here.

It seems another possible way this issue can happen is by some version mismatch between the Python used to compile the boost you installed via homebrew and the Python you are trying to link now in the library. The links suggest using Python 3 and reinstalling boost while explicitly removing Python 2 support - maybe doing the opposite in your case may also work. Unfortunately I'm not familiar with how homebrew works, so I can't be sure. Maybe you could also try to see how the boost you installed was compiled and check if the various versions are the same.

Let me know if you find anything out, hopefully we can fix this problem :)

[Svalorzen]

Ah, and as a last note: when trying something out in CMake in order to make stuff compile, it's usually better to always start from a clean slate, or CMake files might mess new runs up. This means deleting the build folder and re-creating it for every attempt. This is annoying but it really prevents some weird classes of errors (at least when changing important parameters such as which libraries to link and so on).

Since in your case this would require re-compiling everything every time, you might want to only specify the library flags -DMAKE_MDP=1 and -DMAKE_PYTHON=1. This means that you only try to compile the MDP part with Python (no tests, no examples, ...), which should take less time than trying to compile everything.

Once you are able to produce a viable .dylib, you can remove the options from the CMake call and get everything.

[cedavidyang]

Thanks for the quick reply. Regarding your comments,

• In the cmake command, -MAKE_LIB=1 -MAKE_PYTHON=1 should be -DMAKE_LIB=1 -DMAKE_PYTHON=1 or they are not going to be read. In this case it doesn't matter though, it's just for clarification. You may also want to set -DPYTHON_VERSION=2, but it's also probably not necessary.

Thanks, these changes have been made. As expected, this mistake itself won't solve the error in binding Python.

• I have no idea why make (gmake) is being called during the cmake call. In particular, there's some mentions in there of a foo dependency (Scanning dependencies of target foo), and I have no idea what those are supposed to be.

I think make(gmake) is called for testing "IPO / LTO not supported", for which a simple and temporary file foo.cpp is created. However, I am not 100% sure.

• It's also very weird that you end up with a AIToolbox.dylib file if the linking process fails.. usually nothing should be produced at all. I don't know how this could have happened.

Sorry for the confusion. The AIToolbox.dylib is only created when g++ shipped with macOS (clang-1000.10.44.4) is used. Neverthless, this AIToolbox.dylib cannot be imported into Python. When gcc-7 from homebrew is used, AIToolbox.dylib cannot be created. The make process ends at around 47% with messages shown in the original post.

An important thing is that you mention changing set(BOOST_PYTHON_LIBRARY_NAME "Boost_PYTHON27_LIBRARY"). I'm not sure whether this is 100% correct: please make sure that it is. You can test this by simply adding a:

message(${Boost_PYTHON27_LIBRARY}) 

and checking that the found library is printed. If nothing is printed, it means that the library won't be linked anymore, which can result in the errors you are seeing. If that's the case, please revert to Boost_PYTHON_LIBRARY (and again, check with message that the library filename is in there) - if not, ignore this step.

I did the test. ${Boost_PYTHON27_LIBRARY} is the correct one. I think this may be related to the boost I am using (1.68).

I have found by Googling some mentions of a similar problem: here, here, and here.

It seems another possible way this issue can happen is by some version mismatch between the Python used to compile the boost you installed via homebrew and the Python you are trying to link now in the library. The links suggest using Python 3 and reinstalling boost while explicitly removing Python 2 support - maybe doing the opposite in your case may also work. Unfortunately I'm not familiar with how homebrew works, so I can't be sure. Maybe you could also try to see how the boost you installed was compiled and check if the various versions are the same.

I'll try the solutions there to see if any luck. I'll keep you posted on the results.

[Svalorzen]

Sorry for the confusion. The AIToolbox.dylib is only created when g++ shipped with macOS (clang-1000.10.44.4) is used. Neverthless, this AIToolbox.dylib cannot be imported into Python. When gcc-7 from homebrew is used, AIToolbox.dylib cannot be created. The make process ends at around 47% with messages shown in the original post.

Ahhhhh, sorry, it was my bad for misunderstanding. I then have the strong suspicion that if you managed to compile it with the default g++, then you should be able to make it work. I wouldn't try to switch gcc, as this may also cause problems (if you compile something with a compiler version - in this case could be boost - and something else with another version - in this case the ilbrary - weird stuff might also happen since C++ does not have an ABI).

This post suggests to just rename the .dylib into a .so, and that not being able to load .dylibs directly is just a Python problem. This issue also refers to something similar.

Let me know if this solves it!

[cedavidyang]

This post suggests to just rename the .dylib into a .so, and that not being able to load .dylibs directly is just a Python problem. This issue also refers to something similar.

This solves the problem. Thank you very much!

In summary, no need to install new compiler. Apple g++ can do the work. All dependencies can be installed directly from homebrew. Changing to set(BOOST_PYTHON_LIBRARY_NAME "Boost_PYTHON27_LIBRARY") in CMakeLists.txt is indeed needed for Boost 1.68. --- For Python3.7, change COMPONENTS python3 REQUIRED to COMPONENTS python37 REQUIRED (line 167 of CMakeList.txt) and set(BOOST_PYTHON_LIBRARY_NAME "Boost_PYTHON37_LIBRARY") (line 168 of CMakeList.txt)

Only one caveat for me to make this work --- if I install lp_solve through homebrew, I need to create a folder /usr/local/include/lpsolve and move (or copy to be safe) the created headers (lp_*.h) from /usr/local/include to /usr/local/include/lpsolve. It seems that homebrew install behaves differently from apt install on Ubuntu.

I left my working bash file here for anyone who might be interested

#!/bin/bash

mkdir build
cd build

PYTHON_VERSION=`python -c "import sys;t='{v[0]}.{v[1]}'.format(v=list(sys.version_info[:2]));sys.stdout.write(t)";`
PYTHON_LIBRARY=/usr/local/Frameworks/Python.framework/Versions/$PYTHON_VERSION/lib/libpython$PYTHON_VERSION.dylib
PYTHON_INCLUDE_DIR=/usr/local/Frameworks/Python.framework/Versions/$PYTHON_VERSION/Headers/
##only build MDP to save compiling time
#cmake -DPYTHON_LIBRARY=$PYTHON_LIBRARY \
    #-DPYTHON_INCLUDE_DIR=$PYTHON_INCLUDE_DIR \
    #-DMAKE_MDP=1 -DPYTHON_VERSION=2 -DMAKE_PYTHON=1 ..
cmake -DPYTHON_LIBRARY=$PYTHON_LIBRARY \
    -DPYTHON_INCLUDE_DIR=$PYTHON_INCLUDE_DIR ..

[Svalorzen]

Awesome! And thanks a lot for leaving scripts and instructions!!

I'll close this issue then, if you need help for anything else feel free to open more issues :)

[danra]

Only one caveat for me to make this work --- if I install lp_solve through homebrew, I need to create a folder /usr/local/include/lpsolve and move (or copy to be safe) the created headers (lp_*.h) from /usr/local/include to /usr/local/include/lpsolve

I witnessed the same; with current brew, the lp_*.h headers install directly into /opt/homebrew/include rather than into /opt/homebrew/include/lpsolve, where they would have been found by the FindLpSolve.cmake module. The quick workaround I did was just making /opt/homebrew/include/lpsolve a soft link into /opt/homebrew/include/. I think a proper fix would require removing the assumption in the cmake module as well as in a bit of code that lp_lib.h (for instance) can always be included as <lpsolve/lp_lib.h>: that's a good convention for how packages should provide their includes, but apparently lpsolve doesn't force that convention (the headers themselves don't make that assumption which I guess(?) is why the brew formula was set up this way).

(If the brew formula changed to place the headers under lpsolve, that would also work (but would again break if in the future some other package provider didn't do that)).