make pycaffe/make distribute: undefined symbols boost::python

[pkmital]

I'm on OSX Mavericks, have installed dependencies with brew and edited all ruby files to include the libstdc++ flags. I've run make all, test and runtest successfully. When trying to make the python interface, though, I get linker errors referring to the boost python interface. This is the output of file on the boost python lib:

/usr/local/lib$file libboost_python.dylib
libboost_python.dylib: Mach-O 64-bit dynamically linked shared library x86_64

And the output of the make script:

$make pycaffe
/usr/bin/clang++ -shared -o python/caffe/_caffe.so python/caffe/_caffe.cpp \
        build/lib/libcaffe.a -stdlib=libstdc++ -pthread -fPIC -DNDEBUG -O2 -I/usr/local/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/lib/python2.7/site-packages -I/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/include/python2.7/ -I/usr/local/Cellar/numpy/1.8.1/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include -Ibuild/src -I./src -I./include -I/usr/local/cuda/include -I/System/Library/Frameworks/vecLib.framework/Versions/Current/Headers/ -framework vecLib -L/usr/local/lib -L/usr/local/Cellar/boost/1.55.0_1/lib -L/usr/local/lib -L/usr/lib -L/usr/local/Cellar/boost/1.55.0_1/lib -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib -lcudart -lcublas -lcurand -lpthread -lglog -lprotobuf -lleveldb -lsnappy -lboost_system -lhdf5_hl -lhdf5 -lopencv_core -lopencv_highgui -lopencv_imgproc -lcblas -lboost_python -lpython2.7
ld: warning: directory not found for option '-L/usr/local/cuda/lib64'
Undefined symbols for architecture x86_64:
  "boost::python::objects::function_object(boost::python::objects::py_function const&, std::pair<boost::python::detail::keyword const*, boost::python::detail::keyword const*> const&)", referenced from:
      void boost::python::class_<std::vector<CaffeLayer, std::allocator<CaffeLayer> >, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified>::def_impl<std::vector<CaffeLayer, std::allocator<CaffeLayer> >, void (*)(std::vector<CaffeLayer, std::allocator<CaffeLayer> >&, boost::python::api::object), boost::python::detail::def_helper<char const*, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified> >(std::vector<CaffeLayer, std::allocator<CaffeLayer> >*, char const*, void (*)(std::vector<CaffeLayer, std::allocator<CaffeLayer> >&, boost::python::api::object), boost::python::detail::def_helper<char const*, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified> const&, ...) in _caffe-fe521f.o
      void boost::python::class_<std::vector<CaffeLayer, std::allocator<CaffeLayer> >, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified>::def_impl<std::vector<CaffeLayer, std::allocator<CaffeLayer> >, bool (*)(std::vector<CaffeLayer, std::allocator<CaffeLayer> >&, _object*), boost::python::detail::def_helper<char const*, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified> >(std::vector<CaffeLayer, std::allocator<CaffeLayer> >*, char const*, bool (*)(std::vector<CaffeLayer, std::allocator<CaffeLayer> >&, _object*), boost::python::detail::def_helper<char const*, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified> const&, ...) in _caffe-fe521f.o
      void boost::python::class_<std::vector<CaffeLayer, std::allocator<CaffeLayer> >, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified>::def_impl<std::vector<CaffeLayer, std::allocator<CaffeLayer> >, boost::python::api::object (*)(boost::python::back_reference<std::vector<CaffeLayer, std::allocator<CaffeLayer> >&>, _object*), boost::python::detail::def_helper<char const*, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified> >(std::vector<CaffeLayer, std::allocator<CaffeLayer> >*, char const*, boost::python::api::object (*)(boost::python::back_reference<std::vector<CaffeLayer, std::allocator<CaffeLayer> >&>, _object*), boost::python::detail::def_helper<char const*, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified> const&, ...) in _caffe-fe521f.o
      void boost::python::class_<std::vector<CaffeLayer, std::allocator<CaffeLayer> >, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified>::def_impl<std::vector<CaffeLayer, std::allocator<CaffeLayer> >, void (*)(std::vector<CaffeLayer, std::allocator<CaffeLayer> >&, _object*), boost::python::detail::def_helper<char const*, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified> >(std::vector<CaffeLayer, std::allocator<CaffeLayer> >*, char const*, void (*)(std::vector<CaffeLayer, std::allocator<CaffeLayer> >&, _object*), boost::python::detail::def_helper<char const*, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified> const&, ...) in _caffe-fe521f.o
      void boost::python::class_<std::vector<CaffeLayer, std::allocator<CaffeLayer> >, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified>::def_impl<std::vector<CaffeLayer, std::allocator<CaffeLayer> >, void (*)(std::vector<CaffeLayer, std::allocator<CaffeLayer> >&, _object*, _object*), boost::python::detail::def_helper<char const*, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified> >(std::vector<CaffeLayer, std::allocator<CaffeLayer> >*, char const*, void (*)(std::vector<CaffeLayer, std::allocator<CaffeLayer> >&, _object*, _object*), boost::python::detail::def_helper<char const*, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified> const&, ...) in _caffe-fe521f.o
      void boost::python::class_<std::vector<CaffeLayer, std::allocator<CaffeLayer> >, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified>::def_impl<std::vector<CaffeLayer, std::allocator<CaffeLayer> >, unsigned long (*)(std::vector<CaffeLayer, std::allocator<CaffeLayer> >&), boost::python::detail::def_helper<char const*, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified> >(std::vector<CaffeLayer, std::allocator<CaffeLayer> >*, char const*, unsigned long (*)(std::vector<CaffeLayer, std::allocator<CaffeLayer> >&), boost::python::detail::def_helper<char const*, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified> const&, ...) in _caffe-fe521f.o
      void boost::python::detail::def_init_aux<boost::python::class_<std::vector<CaffeLayer, std::allocator<CaffeLayer> >, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified>, boost::python::default_call_policies, boost::mpl::vector0<mpl_::na>, boost::mpl::size<boost::mpl::vector0<mpl_::na> > >(boost::python::class_<std::vector<CaffeLayer, std::allocator<CaffeLayer> >, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified>&, boost::mpl::vector0<mpl_::na> const&, boost::mpl::size<boost::mpl::vector0<mpl_::na> >, boost::python::default_call_policies const&, char const*, std::pair<boost::python::detail::keyword const*, boost::python::detail::keyword const*> const&) in _caffe-fe521f.o
      ...
  "boost::python::objects::register_dynamic_id_aux(boost::python::type_info, std::pair<void*, boost::python::type_info> (*)(void*))", referenced from:
      init_module__caffe() in _caffe-fe521f.o
      boost::python::class_<boost::python::objects::iterator_range<boost::python::return_value_policy<boost::python::return_by_value, boost::python::default_call_policies>, __gnu_cxx::__normal_iterator<CaffeLayer*, std::vector<CaffeLayer, std::allocator<CaffeLayer> > > >, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified>::class_(char const*, boost::python::no_init_t) in _caffe-fe521f.o
      boost::python::class_<std::vector<CaffeLayer, std::allocator<CaffeLayer> >, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified>::class_(char const*, char const*) in _caffe-fe521f.o
      boost::python::class_<boost::python::objects::iterator_range<boost::python::return_value_policy<boost::python::return_by_value, boost::python::default_call_policies>, __gnu_cxx::__normal_iterator<CaffeBlob*, std::vector<CaffeBlob, std::allocator<CaffeBlob> > > >, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified>::class_(char const*, boost::python::no_init_t) in _caffe-fe521f.o
      boost::python::class_<std::vector<CaffeBlob, std::allocator<CaffeBlob> >, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified>::class_(char const*, char const*) in _caffe-fe521f.o
      boost::python::class_<CaffeSGDSolver, boost::noncopyable_::noncopyable, boost::python::detail::not_specified, boost::python::detail::not_specified>::class_<boost::python::init<std::string, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_> >(char const*, boost::python::init_base<boost::python::init<std::string, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_> > const&) in _caffe-fe521f.o
      boost::python::class_<CaffeLayer, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified>::class_(char const*, boost::python::no_init_t) in _caffe-fe521f.o
      ...
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [python/caffe/_caffe.so] Error 1

This is the header for function_object:

$cat /usr/local/include/boost/python/object/function_object.hpp
// Copyright David Abrahams 2002.
// Distributed under the Boost Software License, Version 1.0. (See
// accompanying file LICENSE_1_0.txt or copy at
// http://www.boost.org/LICENSE_1_0.txt)
#ifndef FUNCTION_OBJECT_DWA2002725_HPP
# define FUNCTION_OBJECT_DWA2002725_HPP
# include <boost/python/detail/prefix.hpp>
# include <boost/function/function2.hpp>
# include <boost/python/object_core.hpp>
# include <boost/python/args_fwd.hpp>
# include <boost/python/object/py_function.hpp>

namespace boost { namespace python {

namespace objects
{
  BOOST_PYTHON_DECL api::object function_object(
      py_function const& f
      , python::detail::keyword_range const&);

  BOOST_PYTHON_DECL api::object function_object(
      py_function const& f
      , python::detail::keyword_range const&);

  BOOST_PYTHON_DECL api::object function_object(py_function const& f);

  // Add an attribute to the name_space with the given name. If it is
  // a Boost.Python function object
  // (boost/python/object/function.hpp), and an existing function is
  // already there, add it as an overload.
  BOOST_PYTHON_DECL void add_to_namespace(
      object const& name_space, char const* name, object const& attribute);

  BOOST_PYTHON_DECL void add_to_namespace(
      object const& name_space, char const* name, object const& attribute, char const* doc);
}

}} // namespace boost::python::objects

#endif // FUNCTION_OBJECT_DWA2002725_HPP

It doesn't seem to have any function like the one the linker error mentioned (3 arguments?)... I'm using boost 1.55 as per the brew installation (1.55.0 git checkout c91d4d5 Library/Formula/boost.rb)

I've also tried recompiling boost with and without c++11 with no success.

Any help appreciated!

[pkmital]

This seems to be an issue with some libraries still being linked again libstdc++ and other with libc++. The output of otool shows:

$otool -L /usr/local/lib/libboost_python.dylib
/usr/local/lib/libboost_python.dylib:
    /usr/local/lib/libboost_python.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

When it should be linked against libstdc++ instead. Despite adding the code as shown on caffe's website, you should modify the b2 call in the ruby script. I've done it like so:

    args << "cxxflags=-stdlib=libstdc++" << "linkflags=-stdlib=libstdc++"

And now everything works.

[shelhamer]

A change in the latest homebrew seems to be guilty here. Switching the formula edits to use ENV["CXX"] = '/usr/bin/clang++ -stdlib=libstdc++' as in #449 fixes this boost problem, and likewise fixes the installation of the other dependencies. The docs will be updated shortly.

[pkmital]

Yes the new convention for editing ENV effects every package except boost. Though also for boost, I found that you have to edit the b2 call, using the args command pasted above.

[liuxianming]

Hi, I'm compiling the pycaffe on Redhat linux. I met the similar problem. Do you have any suggestions on how to deal with that? I use libboost 1.5.5. Thx

[shelhamer]

@pkmital I did not find I had to edit the b2 call; instead editing ENV was sufficient. Perhaps we compiled different versions or platforms (I did boost 1.55 on Ubuntu 12.04 and OS X 10.9).

@liuxianming you have to be very careful to compile with the correct standard library and to make sure boost::python is built. See the latest docs in the dev branch https://github.com/BVLC/caffe/blob/dev/docs/installation.md Good luck!

[liuxianming]

@shelhamer Thx, I fixed it. Yes, need to compile very carefully, especially when you are using anaconda.

[synchro--]

@liuxianming Still have the same error, how did you fixed ?

[sebastianfelipe]

How did you solve this issue? Need to compile Caffe on OSX Mojave. Thanks.