CMake cannot find libraries despite anaconda install

[Magalame]

After cloning the library, I attempted to install it. I got stuck at the cmake ../src part. I followed exactly the tutorial, namely:

conda create -n dataframes python=3.6
conda activate dataframes
conda install arrow-cpp=0.10.* -c conda-forge
conda install pyarrow=0.10.* -c conda-forge
conda install rapidjson

Then I added

set(CMAKE_LIBRARY_PATH "/home/user/anaconda3/envs/dataframes/lib")
set(CMAKE_INCLUDE_PATH "/home/use/anaconda3/envs/dataframes/include")

to CMakeLists.txt

then:

cd Dataframes/native_libs
mkdir build
cd build
cmake ../src

and it crashes with:

-- The C compiler identification is GNU 7.4.0
-- The CXX compiler identification is GNU 7.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Warning at /usr/share/cmake-3.10/Modules/FindBoost.cmake:567 (message):
  Imported targets and dependency information not available for Boost version
  (all versions older than 1.33)
Call Stack (most recent call first):
  /usr/share/cmake-3.10/Modules/FindBoost.cmake:907 (_Boost_COMPONENT_DEPENDENCIES)
  /usr/share/cmake-3.10/Modules/FindBoost.cmake:1558 (_Boost_MISSING_DEPENDENCIES)
  CMakeLists.txt:25 (find_package)


CMake Warning at /usr/share/cmake-3.10/Modules/FindBoost.cmake:567 (message):
  Imported targets and dependency information not available for Boost version
  (all versions older than 1.33)
Call Stack (most recent call first):
  /usr/share/cmake-3.10/Modules/FindBoost.cmake:907 (_Boost_COMPONENT_DEPENDENCIES)
  /usr/share/cmake-3.10/Modules/FindBoost.cmake:1558 (_Boost_MISSING_DEPENDENCIES)
  CMakeLists.txt:25 (find_package)


CMake Error at /usr/share/cmake-3.10/Modules/FindBoost.cmake:1947 (message):
  Unable to find the requested Boost libraries.

  Unable to find the Boost header files.  Please set BOOST_ROOT to the root
  directory containing Boost or BOOST_INCLUDEDIR to the directory containing
  Boost's headers.
Call Stack (most recent call first):
  CMakeLists.txt:25 (find_package)


CMake Warning at CMakeLists.txt:27 (message):
  Cannot find Boost libraries


CMake Warning at CMakeLists.txt:32 (message):
  Cannot find rapidjson include dir with rapidjson/document.h.  If it is
  present, consider setting CMAKE_PREFIX_PATH or CMAKE_INCLUDE_PATH.


CMake Warning at CMakeLists.txt:37 (message):
  Cannot find {fmt} include dir with fmt/format.h.  If it is present,
  consider setting CMAKE_PREFIX_PATH or CMAKE_INCLUDE_PATH.


CMake Warning at CMakeLists.txt:42 (message):
  Cannot find fmt library.  If it is present, consider setting
  CMAKE_PREFIX_PATH or CMAKE_LIBRARY_PATH.


CMake Warning at CMakeLists.txt:48 (message):
  Cannot find xlnt include dir with xlnt/xlnt.hpp.  If it is present,
  consider setting CMAKE_PREFIX_PATH or CMAKE_INCLUDE_PATH.


CMake Warning at CMakeLists.txt:58 (message):
  Cannot find xlnt library.  If it is present, consider setting
  CMAKE_PREFIX_PATH or CMAKE_LIBRARY_PATH.


CMake Warning at CMakeLists.txt:63 (message):
  Cannot find Apache Arrow include dir with arrow/array.h.  If it is present,
  consider setting CMAKE_PREFIX_PATH or CMAKE_INCLUDE_PATH.


CMake Warning at CMakeLists.txt:73 (message):
  Cannot find arrow library.  If it is present, consider setting
  CMAKE_PREFIX_PATH or CMAKE_LIBRARY_PATH.


CMake Warning at CMakeLists.txt:79 (message):
  Cannot find date library include dir with date/date.h.  If it is present,
  consider setting CMAKE_PREFIX_PATH or CMAKE_INCLUDE_PATH.


CMake Warning at CMakeLists.txt:84 (message):
  Cannot find pybind11 library include dir with pybind11/embed.h.  If it is
  present, consider setting CMAKE_PREFIX_PATH or CMAKE_INCLUDE_PATH.


-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.6m.so (found suitable version "3.6.7", minimum required is "3.6") 
-- Found PythonInterp: /home/user/anaconda3/envs/dataframes/bin/python3.6 (found version "3.6.8") 
-- Found NumPy: /home/user/anaconda3/envs/dataframes/lib/python3.6/site-packages/numpy/core/include (found version "1.16.3") 
CMake Warning at CMakeLists.txt:125 (message):
  DataframeHelper will be built without XLSX format support, as xlnt library
  was not found!


CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
/home/user/Projects/dataframes/native_libs/src/ARROW_INCLUDE
   used as include directory in directory /home/user/Projects/dataframes/native_libs/src
/home/user/Projects/dataframes/native_libs/src/Boost_INCLUDE_DIR
   used as include directory in directory /home/user/Projects/dataframes/native_libs/src
   used as include directory in directory /home/user/Projects/dataframes/native_libs/src
/home/user/Projects/dataframes/native_libs/src/DATE_INCLUDE
   used as include directory in directory /home/user/Projects/dataframes/native_libs/src
/home/user/Projects/dataframes/native_libs/src/FMT_INCLUDE
   used as include directory in directory /home/user/Projects/dataframes/native_libs/src
/home/user/Projects/dataframes/native_libs/src/PYBIND_INCLUDE
   used as include directory in directory /home/user/Projects/dataframes/native_libs/src
/home/user/Projects/dataframes/native_libs/src/RAPIDJSON_INCLUDE
   used as include directory in directory /home/user/Projects/dataframes/native_libs/src
ARROW_LIBRARY
    linked by target "DataframeHelper" in directory /home/user/Projects/dataframes/native_libs/src
FMT_LIBRARY
    linked by target "DataframeHelper" in directory /home/user/Projects/dataframes/native_libs/src

-- Configuring incomplete, errors occurred!
See also "/home/user/Projects/dataframes/native_libs/build/CMakeFiles/CMakeOutput.log".

[mwu-tow]

Sorry about that, obviously our README is obsolete. I'm afraid that anaconda-based setup hasn't been used by anyone for quite a while. I haven't used it (being on the Windows/MSVC setup), so it's hard to me to give specific advice here.

Basically, you can try either:

1. Push this anaconda thing through — check where physically it placed boost on your drive, set up environment variables as required by CMake, then repeat for other libraries. Manually install ones missing from conda forge, or, if necessary, build from sources.
2. Replicate our CI setup which latest version is here: https://github.com/luna/dataframes/blob/build-library5/.ci/dataframes-package/Dockerfile (the one on master works as well, but this uses newer libraries and we will soon deploy it)
   It is known to work, though requires some work. If you are on reasonable up-to-date distribution, you should be able to obtain a number of libraries through system package manager.
3. Use docker image used by our CI: https://hub.docker.com/r/lunalang/dataframes-package/tags
   That gives you out-of-the-box environment that can build the dataframes library.

[Magalame]

So part of it was purely my fault, I had placed the lines at the end of the file, which makes no sense. Some libraries were still missing: fmt, pybind, date. fmt and pybind are available from anaconda, but I had to manually install date.
Now I'm facing another issue:

CMake Warning at /usr/share/cmake-3.10/Modules/FindBoost.cmake:801 (message):
  New Boost version may have incorrect or missing dependencies and imported
  targets
Call Stack (most recent call first):
  /usr/share/cmake-3.10/Modules/FindBoost.cmake:907 (_Boost_COMPONENT_DEPENDENCIES)
  /usr/share/cmake-3.10/Modules/FindBoost.cmake:1558 (_Boost_MISSING_DEPENDENCIES)
  CMakeLists.txt:28 (find_package)


CMake Warning at /usr/share/cmake-3.10/Modules/FindBoost.cmake:801 (message):
  New Boost version may have incorrect or missing dependencies and imported
  targets
Call Stack (most recent call first):
  /usr/share/cmake-3.10/Modules/FindBoost.cmake:907 (_Boost_COMPONENT_DEPENDENCIES)
  /usr/share/cmake-3.10/Modules/FindBoost.cmake:1558 (_Boost_MISSING_DEPENDENCIES)
  CMakeLists.txt:28 (find_package)


CMake Warning at /usr/share/cmake-3.10/Modules/FindBoost.cmake:801 (message):
  New Boost version may have incorrect or missing dependencies and imported
  targets
Call Stack (most recent call first):
  /usr/share/cmake-3.10/Modules/FindBoost.cmake:907 (_Boost_COMPONENT_DEPENDENCIES)
  /usr/share/cmake-3.10/Modules/FindBoost.cmake:1558 (_Boost_MISSING_DEPENDENCIES)
  CMakeLists.txt:28 (find_package)


-- Boost version: 1.67.0
-- Found the following Boost libraries:
--   filesystem
--   unit_test_framework
--   system
CMake Warning at CMakeLists.txt:51 (message):
  Cannot find xlnt include dir with xlnt/xlnt.hpp.  If it is present,
  consider setting CMAKE_PREFIX_PATH or CMAKE_INCLUDE_PATH.


CMake Warning at CMakeLists.txt:61 (message):
  Cannot find xlnt library.  If it is present, consider setting
  CMAKE_PREFIX_PATH or CMAKE_LIBRARY_PATH.


CMake Warning at CMakeLists.txt:128 (message):
  DataframeHelper will be built without XLSX format support, as xlnt library
  was not found!


-- Configuring done
CMake Error at CMakeLists.txt:166 (add_executable):
  Target "DataframeHelperTests" links to target "Boost::filesystem" but the
  target was not found.  Perhaps a find_package() call is missing for an
  IMPORTED target, or an ALIAS target is missing?


CMake Error at CMakeLists.txt:166 (add_executable):
  Target "DataframeHelperTests" links to target "Boost::unit_test_framework"
  but the target was not found.  Perhaps a find_package() call is missing for
  an IMPORTED target, or an ALIAS target is missing?


CMake Error at CMakeLists.txt:156 (add_library):
  Target "Learn" links to target "Boost::filesystem" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?


CMake Error at CMakeLists.txt:99 (add_library):
  Target "DataframeHelper" links to target "Boost::filesystem" but the target
  was not found.  Perhaps a find_package() call is missing for an IMPORTED
  target, or an ALIAS target is missing?


CMake Error at CMakeLists.txt:146 (add_library):
  Target "DataframePlotter" links to target "Boost::filesystem" but the
  target was not found.  Perhaps a find_package() call is missing for an
  IMPORTED target, or an ALIAS target is missing?

I'm not sure to understand what's going on since it explicitly says before that these libraries were found

[mwu-tow]

That's just my guess, but please try downloading and using the latest CMake release (after properly cleaning the previous build cache). There is a bunch of warnings that Boost version is newer than CMake, that might be a reason why Boost isn't properly recognized.
Otherwise, I have little idea why it first says it has found boost libraries but then fails to import targets.
If using latest CMake doesn't help, try running CMake with -DBoost_DEBUG=ON — that should give us more information what is going on. You might also want try fiddling with other relevant variables, as provided by their docs.

[Magalame]

Thanks a lot for your help. I ran into another issue:

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
/home/user/Projects/dataframes/native_libs/src/PYTHON_NUMPY_INCLUDE_DIR
   used as include directory in directory /home/user/Projects/dataframes/native_libs/src

I solved it by adding find_path(PYTHON_NUMPY_INCLUDE_DIR /home/user/anaconda3/envs/dataframes/lib/python3.6/site-packages/numpy/core/include/) to the CMake list.

But then with make, I hit:

/home/user/Projects/dataframes/native_libs/src/Python/PythonInterpreter.cpp: In member function ‘pybind11::object PythonInterpreter::toPyDateTime(Timestamp) const’:
/home/user/Projects/dataframes/native_libs/src/Python/PythonInterpreter.cpp:223:17: error: missing template arguments before ‘tod’
     time_of_day tod = make_time(timestamp - daypoint); // Yields time_of_day type
                 ^~~
/home/user/Projects/dataframes/native_libs/src/Python/PythonInterpreter.cpp:229:19: error: ‘tod’ was not declared in this scope
     auto h = (int)tod.hours().count();
                   ^~~
/home/user/Projects/dataframes/native_libs/src/Python/PythonInterpreter.cpp:229:19: note: suggested alternative: ‘_mod’
     auto h = (int)tod.hours().count();

[mwu-tow]

@Magalame
Looks like an issue with GCC-7 that we don't cover anymore with CI. GCC-8, MSVC and Clang all accept this (and this should work, I believe, thanks to CTAD). In the mid-term we will be likely dropping support for older GCC's, for now please workaround by replacing the time_of_day with auto. Or, if you like being explicit, by time_of_day<std::chrono::nanoseconds>. I'm fine with both.

[mwu-tow]

@Magalame BTW — for my future reference: what solved the finding Boost issue? Was it the CMake version?

[Magalame]

@mwu-tow I was using gcc 7.4, then I tried again with 8.3, and it failed the same way, the auto fix worked well! But I still hit an error at the very last stage:

/home/user/Projects/dataframes/native_libs/linux/libDataframeHelper.so: undefined reference to `arrow::Schema::ToString[abi:cxx11]() const'
/home/user/Projects/dataframes/native_libs/linux/libDataframeHelper.so: undefined reference to `arrow::Field::ToString[abi:cxx11]() const'
/home/user/Projects/dataframes/native_libs/linux/libDataframeHelper.so: undefined reference to `arrow::ipc::feather::TableWriter::Append(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, arrow::Array const&)'
/home/user/Projects/dataframes/native_libs/linux/libDataframeHelper.so: undefined reference to `arrow::io::FileOutputStream::Open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<arrow::io::FileOutputStream>*)'
/home/user/Projects/dataframes/native_libs/linux/libDataframeHelper.so: undefined reference to `arrow::io::ReadableFile::Open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<arrow::io::ReadableFile>*)'
/home/user/Projects/dataframes/native_libs/linux/libDataframeHelper.so: undefined reference to `arrow::Schema::GetFieldIndex(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
/home/user/Projects/dataframes/native_libs/linux/libDataframeHelper.so: undefined reference to `arrow::Schema::GetFieldByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'

Also, it does seem like it was the cmake version. I re-installed it from the latest sources, and it went through with no trouble

[mwu-tow]

I was using gcc 7.4, then I tried again with 8.3, and it failed the same way, the auto fix worked well!

Huh… so rather not a cimpiler issue. What is the version of date library you use?

But I still hit an error at the very last stage:

All missing symbols have that [abi:cxx11] so it seems that your build of Apache Arrow doesn't support C++11 ABI of libstdc++. I have never really dealt with this, having always built Arrow on my own. It might be specific to the Anaconda-distributed binaries.

You can try forcing dataframes to use old ABI by adding the following line to CMakeList.txt:
add_definitions(-D_GLIBCXX_USE_CXX11_ABI=0)
Or compile the Apache Arrow on your own, so it will have the same compiler defaults as Dataframes.
(btw — Arrow doesn't need to be 0.10, 0.13 works as well)

[Magalame]

I'm using the latest version of date, I pulled master from github, and built it from there, there might be some breaking changes?

I'll try with the add_definitions workaround, so that you can have some feedback, but I'll eventually go with building Arrow myself I think, it seems like a more viable solution

[mwu-tow]

I'm using the latest version of date, I pulled master from github, and built it from there, there might be some breaking changes?

Yep, that's it. The time_of_day definition has changed, it is no longer a class, so class template argument deduction stopped working.
Would you be willing to send us PR with the auto fix?

[Magalame]

So, somehow in the meantime, something with linking numpy broke, so now what I do is:
add "/home/user/anaconda3/envs/dataframes/lib/python3.6/site-packages/numpy/core/include/" to CMAKE_INCLUDE_PATH, then find_path(PYTHON_NUMPY_INCLUDE_DIR numpy)

[Magalame]

now everything works!

[Magalame]

Actually I'm not sure how to actually use the library in Luna Studio now? I mean I probably need to point to the dataframes folder for it to find it

[mwu-tow]

The Luna Studio already ships the dataframes library. There are basically 2 options:

1. overwrite dataframes in Luna Studio — but this is unpleasant, as it requires repacking the installed AppImage file.
2. you should be able to open the Dataframes library (i.e. the built working copy of this repository) as a project in Luna Studio, and it will run https://github.com/luna/dataframes/blob/master/src/Main.luna . Replace the main with your test code, so you can create and show/save your new chart type.

I recommend the latter. (though it has some pitfalls, like old dataframes code still visible — but the opened project has higher priority and will overshadow it)
Also, I find it convenient to use command-line Luna interpreter. It cannot show visualizations but for for
"create-chart-write png" flow should be sufficent — and its generally faster to re-run shell Luna interpreter than restarting Luna Studio after changing the code.