linking sc-ide fails with system boost 1.68, if bundled boost is still there

[dvzrv]

Environment

• Your SuperCollider version: current (as of writing) develop HEAD f27885f74

• Your operating system and version: Arch Linux

• boost 1.68.0

• Qt 5.11.2

Steps to reproduce (for bugs)

AUR build script

Error message (for bugs)

[ 91%] Linking CXX executable scide                                                                                                                                      
/usr/bin/ld: CMakeFiles/SuperCollider.dir/core/main.cpp.o: in function `boost::system::error_category::std_category::equivalent(std::error_code const&, int) const':     
main.cpp:(.text._ZNK5boost6system14error_category12std_category10equivalentERKSt10error_codei[_ZNK5boost6system14error_category12std_category10equivalentERKSt10error_code
i]+0x42): undefined reference to `boost::system::generic_category()'                                                                                                     
/usr/bin/ld: main.cpp:(.text._ZNK5boost6system14error_category12std_category10equivalentERKSt10error_codei[_ZNK5boost6system14error_category12std_category10equivalentERKS
t10error_codei]+0xba): undefined reference to `boost::system::generic_category()'                                                                                        
/usr/bin/ld: main.cpp:(.text._ZNK5boost6system14error_category12std_category10equivalentERKSt10error_codei[_ZNK5boost6system14error_category12std_category10equivalentERKS
t10error_codei]+0x142): undefined reference to `boost::system::generic_category()'                                                                                       
/usr/bin/ld: CMakeFiles/SuperCollider.dir/core/main.cpp.o: in function `boost::system::error_category::std_category::equivalent(int, std::error_condition const&) const':
main.cpp:(.text._ZNK5boost6system14error_category12std_category10equivalentEiRKSt15error_condition[_ZNK5boost6system14error_category12std_category10equivalentEiRKSt15erro
r_condition]+0x41): undefined reference to `boost::system::generic_category()'                                                                                           
/usr/bin/ld: main.cpp:(.text._ZNK5boost6system14error_category12std_category10equivalentEiRKSt15error_condition[_ZNK5boost6system14error_category12std_category10equivalen
tEiRKSt15error_condition]+0xd2): undefined reference to `boost::system::generic_category()'                                                                              
collect2: error: ld returned 1 exit status                                                                                                                               
make[2]: *** [editors/sc-ide/CMakeFiles/SuperCollider.dir/build.make:1994: editors/sc-ide/scide] Error 1                                                                 
make[1]: *** [CMakeFiles/Makefile2:3599: editors/sc-ide/CMakeFiles/SuperCollider.dir/all] Error 2                                                                        
make: *** [Makefile:163: all] Error 2                                                                                                                                    

Full build log:
supercollider-boost1.68.0.txt

Expected Behavior

Linking for sc-ide executable succeeds.

Current Behavior

Linking fails with above output.

[dvzrv]

For 3.9.3 and the boost 1.68.0 rebuild the external_libraries/boost had to be explicitely removed to make sure those headers were not used.
I guess this means, that the CmakeLists.txt has to be fixed to really prefer system boost if that is selected (which it is in AUR and the official community repositories)!

[mossheim]

For 3.9.3 and the boost 1.68.0 rebuild the external_libraries/boost had to be explicitely removed to make sure those headers were not used.
I guess this means, that the CmakeLists.txt has to be fixed to really prefer system boost if that is selected (which it is in AUR and the official community repositories)!

Can you please open a second issue for that if there isn't one already? It is irrelevant to this ticket.

[mossheim]

I noticed this the other day while checking out whether we could upgrade locally. The problem (which I'm actually surprised wasn't an issue earlier) is that because of supernova using C++14, we are attempting to link a C++14 build of the library with code using the headers in C++11 mode. Arch Linux requires that you use the latest boost libraries and don't use anything bundled here right?

I think the easiest thing to do would be for us to move to C++14 for the entire project. Some of the devs have discussed this before and I've heard no strong opposition, but it is something that needs more discussion.

For now, if using older releases of boost is not an option, you could force compilation of the entire project in C++14. Not sure off the top of my head how, you might have to patch the main CMakeLists.txt. I don't think passing it as a flag on the command line would work as I'm pretty sure the C++11 flag will come later and take precedence.

[mossheim]

This should be enough. I don't have 1.68 on my machine anymore (Homebrew hasn't updated their formula yet) so I can't test:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 0de34a5a2..4e9520b6e 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -297,7 +297,7 @@ if (CMAKE_COMPILER_IS_GNUCXX OR CMAKE_COMPILER_IS_CLANG)
             set(CMAKE_CXX_LINK_FLAGS "${CMAKE_CXX_LINK_FLAGS} -stdlib=libc++")
         endif()
     else()
-        set( CMAKE_CXX_STANDARD 11 )
+        set( CMAKE_CXX_STANDARD 14 )

         if(CMAKE_COMPILER_IS_CLANG)
             # workaround

[dvzrv]

Hm, the problem seems to be, that cmake (for some bizarre reason) doesn't pickup the system boost (also yamlcpp is not detected!). However, this works with the 3.9.3 release and everything is properly compiled there using system boost and yamlcpp!
If the external_libraries/boost directory is deleted on current HEAD, the build fails (because it always wants to compile boost itself and then misses the sources).
If the external_libraries/boost directory is left untouched, it leads to the above issue.

[dvzrv]

Eh... sorry, please disregard (parts of) my last comment (I broke my build script with a bunch of tests leading to flaky behavior - I striked through what doesn't apply!).

I narrowed this down to the following behavior (by selectively rebuilding with different approaches):

• do not apply modifications - fails (tries to link against bundled boost)
• setting C++14 - fails (tries to link against bundled boost)
• removing external_libraries/boost - builds successfully (with system libraries)
• removing external_libraries/boost, setting C++14 - builds successfully (with system libraries)

Therefore I assume that my experienced behavior is unrelated to setting C++14, but is solely triggered by having the bundled boost around, which is then (also?) linked against.

[mossheim]

@dvzrv If you could supply verbose build logs, including configuration phase and the output of cmake -LAH for these results I might be able to help you. I am not on a Linux system so our environments and build procedures differ a lot. Please also make extra certain you are starting from a clean build each time.

Are the sytem boost libraries you're linking with prebuilt? Are you required to use them?

[dvzrv]

Alright. I did the following:

# this is only relevant in the build script, but boils down to: clone supercollider and its submodules separately, then configure the submodules to point to the separate clones
  cd "${_name}"
  git submodule init
  git config submodule.external_libraries/nova-simd.url "${srcdir}/nova-simd"
  git config submodule.external_libraries/nova-tt.url "${srcdir}/nova-tt"
  git config submodule.external_libraries/hidapi.url "${srcdir}/hidapi"
  git config submodule.external_libraries/portaudio_sc_org.url "${srcdir}/portaudio"
  git config submodule.external_libraries/portaudio_sc_org.branch "${_name}"
  git config submodule.external_libraries/yaml-cpp.url "${srcdir}/yaml-cpp"
  git submodule update

# from here on it's how we build this
  mkdir -p build
  cd build
    cmake -DCMAKE_INSTALL_PREFIX=/usr \
          -DCMAKE_BUILD_TYPE=Release \
          -DLIBSCSYNTH=ON \
          -DFORTIFY=ON \
          -DSC_VIM=OFF \
          -DSYSTEM_BOOST=ON \
          -DSYSTEM_YAMLCPP=ON \
          ..
    cmake -LAH ..
# not sure the VERBOSE=1 actually does anything...
  make VERBOSE=1

Build log:
supercollider-boost1.68.0_verbose.txt

[dvzrv]

And to answer the remaining questions: Yes, pristine build environment every time (containered build, using systemd-nspawn). Yes, the boost libraries are pre-built and yes, ideally I link against what my distribution offers, because shared lib reusability ftw! :)

[dvzrv]

I think I know why this happens:
external_libraries/oscpack is linked against external_libraries/boost, even if the external boost is used (see line 2249 of my build log). Later on scide is linked against oscpack and boom.

[mossheim]

@dvzrv good catch. :) Does this do anything for you?

diff --git a/editors/sc-ide/CMakeLists.txt b/editors/sc-ide/CMakeLists.txt
index 50059bc2f..60d984333 100644
--- a/editors/sc-ide/CMakeLists.txt
+++ b/editors/sc-ide/CMakeLists.txt
@@ -216,10 +216,15 @@ elseif(WIN32)
     list(APPEND ide_src ${CMAKE_SOURCE_DIR}/common/SC_Win32Utils.cpp)
 endif()

+if (Boost_FOUND)
+    include_directories(${Boost_INCLUDE_DIRS})
+else()
+    include_directories(${CMAKE_SOURCE_DIR}/external_libraries/boost)
+endif()
+
 include_directories(${CMAKE_SOURCE_DIR}/include/common)
 include_directories(${CMAKE_SOURCE_DIR}/common)
 include_directories(${CMAKE_SOURCE_DIR}/include/plugin_interface)
-include_directories(${CMAKE_SOURCE_DIR}/external_libraries/boost)
 include_directories(${YAMLCPP_INCLUDE_DIR})
 # For QtCollider headers:
 include_directories(${CMAKE_SOURCE_DIR})
diff --git a/external_libraries/CMakeLists.txt b/external_libraries/CMakeLists.txt
index df33dc6f1..e1f28b670 100644
--- a/external_libraries/CMakeLists.txt
+++ b/external_libraries/CMakeLists.txt
@@ -84,7 +84,13 @@ set_property(TARGET tlsf PROPERTY POSITION_INDEPENDENT_CODE TRUE)

 #oscpack
 add_library(oscpack STATIC EXCLUDE_FROM_ALL "oscpack_build.cpp")
-target_include_directories(oscpack PRIVATE boost)
+
+if (Boost_FOUND)
+    target_include_directories(oscpack PRIVATE ${Boost_INCLUDE_DIRS})
+else()
+    target_include_directories(oscpack PRIVATE boost)
+endif()
+
 target_include_directories(oscpack INTERFACE oscpack_1_1_0 )

This is a hasty patch, ideally we should switch on SYSTEM_BOOST not Boost_FOUND, and then use Boost_INCLUDE_DIRS everywhere, like we do for yamlcpp. I just want to make sure this more obvious brute-force solution works first as a proof-of-concept before trying the more elegant one.

[dvzrv]

@brianlheim thanks for the fast response. I'll try asap.
I think another candidate is yaml-cpp itself, when using the bundled version (which I'm not for packaging): https://github.com/supercollider/supercollider/blob/develop/external_libraries/CMakeLists.txt#L107

[dvzrv]

Hm, no, seems we have some more dangling references here and there. This is what happens with you proposed patch:

Scanning dependencies of target server_dsp_thread_test
[ 97%] Building CXX object testsuite/supernova/CMakeFiles/server_dsp_thread_test.dir/server_dsp_thread_test.cpp.o
[ 97%] Linking CXX executable server_dsp_thread_test
/usr/bin/ld: CMakeFiles/server_dsp_thread_test.dir/server_dsp_thread_test.cpp.o: in function `void boost::sync::detail::throw_exception<boost::sync::resource_error, int const&, char const (&) [54]>(char const*, char const*, int, int const&, char const (&) [54])':                                                                                                                       
server_dsp_thread_test.cpp:(.text._ZN5boost4sync6detail15throw_exceptionINS0_14resource_errorEJRKiRA54_KcEEEvPS6_S9_iDpOT0_[_ZN5boost4sync6detail15throw_exceptionINS0_14resource_errorEJRKiRA54_KcEEEvPS6_S9_iDpOT0_]+0x3c): undefined reference to `boost::system::system_category()'                                                                                                       
/usr/bin/ld: CMakeFiles/server_dsp_thread_test.dir/server_dsp_thread_test.cpp.o: in function `void boost::sync::detail::throw_exception<boost::sync::overflow_error, int const&, char const (&) [50]>(char const*, char const*, int, int const&, char const (&) [50])':                                                                                                                       
server_dsp_thread_test.cpp:(.text._ZN5boost4sync6detail15throw_exceptionINS0_14overflow_errorEJRKiRA50_KcEEEvPS6_S9_iDpOT0_[_ZN5boost4sync6detail15throw_exceptionINS0_14overflow_errorEJRKiRA50_KcEEEvPS6_S9_iDpOT0_]+0x3c): undefined reference to `boost::system::system_category()'                                                                                                       
/usr/bin/ld: CMakeFiles/server_dsp_thread_test.dir/server_dsp_thread_test.cpp.o: in function `_GLOBAL__sub_I__ZN17dsp_thread_test_111test_methodEv':
server_dsp_thread_test.cpp:(.text.startup+0x5f): undefined reference to `boost::system::generic_category()'
/usr/bin/ld: server_dsp_thread_test.cpp:(.text.startup+0x65): undefined reference to `boost::system::generic_category()'
/usr/bin/ld: server_dsp_thread_test.cpp:(.text.startup+0x6b): undefined reference to `boost::system::system_category()'
collect2: error: ld returned 1 exit status
make[2]: *** [testsuite/supernova/CMakeFiles/server_dsp_thread_test.dir/build.make:96: testsuite/supernova/server_dsp_thread_test] Error 1
make[1]: *** [CMakeFiles/Makefile2:4398: testsuite/supernova/CMakeFiles/server_dsp_thread_test.dir/all] Error 2
make: *** [Makefile:163: all] Error 2

[mossheim]

@dvzrv I'm fairly certain this is due to the C++11/14 mismatch I mentioned above.

[revast]

having the same problems with 3.9.3 and 3.10 beta2 on Ubuntu 18.04.

when I apply all patches from above, also the C++11/14 mismatch one, I get

libsclang.a(PyrStringPrim.cpp.o): In function boost::re_detail_106600::perl_matcher<char const*, std::allocator<boost::sub_match<char const*> >, boost::regex_traits<char, boost::cpp_regex_traits > >::unwind_extra_block(bool)':
./obj-x86_64-linux-gnu/lang/./external_libraries/boost/boost/regex/v4/perl_matcher_non_recursive.hpp:1368: undefined reference to boost::re_detail_106600::put_mem_block(void*)' libsclang.a(PyrStringPrim.cpp.o): In function char const* boost::re_detail_106600::re_is_set_member<char const*, char, boost::regex_traits<char`

https://pastebin.com/9TJJQuES

[dvzrv]

Sorry for the long delay!

I have now fixed this for my packaging use-case with the following patch (will open a pull request shortly) for 3.10.0 (as I have to update the package):

diff -ruN a/editors/sc-ide/CMakeLists.txt b/editors/sc-ide/CMakeLists.txt
--- a/editors/sc-ide/CMakeLists.txt	2018-09-25 01:20:49.000000000 +0200
+++ b/editors/sc-ide/CMakeLists.txt	2018-11-28 15:53:19.827910183 +0100
@@ -219,7 +219,6 @@
 include_directories(${CMAKE_SOURCE_DIR}/include/common)
 include_directories(${CMAKE_SOURCE_DIR}/common)
 include_directories(${CMAKE_SOURCE_DIR}/include/plugin_interface)
-include_directories(${CMAKE_SOURCE_DIR}/external_libraries/boost)
 include_directories(${YAMLCPP_INCLUDE_DIR})
 # For QtCollider headers:
 include_directories(${CMAKE_SOURCE_DIR})
@@ -304,6 +303,7 @@
 else()
     # in-house-built boost system
     target_link_libraries( SuperCollider boost_system boost_filesystem )
+    include_directories(${CMAKE_SOURCE_DIR}/external_libraries/boost)
 endif()
 
 if(APPLE)
diff -ruN a/external_libraries/CMakeLists.txt b/external_libraries/CMakeLists.txt
--- a/external_libraries/CMakeLists.txt	2018-08-28 03:12:47.000000000 +0200
+++ b/external_libraries/CMakeLists.txt	2018-11-28 17:39:57.981205691 +0100
@@ -74,6 +74,14 @@
         PROPERTY FOLDER 3rdparty
     )
 
+  #oscpack
+  add_library(oscpack STATIC EXCLUDE_FROM_ALL "oscpack_build.cpp")
+  target_include_directories(oscpack PRIVATE boost)
+  target_include_directories(oscpack INTERFACE oscpack_1_1_0 )
+else()
+  #oscpack
+  add_library(oscpack STATIC EXCLUDE_FROM_ALL "oscpack_build.cpp")
+  target_include_directories(oscpack INTERFACE oscpack_1_1_0 )
 endif()
 
 # tlsf
@@ -82,11 +90,6 @@
 target_include_directories( tlsf INTERFACE TLSF-2.4.6/src )
 set_property(TARGET tlsf PROPERTY POSITION_INDEPENDENT_CODE TRUE)
 
-#oscpack
-add_library(oscpack STATIC EXCLUDE_FROM_ALL "oscpack_build.cpp")
-target_include_directories(oscpack PRIVATE boost)
-target_include_directories(oscpack INTERFACE oscpack_1_1_0 )
-
 
 if(LTO)
 	set_property(TARGET oscpack tlsf
@@ -104,7 +107,11 @@
   CREATE_FINAL_FILE(${CMAKE_CURRENT_BINARY_DIR}/libyamlcpp.cpp ${yaml_src})
 
   add_library(yaml STATIC EXCLUDE_FROM_ALL ${CMAKE_CURRENT_BINARY_DIR}/libyamlcpp.cpp)
-  target_include_directories(yaml PUBLIC ${CMAKE_SOURCE_DIR}/external_libraries/yaml-cpp/include boost)
+  if(Boost_FOUND)
+    target_include_directories(yaml PUBLIC ${Boost_INCLUDE_DIRS})
+  else()
+    target_include_directories(yaml PUBLIC ${CMAKE_SOURCE_DIR}/external_libraries/yaml-cpp/include boost)
+  endif()
   set_property( TARGET yaml PROPERTY FOLDER 3rdparty )
   set(YAMLCPP_LIBRARY yaml)
   set(YAMLCPP_INCLUDE_DIR ${CMAKE_SOURCE_DIR}/external_libraries/yaml-cpp/include)
diff -ruN a/testsuite/server/CMakeLists.txt b/testsuite/server/CMakeLists.txt
--- a/testsuite/server/CMakeLists.txt	2018-08-28 03:12:47.000000000 +0200
+++ b/testsuite/server/CMakeLists.txt	2018-11-28 16:24:08.449255929 +0100
@@ -5,4 +5,8 @@
 endif()
 
 add_library(boost_test STATIC boost_test.cpp)
-target_include_directories(boost_test PUBLIC ${CMAKE_SOURCE_DIR}/external_libraries/boost)
+if (Boost_FOUND)
+  target_include_directories(boost_test PUBLIC ${Boost_INCLUDE_DIRS})
+else()
+  target_include_directories(boost_test PUBLIC ${CMAKE_SOURCE_DIR}/external_libraries/boost)
+endif()

Multiple components are (additionally) explicitely linking against external_libraries/boost or are including headers from there (even if BOOST_FOUND), so I changed that (conditionally). However, now I have problems building the vendored yaml-cpp against system boost 1.68 (I guess that validates yet another bug report or maybe one has to look into updating yaml-cpp to >= 0.6 to make it drop the boost dependency altogether) and I have a general problem building 3.10.0 (with vendored external libs), which prevents me from testing against my patch (fails the same way with and without my patch though!).

[dvzrv]

I have now based my changes on the develop branch (as that's what I build supercollider-git in the AUR from). I'm not sure it makes sense to only hotfix this in 3.10.1 and I think my fix is potentially not complete yet.

[mossheim]

Closed in #4185 -- thanks @dvzrv for your patience and work on that!