Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

linking sc-ide fails with system boost 1.68, if bundled boost is still there #4096

Closed
dvzrv opened this issue Sep 29, 2018 · 18 comments

Comments

@dvzrv
Copy link
Member

commented Sep 29, 2018

Environment

  • Your SuperCollider version: current (as of writing) develop HEAD f27885f74

  • Your operating system and version: Arch Linux

  • boost 1.68.0

  • Qt 5.11.2

Steps to reproduce (for bugs)

AUR build script

Error message (for bugs)

[ 91%] Linking CXX executable scide                                                                                                                                      
/usr/bin/ld: CMakeFiles/SuperCollider.dir/core/main.cpp.o: in function `boost::system::error_category::std_category::equivalent(std::error_code const&, int) const':     
main.cpp:(.text._ZNK5boost6system14error_category12std_category10equivalentERKSt10error_codei[_ZNK5boost6system14error_category12std_category10equivalentERKSt10error_code
i]+0x42): undefined reference to `boost::system::generic_category()'                                                                                                     
/usr/bin/ld: main.cpp:(.text._ZNK5boost6system14error_category12std_category10equivalentERKSt10error_codei[_ZNK5boost6system14error_category12std_category10equivalentERKS
t10error_codei]+0xba): undefined reference to `boost::system::generic_category()'                                                                                        
/usr/bin/ld: main.cpp:(.text._ZNK5boost6system14error_category12std_category10equivalentERKSt10error_codei[_ZNK5boost6system14error_category12std_category10equivalentERKS
t10error_codei]+0x142): undefined reference to `boost::system::generic_category()'                                                                                       
/usr/bin/ld: CMakeFiles/SuperCollider.dir/core/main.cpp.o: in function `boost::system::error_category::std_category::equivalent(int, std::error_condition const&) const':
main.cpp:(.text._ZNK5boost6system14error_category12std_category10equivalentEiRKSt15error_condition[_ZNK5boost6system14error_category12std_category10equivalentEiRKSt15erro
r_condition]+0x41): undefined reference to `boost::system::generic_category()'                                                                                           
/usr/bin/ld: main.cpp:(.text._ZNK5boost6system14error_category12std_category10equivalentEiRKSt15error_condition[_ZNK5boost6system14error_category12std_category10equivalen
tEiRKSt15error_condition]+0xd2): undefined reference to `boost::system::generic_category()'                                                                              
collect2: error: ld returned 1 exit status                                                                                                                               
make[2]: *** [editors/sc-ide/CMakeFiles/SuperCollider.dir/build.make:1994: editors/sc-ide/scide] Error 1                                                                 
make[1]: *** [CMakeFiles/Makefile2:3599: editors/sc-ide/CMakeFiles/SuperCollider.dir/all] Error 2                                                                        
make: *** [Makefile:163: all] Error 2                                                                                                                                    

Full build log:
supercollider-boost1.68.0.txt

Expected Behavior

Linking for sc-ide executable succeeds.

Current Behavior

Linking fails with above output.

@dvzrv

This comment has been minimized.

Copy link
Member Author

commented Sep 29, 2018

For 3.9.3 and the boost 1.68.0 rebuild the external_libraries/boost had to be explicitely removed to make sure those headers were not used.
I guess this means, that the CmakeLists.txt has to be fixed to really prefer system boost if that is selected (which it is in AUR and the official community repositories)!

@brianlheim

This comment has been minimized.

Copy link
Member

commented Sep 29, 2018

For 3.9.3 and the boost 1.68.0 rebuild the external_libraries/boost had to be explicitely removed to make sure those headers were not used.
I guess this means, that the CmakeLists.txt has to be fixed to really prefer system boost if that is selected (which it is in AUR and the official community repositories)!

Can you please open a second issue for that if there isn't one already? It is irrelevant to this ticket.

@brianlheim

This comment has been minimized.

Copy link
Member

commented Sep 29, 2018

I noticed this the other day while checking out whether we could upgrade locally. The problem (which I'm actually surprised wasn't an issue earlier) is that because of supernova using C++14, we are attempting to link a C++14 build of the library with code using the headers in C++11 mode. Arch Linux requires that you use the latest boost libraries and don't use anything bundled here right?

I think the easiest thing to do would be for us to move to C++14 for the entire project. Some of the devs have discussed this before and I've heard no strong opposition, but it is something that needs more discussion.

For now, if using older releases of boost is not an option, you could force compilation of the entire project in C++14. Not sure off the top of my head how, you might have to patch the main CMakeLists.txt. I don't think passing it as a flag on the command line would work as I'm pretty sure the C++11 flag will come later and take precedence.

@brianlheim

This comment has been minimized.

Copy link
Member

commented Sep 29, 2018

This should be enough. I don't have 1.68 on my machine anymore (Homebrew hasn't updated their formula yet) so I can't test:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 0de34a5a2..4e9520b6e 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -297,7 +297,7 @@ if (CMAKE_COMPILER_IS_GNUCXX OR CMAKE_COMPILER_IS_CLANG)
             set(CMAKE_CXX_LINK_FLAGS "${CMAKE_CXX_LINK_FLAGS} -stdlib=libc++")
         endif()
     else()
-        set( CMAKE_CXX_STANDARD 11 )
+        set( CMAKE_CXX_STANDARD 14 )

         if(CMAKE_COMPILER_IS_CLANG)
             # workaround
@dvzrv

This comment has been minimized.

Copy link
Member Author

commented Sep 29, 2018

Hm, the problem seems to be, that cmake (for some bizarre reason) doesn't pickup the system boost (also yamlcpp is not detected!). However, this works with the 3.9.3 release and everything is properly compiled there using system boost and yamlcpp!
If the external_libraries/boost directory is deleted on current HEAD, the build fails (because it always wants to compile boost itself and then misses the sources).

If the external_libraries/boost directory is left untouched, it leads to the above issue.

@dvzrv

This comment has been minimized.

Copy link
Member Author

commented Sep 29, 2018

Eh... sorry, please disregard (parts of) my last comment (I broke my build script with a bunch of tests leading to flaky behavior - I striked through what doesn't apply!).

I narrowed this down to the following behavior (by selectively rebuilding with different approaches):

  • do not apply modifications - fails (tries to link against bundled boost)
  • setting C++14 - fails (tries to link against bundled boost)
  • removing external_libraries/boost - builds successfully (with system libraries)
  • removing external_libraries/boost, setting C++14 - builds successfully (with system libraries)

Therefore I assume that my experienced behavior is unrelated to setting C++14, but is solely triggered by having the bundled boost around, which is then (also?) linked against.

@dvzrv dvzrv changed the title linking sc-ide fails with boost 1.68 linking sc-ide fails with system boost 1.68, if bundled boost is still there Sep 29, 2018
@brianlheim

This comment has been minimized.

Copy link
Member

commented Sep 30, 2018

@dvzrv If you could supply verbose build logs, including configuration phase and the output of cmake -LAH for these results I might be able to help you. I am not on a Linux system so our environments and build procedures differ a lot. Please also make extra certain you are starting from a clean build each time.

Are the sytem boost libraries you're linking with prebuilt? Are you required to use them?

@dvzrv

This comment has been minimized.

Copy link
Member Author

commented Oct 1, 2018

Alright. I did the following:

# this is only relevant in the build script, but boils down to: clone supercollider and its submodules separately, then configure the submodules to point to the separate clones
  cd "${_name}"
  git submodule init
  git config submodule.external_libraries/nova-simd.url "${srcdir}/nova-simd"
  git config submodule.external_libraries/nova-tt.url "${srcdir}/nova-tt"
  git config submodule.external_libraries/hidapi.url "${srcdir}/hidapi"
  git config submodule.external_libraries/portaudio_sc_org.url "${srcdir}/portaudio"
  git config submodule.external_libraries/portaudio_sc_org.branch "${_name}"
  git config submodule.external_libraries/yaml-cpp.url "${srcdir}/yaml-cpp"
  git submodule update

# from here on it's how we build this
  mkdir -p build
  cd build
    cmake -DCMAKE_INSTALL_PREFIX=/usr \
          -DCMAKE_BUILD_TYPE=Release \
          -DLIBSCSYNTH=ON \
          -DFORTIFY=ON \
          -DSC_VIM=OFF \
          -DSYSTEM_BOOST=ON \
          -DSYSTEM_YAMLCPP=ON \
          ..
    cmake -LAH ..
# not sure the VERBOSE=1 actually does anything...
  make VERBOSE=1

Build log:
supercollider-boost1.68.0_verbose.txt

@dvzrv

This comment has been minimized.

Copy link
Member Author

commented Oct 1, 2018

And to answer the remaining questions: Yes, pristine build environment every time (containered build, using systemd-nspawn). Yes, the boost libraries are pre-built and yes, ideally I link against what my distribution offers, because shared lib reusability ftw! :)

@dvzrv

This comment has been minimized.

Copy link
Member Author

commented Oct 3, 2018

I think I know why this happens:
external_libraries/oscpack is linked against external_libraries/boost, even if the external boost is used (see line 2249 of my build log). Later on scide is linked against oscpack and boom.

@brianlheim

This comment has been minimized.

Copy link
Member

commented Oct 7, 2018

@dvzrv good catch. :) Does this do anything for you?

diff --git a/editors/sc-ide/CMakeLists.txt b/editors/sc-ide/CMakeLists.txt
index 50059bc2f..60d984333 100644
--- a/editors/sc-ide/CMakeLists.txt
+++ b/editors/sc-ide/CMakeLists.txt
@@ -216,10 +216,15 @@ elseif(WIN32)
     list(APPEND ide_src ${CMAKE_SOURCE_DIR}/common/SC_Win32Utils.cpp)
 endif()

+if (Boost_FOUND)
+    include_directories(${Boost_INCLUDE_DIRS})
+else()
+    include_directories(${CMAKE_SOURCE_DIR}/external_libraries/boost)
+endif()
+
 include_directories(${CMAKE_SOURCE_DIR}/include/common)
 include_directories(${CMAKE_SOURCE_DIR}/common)
 include_directories(${CMAKE_SOURCE_DIR}/include/plugin_interface)
-include_directories(${CMAKE_SOURCE_DIR}/external_libraries/boost)
 include_directories(${YAMLCPP_INCLUDE_DIR})
 # For QtCollider headers:
 include_directories(${CMAKE_SOURCE_DIR})
diff --git a/external_libraries/CMakeLists.txt b/external_libraries/CMakeLists.txt
index df33dc6f1..e1f28b670 100644
--- a/external_libraries/CMakeLists.txt
+++ b/external_libraries/CMakeLists.txt
@@ -84,7 +84,13 @@ set_property(TARGET tlsf PROPERTY POSITION_INDEPENDENT_CODE TRUE)

 #oscpack
 add_library(oscpack STATIC EXCLUDE_FROM_ALL "oscpack_build.cpp")
-target_include_directories(oscpack PRIVATE boost)
+
+if (Boost_FOUND)
+    target_include_directories(oscpack PRIVATE ${Boost_INCLUDE_DIRS})
+else()
+    target_include_directories(oscpack PRIVATE boost)
+endif()
+
 target_include_directories(oscpack INTERFACE oscpack_1_1_0 )

This is a hasty patch, ideally we should switch on SYSTEM_BOOST not Boost_FOUND, and then use Boost_INCLUDE_DIRS everywhere, like we do for yamlcpp. I just want to make sure this more obvious brute-force solution works first as a proof-of-concept before trying the more elegant one.

@dvzrv

This comment has been minimized.

Copy link
Member Author

commented Oct 12, 2018

@brianlheim thanks for the fast response. I'll try asap.
I think another candidate is yaml-cpp itself, when using the bundled version (which I'm not for packaging): https://github.com/supercollider/supercollider/blob/develop/external_libraries/CMakeLists.txt#L107

@dvzrv

This comment has been minimized.

Copy link
Member Author

commented Oct 19, 2018

Hm, no, seems we have some more dangling references here and there. This is what happens with you proposed patch:

Scanning dependencies of target server_dsp_thread_test
[ 97%] Building CXX object testsuite/supernova/CMakeFiles/server_dsp_thread_test.dir/server_dsp_thread_test.cpp.o
[ 97%] Linking CXX executable server_dsp_thread_test
/usr/bin/ld: CMakeFiles/server_dsp_thread_test.dir/server_dsp_thread_test.cpp.o: in function `void boost::sync::detail::throw_exception<boost::sync::resource_error, int const&, char const (&) [54]>(char const*, char const*, int, int const&, char const (&) [54])':                                                                                                                       
server_dsp_thread_test.cpp:(.text._ZN5boost4sync6detail15throw_exceptionINS0_14resource_errorEJRKiRA54_KcEEEvPS6_S9_iDpOT0_[_ZN5boost4sync6detail15throw_exceptionINS0_14resource_errorEJRKiRA54_KcEEEvPS6_S9_iDpOT0_]+0x3c): undefined reference to `boost::system::system_category()'                                                                                                       
/usr/bin/ld: CMakeFiles/server_dsp_thread_test.dir/server_dsp_thread_test.cpp.o: in function `void boost::sync::detail::throw_exception<boost::sync::overflow_error, int const&, char const (&) [50]>(char const*, char const*, int, int const&, char const (&) [50])':                                                                                                                       
server_dsp_thread_test.cpp:(.text._ZN5boost4sync6detail15throw_exceptionINS0_14overflow_errorEJRKiRA50_KcEEEvPS6_S9_iDpOT0_[_ZN5boost4sync6detail15throw_exceptionINS0_14overflow_errorEJRKiRA50_KcEEEvPS6_S9_iDpOT0_]+0x3c): undefined reference to `boost::system::system_category()'                                                                                                       
/usr/bin/ld: CMakeFiles/server_dsp_thread_test.dir/server_dsp_thread_test.cpp.o: in function `_GLOBAL__sub_I__ZN17dsp_thread_test_111test_methodEv':
server_dsp_thread_test.cpp:(.text.startup+0x5f): undefined reference to `boost::system::generic_category()'
/usr/bin/ld: server_dsp_thread_test.cpp:(.text.startup+0x65): undefined reference to `boost::system::generic_category()'
/usr/bin/ld: server_dsp_thread_test.cpp:(.text.startup+0x6b): undefined reference to `boost::system::system_category()'
collect2: error: ld returned 1 exit status
make[2]: *** [testsuite/supernova/CMakeFiles/server_dsp_thread_test.dir/build.make:96: testsuite/supernova/server_dsp_thread_test] Error 1
make[1]: *** [CMakeFiles/Makefile2:4398: testsuite/supernova/CMakeFiles/server_dsp_thread_test.dir/all] Error 2
make: *** [Makefile:163: all] Error 2
@brianlheim

This comment has been minimized.

Copy link
Member

commented Oct 19, 2018

@dvzrv I'm fairly certain this is due to the C++11/14 mismatch I mentioned above.

@revast

This comment has been minimized.

Copy link

commented Oct 30, 2018

having the same problems with 3.9.3 and 3.10 beta2 on Ubuntu 18.04.

when I apply all patches from above, also the C++11/14 mismatch one, I get

libsclang.a(PyrStringPrim.cpp.o): In function boost::re_detail_106600::perl_matcher<char const*, std::allocator<boost::sub_match<char const*> >, boost::regex_traits<char, boost::cpp_regex_traits > >::unwind_extra_block(bool)':
./obj-x86_64-linux-gnu/lang/./external_libraries/boost/boost/regex/v4/perl_matcher_non_recursive.hpp:1368: undefined reference to boost::re_detail_106600::put_mem_block(void*)' libsclang.a(PyrStringPrim.cpp.o): In function char const* boost::re_detail_106600::re_is_set_member<char const*, char, boost::regex_traits<char`

https://pastebin.com/9TJJQuES

@dvzrv

This comment has been minimized.

Copy link
Member Author

commented Nov 28, 2018

Sorry for the long delay!

I have now fixed this for my packaging use-case with the following patch (will open a pull request shortly) for 3.10.0 (as I have to update the package):

diff -ruN a/editors/sc-ide/CMakeLists.txt b/editors/sc-ide/CMakeLists.txt
--- a/editors/sc-ide/CMakeLists.txt	2018-09-25 01:20:49.000000000 +0200
+++ b/editors/sc-ide/CMakeLists.txt	2018-11-28 15:53:19.827910183 +0100
@@ -219,7 +219,6 @@
 include_directories(${CMAKE_SOURCE_DIR}/include/common)
 include_directories(${CMAKE_SOURCE_DIR}/common)
 include_directories(${CMAKE_SOURCE_DIR}/include/plugin_interface)
-include_directories(${CMAKE_SOURCE_DIR}/external_libraries/boost)
 include_directories(${YAMLCPP_INCLUDE_DIR})
 # For QtCollider headers:
 include_directories(${CMAKE_SOURCE_DIR})
@@ -304,6 +303,7 @@
 else()
     # in-house-built boost system
     target_link_libraries( SuperCollider boost_system boost_filesystem )
+    include_directories(${CMAKE_SOURCE_DIR}/external_libraries/boost)
 endif()
 
 if(APPLE)
diff -ruN a/external_libraries/CMakeLists.txt b/external_libraries/CMakeLists.txt
--- a/external_libraries/CMakeLists.txt	2018-08-28 03:12:47.000000000 +0200
+++ b/external_libraries/CMakeLists.txt	2018-11-28 17:39:57.981205691 +0100
@@ -74,6 +74,14 @@
         PROPERTY FOLDER 3rdparty
     )
 
+  #oscpack
+  add_library(oscpack STATIC EXCLUDE_FROM_ALL "oscpack_build.cpp")
+  target_include_directories(oscpack PRIVATE boost)
+  target_include_directories(oscpack INTERFACE oscpack_1_1_0 )
+else()
+  #oscpack
+  add_library(oscpack STATIC EXCLUDE_FROM_ALL "oscpack_build.cpp")
+  target_include_directories(oscpack INTERFACE oscpack_1_1_0 )
 endif()
 
 # tlsf
@@ -82,11 +90,6 @@
 target_include_directories( tlsf INTERFACE TLSF-2.4.6/src )
 set_property(TARGET tlsf PROPERTY POSITION_INDEPENDENT_CODE TRUE)
 
-#oscpack
-add_library(oscpack STATIC EXCLUDE_FROM_ALL "oscpack_build.cpp")
-target_include_directories(oscpack PRIVATE boost)
-target_include_directories(oscpack INTERFACE oscpack_1_1_0 )
-
 
 if(LTO)
 	set_property(TARGET oscpack tlsf
@@ -104,7 +107,11 @@
   CREATE_FINAL_FILE(${CMAKE_CURRENT_BINARY_DIR}/libyamlcpp.cpp ${yaml_src})
 
   add_library(yaml STATIC EXCLUDE_FROM_ALL ${CMAKE_CURRENT_BINARY_DIR}/libyamlcpp.cpp)
-  target_include_directories(yaml PUBLIC ${CMAKE_SOURCE_DIR}/external_libraries/yaml-cpp/include boost)
+  if(Boost_FOUND)
+    target_include_directories(yaml PUBLIC ${Boost_INCLUDE_DIRS})
+  else()
+    target_include_directories(yaml PUBLIC ${CMAKE_SOURCE_DIR}/external_libraries/yaml-cpp/include boost)
+  endif()
   set_property( TARGET yaml PROPERTY FOLDER 3rdparty )
   set(YAMLCPP_LIBRARY yaml)
   set(YAMLCPP_INCLUDE_DIR ${CMAKE_SOURCE_DIR}/external_libraries/yaml-cpp/include)
diff -ruN a/testsuite/server/CMakeLists.txt b/testsuite/server/CMakeLists.txt
--- a/testsuite/server/CMakeLists.txt	2018-08-28 03:12:47.000000000 +0200
+++ b/testsuite/server/CMakeLists.txt	2018-11-28 16:24:08.449255929 +0100
@@ -5,4 +5,8 @@
 endif()
 
 add_library(boost_test STATIC boost_test.cpp)
-target_include_directories(boost_test PUBLIC ${CMAKE_SOURCE_DIR}/external_libraries/boost)
+if (Boost_FOUND)
+  target_include_directories(boost_test PUBLIC ${Boost_INCLUDE_DIRS})
+else()
+  target_include_directories(boost_test PUBLIC ${CMAKE_SOURCE_DIR}/external_libraries/boost)
+endif()

Multiple components are (additionally) explicitely linking against external_libraries/boost or are including headers from there (even if BOOST_FOUND), so I changed that (conditionally). However, now I have problems building the vendored yaml-cpp against system boost 1.68 (I guess that validates yet another bug report or maybe one has to look into updating yaml-cpp to >= 0.6 to make it drop the boost dependency altogether) and I have a general problem building 3.10.0 (with vendored external libs), which prevents me from testing against my patch (fails the same way with and without my patch though!).

@dvzrv dvzrv referenced this issue Nov 29, 2018
3 of 3 tasks complete
@dvzrv

This comment has been minimized.

Copy link
Member Author

commented Nov 29, 2018

I have now based my changes on the develop branch (as that's what I build supercollider-git in the AUR from). I'm not sure it makes sense to only hotfix this in 3.10.1 and I think my fix is potentially not complete yet.

@dvzrv dvzrv added the in progress label Jan 22, 2019
@brianlheim brianlheim closed this Mar 3, 2019
@brianlheim

This comment has been minimized.

Copy link
Member

commented Mar 3, 2019

Closed in #4185 -- thanks @dvzrv for your patience and work on that!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.