Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenMVG crashing when used as library on macOS 10.14 #1474

Open
brejchajan opened this issue Feb 8, 2019 · 18 comments
Open

OpenMVG crashing when used as library on macOS 10.14 #1474

brejchajan opened this issue Feb 8, 2019 · 18 comments

Comments

@brejchajan
Copy link

@brejchajan brejchajan commented Feb 8, 2019

Hi, first of all, thank you very much for this wonderful project. I have been using it for quite some time in my own project and I had no difficulties until I updated to MacOS 10.14. Now when I compile the OpenMVG library, sample code & software works well, but when I link OpenMVG with my own code using CMake project, it always crashes in openMVG::sfm::Load(). That is probably due to some changes in CMake or the MacOS setup itself, but this was a headache for several weeks now and I was unable to find a solution. To be able to debug the issue better, I created a minimal cmake project. Could anybody verify whether the problem is reproducible on MacOS 10.14.2, XCode 10.1 (10B61), CMake 3.13.3?

I tried using Develop branch, master branch, and latest release, I also tried to build as static and shared libraries and the problem is always the same. I am using external Eigen 3.3.4, libtiff 4.0.3, libpng 1.4.12, libjpeg 13.0.0 (CMake did not show the version during configuration, I listed it manually using otool -L <binary>). Is anyone experiencing similar problems? I found several similar issues, but they were usually resolved by using the correct linking procedure in CMake, which I believe I use.

The debugger output:

openMVGTest(10825,0x10027a5c0) malloc: *** set a breakpoint in malloc_error_break to debug
Process 10825 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x00007fff70a8623e libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff70b3cc1c libsystem_pthread.dylib`pthread_kill + 285
    frame #2: 0x00007fff709ef1c9 libsystem_c.dylib`abort + 127
    frame #3: 0x00007fff70afe6e2 libsystem_malloc.dylib`malloc_vreport + 545
    frame #4: 0x00007fff70afe4a3 libsystem_malloc.dylib`malloc_report + 152
    frame #5: 0x000000010008955f openMVGTest`std::__1::__hash_table<std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::__unordered_map_hasher<unsigned int, std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::hash<unsigned int>, true>, std::__1::__unordered_map_equal<unsigned int, std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::equal_to<unsigned int>, true>, std::__1::allocator<std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> > > >::__rehash(unsigned long) + 63
    frame #6: 0x000000010008e3e0 openMVGTest`void cereal::load<cereal::PortableBinaryInputArchive, std::__1::unordered_map, unsigned int, std::__1::shared_ptr<openMVG::sfm::View>, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, std::__1::shared_ptr<openMVG::sfm::View> > >, std::__1::shared_ptr<openMVG::sfm::View> >(cereal::PortableBinaryInputArchive&, std::__1::unordered_map<unsigned int, std::__1::shared_ptr<openMVG::sfm::View>, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, std::__1::shared_ptr<openMVG::sfm::View> > > >&) + 1056
    frame #7: 0x0000000100009d78 openMVGTest`bool openMVG::sfm::Load_Cereal<cereal::PortableBinaryInputArchive>(openMVG::sfm::SfM_Data&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, openMVG::sfm::ESfM_Data) + 1192
    frame #8: 0x000000010000476e openMVGTest`openMVG::sfm::Load(openMVG::sfm::SfM_Data&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, openMVG::sfm::ESfM_Data) + 478
    frame #9: 0x0000000100002b09 openMVGTest`main + 393
    frame #10: 0x00007fff70946ed9 libdyld.dylib`start + 1
@pmoulon
Copy link
Member

@pmoulon pmoulon commented Feb 8, 2019

Hi @brejchajan
It could be related to a compilation flag issue or a thread (lib) issue.
I'm using OpenMVG on Mac 10.13.6 without issue on my side.

As a temporary solution, you can still work with a Docker on your Mac.

Minor comments for your CMakeLists.txt
-> https://github.com/brejchajan/OpenMVGTest/blob/master/CMakeLists.txt#L10
Eigen should come by transitivity from OpenMVG

Can you try to link only OpenMVG::openMVG_sfm first and then add the other libraries as they are missing?

Loading

@brejchajan
Copy link
Author

@brejchajan brejchajan commented Feb 10, 2019

Hi @pmoulon

many thanks for your response and comments. I removed the unneeded libraries and removed Eigen from CMakeLists so that it can come by transitivity from OpenMVG. However, nothing really helped, the problem persists.

There is, however, one thing that really bugs me. I installed a clean MacOS 10.13.6 in VirtualBox virtual machine (running on my MacOS 10.14 host). OpenMVG and my mimimal project compiles OK, but the bug persists as well! Therefore it seems, that maybe it might be a problem of CMake itself (3.14 in virtual machine, 3.13.3 on my host). May I ask what the versions of CMake and clang are you using?

Thanks a lot, Jan.

Loading

@rhiestan
Copy link
Contributor

@rhiestan rhiestan commented Feb 17, 2019

I am experiencing the same issue. This is one call stack:

 * thread #17, stop reason = signal SIGABRT
  * frame #0: 0x00007fff63c14b66 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff63ddf080 libsystem_pthread.dylib`pthread_kill + 333
    frame #2: 0x00007fff63b701ae libsystem_c.dylib`abort + 127
    frame #3: 0x00007fff63c6e822 libsystem_malloc.dylib`free + 521
    frame #4: 0x0000000101023d3c Regard3D`std::__1::__hash_table<std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::__unordered_map_hasher<unsigned int, std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::hash<unsigned int>, true>, std::__1::__unordered_map_equal<unsigned int, std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::equal_to<unsigned int>, true>, std::__1::allocator<std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> > > >::__rehash(unsigned long) + 60
    frame #5: 0x0000000101106d22 Regard3D`void cereal::load<cereal::PortableBinaryInputArchive, std::__1::unordered_map, unsigned int, std::__1::shared_ptr<openMVG::sfm::View>, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, std::__1::shared_ptr<openMVG::sfm::View> > >, std::__1::shared_ptr<openMVG::sfm::View> >(cereal::PortableBinaryInputArchive&, std::__1::unordered_map<unsigned int, std::__1::shared_ptr<openMVG::sfm::View>, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, std::__1::shared_ptr<openMVG::sfm::View> > > >&) + 930
    frame #6: 0x0000000101092f59 Regard3D`bool openMVG::sfm::Load_Cereal<cereal::PortableBinaryInputArchive>(openMVG::sfm::SfM_Data&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, openMVG::sfm::ESfM_Data) + 1289
    frame #7: 0x000000010108f57e Regard3D`openMVG::sfm::Load(openMVG::sfm::SfM_Data&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, openMVG::sfm::ESfM_Data) + 478
    frame #8: 0x000000010012f36b Regard3D`OpenMVGHelper::exportToPMVS(pDensification=0x00000001061f8e30) at OpenMVGHelper.cpp:970 [opt]

This is another:

* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x00007fff63c14b66 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff63ddf080 libsystem_pthread.dylib`pthread_kill + 333
    frame #2: 0x00007fff63b701ae libsystem_c.dylib`abort + 127
    frame #3: 0x00007fff63c6e822 libsystem_malloc.dylib`free + 521
    frame #4: 0x0000000101023d3c Regard3D`std::__1::__hash_table<std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::__unordered_map_hasher<unsigned int, std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::hash<unsigned int>, true>, std::__1::__unordered_map_equal<unsigned int, std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::equal_to<unsigned int>, true>, std::__1::allocator<std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> > > >::__rehash(unsigned long) + 60
    frame #5: 0x000000010109295c Regard3D`std::__1::unordered_map<unsigned int, std::__1::shared_ptr<openMVG::sfm::View>, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, std::__1::shared_ptr<openMVG::sfm::View> > > >::operator[](unsigned int const&) + 476
    frame #6: 0x0000000101093178 Regard3D`bool openMVG::sfm::Load_Cereal<cereal::PortableBinaryInputArchive>(openMVG::sfm::SfM_Data&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, openMVG::sfm::ESfM_Data) + 1832
    frame #7: 0x000000010108f57e Regard3D`openMVG::sfm::Load(openMVG::sfm::SfM_Data&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, openMVG::sfm::ESfM_Data) + 478
    frame #8: 0x0000000100124d33 Regard3D`OpenMVGHelper::getBestValidatedPairs(paths=<unavailable>, matchesfilename="pictureset_0/matching_0/matches/matches.f.txt", pairCount=0) at OpenMVGHelper.cpp:352 [opt]

I tried to create a small test program (only with cereal) to reproduce the error, but failed until now. My configuration:
Mac OS 10.13.6, Clang 7.0 (from http://clang.llvm.org/), libc++ 7.0.
I doubt that this is an issue with OpenMVG, I am pretty sure it's an incompatibility between cereal and libc++ and/or Clang.

Loading

@brejchajan
Copy link
Author

@brejchajan brejchajan commented Feb 17, 2019

@rhiestan It also came on my mind that it could be some compatibility problem with cereal, however, I did not know the library until now. Could you share your small test program? I can try it on my system, maybe I will have more luck to reproduce the error.

Loading

@rhiestan
Copy link
Contributor

@rhiestan rhiestan commented Feb 17, 2019

Of course, here you go:

// cmake -G Ninja -DCEREAL_INCLUDE_DIR=.../include/openMVG_dependencies/cereal/include/ ../src

#include <cereal/archives/portable_binary.hpp>
#include <cereal/types/unordered_map.hpp>
#include <cereal/types/memory.hpp>

#include <sstream>
#include <unordered_map>
#include <memory>

struct OtherClass
{
	int i;

	template<class Archive>
	void serialize(Archive & archive)
	{
		archive( cereal::make_nvp("i", i ) );
	}
};
struct TestClass
{
	std::map<int, std::shared_ptr<OtherClass>> otherClassMap;

	template<class Archive>
	void serialize(Archive & archive)
	{
		archive( cereal::make_nvp("otherClassMap", otherClassMap ) );
	}
};

int main(int argc, char **argv)
{
	std::cout << "Libc++ version: " << _LIBCPP_VERSION << std::endl;
	std::stringstream ss;

	{
		cereal::PortableBinaryOutputArchive oarchive(ss);
		TestClass tc;
		tc.otherClassMap[1] = std::shared_ptr<OtherClass>(new OtherClass);
		tc.otherClassMap[2] = std::shared_ptr<OtherClass>(new OtherClass);
		oarchive(cereal::make_nvp("views", tc));
	}

	{
		cereal::PortableBinaryInputArchive iarchive(ss);
		TestClass tc;
		iarchive(cereal::make_nvp("views", tc));
	}

	return 0;
}

and the CMakeLists.txt:

cmake_minimum_required(VERSION 3.2)
project(cereal_test CXX)


ADD_EXECUTABLE(cereal_test cereal_test.cpp)

TARGET_LINK_LIBRARIES(cereal_test ${REGARD3D_LINK_LIBRARIES})
TARGET_INCLUDE_DIRECTORIES(cereal_test PRIVATE ${CEREAL_INCLUDE_DIR})
target_compile_features(cereal_test PUBLIC cxx_std_11)

Loading

@brejchajan
Copy link
Author

@brejchajan brejchajan commented Feb 24, 2019

I tried the test program to test the Cereal and it works fine.

Loading

@brejchajan
Copy link
Author

@brejchajan brejchajan commented Feb 24, 2019

@pmoulon @rhiestan I tried to build OpenMVG and my test application using gcc-8 from Homebrew and now it works. Do you have an idea what could be the reason? Could there be some compatibility issue with cereal and clang? Or maybe compatibility with libc++ (I believe gcc uses libstdc++).

Loading

@rhiestan
Copy link
Contributor

@rhiestan rhiestan commented Mar 3, 2019

Looking at the call stack, I would guess it's a incompatibility of cereal with libc++. I don't think it's a multithreading issue, as at least in Regard3D (I also think it is the case in OpenMVG) serializations do not occur multithreaded.
The difficult thing, however, is to create a small test program to reproduce the error.

Loading

@brejchajan
Copy link
Author

@brejchajan brejchajan commented Mar 9, 2019

@pmoulon, @rhiestan I had some time to dig in and I believe I found the problem. For MacOS clang, openMVG defines at the root CMakeLists.txt to use unordered map: register_definitions(-DOPENMVG_STD_UNORDERED_MAP). It seems, that this is NOT propagated using the CMake transitivity, so this is not defined in third party project. When I added this definition to my CMakeLists.txt, the code started working as expected. See my CMakeLists.txt in my minimal test project.

Can this information be added into some readme? Or do you prefer to fix it somehow using CMake so that users don’t need to define this themselves? I can contribute a fix, but I need some guidance how it should be done.

Loading

@rhiestan
Copy link
Contributor

@rhiestan rhiestan commented Mar 10, 2019

Thank you very much for the investigation! I can confirm that after adding -DOPENMVG_STD_UNORDERED_MAP I did not experience those crashes anymore!
I am not familiar with OpenMVG CMakeListst.txt structure, but I suggest adding something like
target_compile_definitions(... PUBLIC ...) in the OpenMVG CMakeLists.txt.

Loading

@pmoulon
Copy link
Member

@pmoulon pmoulon commented Jan 4, 2020

@brejchajan I'm willing to help fix this.
I think adding this

if (CMAKE_CXX_COMPILER_ID MATCHES "Clang")
target_compile_features(openMVG_sfm INTERFACE OPENMVG_STD_UNORDERED_MAP)
endif()

to openMVG/src/openMVG/sfm/CMakeLists.txt should help.

Can you try?

Loading

@brejchajan
Copy link
Author

@brejchajan brejchajan commented Mar 24, 2020

@pmoulon Sorry for late response. I tried - I added the code you posted at the top of the openMVG/src/openMVG/sfm/CMakeLists.txt, did a clean build, but the issue unfortunately persists.

Loading

@pmoulon
Copy link
Member

@pmoulon pmoulon commented Mar 24, 2020

I never got issues on MacOs Mojave or Catalina.
It could be a special config on your machine since it is working for GCC and did not work for CLANG.

We can try to disable OPENMVG_STD_UNORDERED_MAP
by adding #undef OPENMVG_STD_UNORDERED_MAP on top of this file https://github.com/openMVG/openMVG/blob/develop/src/openMVG/types.hpp#L12

Loading

@brejchajan
Copy link
Author

@brejchajan brejchajan commented Mar 24, 2020

@pmoulon Sorry, my bad. I didn't notice that after adding the code you posted at the top of the CMakeLists.txt file the configuration was not completed with error: Cannot specify compile features for target "openMVG_sfm" which is not built by this project.

So I tried to add it after: add_library(openMVG_sfm ${sfm_files_header} ${sfm_files_cpp}) target_compile_features(openMVG_sfm INTERFACE ${CXX11_FEATURES}).

However, now I get error during the generation stage:

CMake Error in openMVG_Samples/multiview_robust_essential/CMakeLists.txt:
  Specified unknown feature "OPENMVG_STD_UNORDERED_MAP" for target
  "openMVG_sample_multiview_robustEssential"

Please, do you have any ideas?

Loading

@pmoulon
Copy link
Member

@pmoulon pmoulon commented Mar 24, 2020

Hello,
I would try those in order:

  • start from OpenMVG develop (no modification)
  1. Completely deactivate OPENMVG_STD_UNORDERED_MAP by adding the undef

If 1 is not working
2. We have to understand what is wrong with OPENMVG_STD_UNORDERED_MAP on your machine
I don't understand why if we define it for openMVG_sfm, the compile complains for OPENMVG_STD_UNORDERED_MAP (it means that we did not define a compilation preprocess using the following)

target_compile_features(openMVG_sfm INTERFACE OPENMVG_STD_UNORDERED_MAP)
endif()```

Loading

@brejchajan
Copy link
Author

@brejchajan brejchajan commented Mar 26, 2020

Hello,
I would try those in order:

  • start from OpenMVG develop (no modification)
  1. Completely deactivate OPENMVG_STD_UNORDERED_MAP by adding the undef

If 1 is not working
2. We have to understand what is wrong with OPENMVG_STD_UNORDERED_MAP on your machine
I don't understand why if we define it for openMVG_sfm, the compile complains for OPENMVG_STD_UNORDERED_MAP (it means that we did not define a compilation preprocess using the following)

target_compile_features(openMVG_sfm INTERFACE OPENMVG_STD_UNORDERED_MAP)
endif()```

Hi, ok, so I tried to add the #undef OPENMVG_STD_UNORDERED_MAP in top of https://github.com/openMVG/openMVG/blob/develop/src/openMVG/types.hpp#L12, and this works. JFYI I am working now on completely clean machine with MacOS 10.15.4.

Loading

@pmoulon
Copy link
Member

@pmoulon pmoulon commented Mar 26, 2020

  • Does the native build work without modification on your new machine?
  • Does the modification was required on this new machine?

🎉 Happy hacking with OpenMVG! Glad we found a fix, but I never hit this error before on any Mac.

Loading

@brejchajan
Copy link
Author

@brejchajan brejchajan commented Mar 26, 2020

  • Does the native build work without modification on your new machine?

The native build compilation works (it always worked), but when I link OpenMVG as a library to the test project, it crashes with the error described earlier, so it does not really work.

  • Does the modification was required on this new machine?

Yes, the modification was needed on the new machine as well.

🎉 Happy hacking with OpenMVG! Glad we found a fix, but I never hit this error before on any Mac.

So you tried to link OpenMVG as an external library to a different project on a mac and it always worked as expected? That´s interesting. Did you use Clang that comes with XCode?

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants