Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenMVG crashing when used as library on macOS 10.14 #1474

Open
brejchajan opened this issue Feb 8, 2019 · 10 comments

Comments

@brejchajan
Copy link

commented Feb 8, 2019

Hi, first of all, thank you very much for this wonderful project. I have been using it for quite some time in my own project and I had no difficulties until I updated to MacOS 10.14. Now when I compile the OpenMVG library, sample code & software works well, but when I link OpenMVG with my own code using CMake project, it always crashes in openMVG::sfm::Load(). That is probably due to some changes in CMake or the MacOS setup itself, but this was a headache for several weeks now and I was unable to find a solution. To be able to debug the issue better, I created a minimal cmake project. Could anybody verify whether the problem is reproducible on MacOS 10.14.2, XCode 10.1 (10B61), CMake 3.13.3?

I tried using Develop branch, master branch, and latest release, I also tried to build as static and shared libraries and the problem is always the same. I am using external Eigen 3.3.4, libtiff 4.0.3, libpng 1.4.12, libjpeg 13.0.0 (CMake did not show the version during configuration, I listed it manually using otool -L <binary>). Is anyone experiencing similar problems? I found several similar issues, but they were usually resolved by using the correct linking procedure in CMake, which I believe I use.

The debugger output:

openMVGTest(10825,0x10027a5c0) malloc: *** set a breakpoint in malloc_error_break to debug
Process 10825 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x00007fff70a8623e libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff70b3cc1c libsystem_pthread.dylib`pthread_kill + 285
    frame #2: 0x00007fff709ef1c9 libsystem_c.dylib`abort + 127
    frame #3: 0x00007fff70afe6e2 libsystem_malloc.dylib`malloc_vreport + 545
    frame #4: 0x00007fff70afe4a3 libsystem_malloc.dylib`malloc_report + 152
    frame #5: 0x000000010008955f openMVGTest`std::__1::__hash_table<std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::__unordered_map_hasher<unsigned int, std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::hash<unsigned int>, true>, std::__1::__unordered_map_equal<unsigned int, std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::equal_to<unsigned int>, true>, std::__1::allocator<std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> > > >::__rehash(unsigned long) + 63
    frame #6: 0x000000010008e3e0 openMVGTest`void cereal::load<cereal::PortableBinaryInputArchive, std::__1::unordered_map, unsigned int, std::__1::shared_ptr<openMVG::sfm::View>, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, std::__1::shared_ptr<openMVG::sfm::View> > >, std::__1::shared_ptr<openMVG::sfm::View> >(cereal::PortableBinaryInputArchive&, std::__1::unordered_map<unsigned int, std::__1::shared_ptr<openMVG::sfm::View>, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, std::__1::shared_ptr<openMVG::sfm::View> > > >&) + 1056
    frame #7: 0x0000000100009d78 openMVGTest`bool openMVG::sfm::Load_Cereal<cereal::PortableBinaryInputArchive>(openMVG::sfm::SfM_Data&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, openMVG::sfm::ESfM_Data) + 1192
    frame #8: 0x000000010000476e openMVGTest`openMVG::sfm::Load(openMVG::sfm::SfM_Data&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, openMVG::sfm::ESfM_Data) + 478
    frame #9: 0x0000000100002b09 openMVGTest`main + 393
    frame #10: 0x00007fff70946ed9 libdyld.dylib`start + 1
@pmoulon

This comment has been minimized.

Copy link
Member

commented Feb 8, 2019

Hi @brejchajan
It could be related to a compilation flag issue or a thread (lib) issue.
I'm using OpenMVG on Mac 10.13.6 without issue on my side.

As a temporary solution, you can still work with a Docker on your Mac.

Minor comments for your CMakeLists.txt
-> https://github.com/brejchajan/OpenMVGTest/blob/master/CMakeLists.txt#L10
Eigen should come by transitivity from OpenMVG

Can you try to link only OpenMVG::openMVG_sfm first and then add the other libraries as they are missing?

@brejchajan

This comment has been minimized.

Copy link
Author

commented Feb 10, 2019

Hi @pmoulon

many thanks for your response and comments. I removed the unneeded libraries and removed Eigen from CMakeLists so that it can come by transitivity from OpenMVG. However, nothing really helped, the problem persists.

There is, however, one thing that really bugs me. I installed a clean MacOS 10.13.6 in VirtualBox virtual machine (running on my MacOS 10.14 host). OpenMVG and my mimimal project compiles OK, but the bug persists as well! Therefore it seems, that maybe it might be a problem of CMake itself (3.14 in virtual machine, 3.13.3 on my host). May I ask what the versions of CMake and clang are you using?

Thanks a lot, Jan.

@rhiestan

This comment has been minimized.

Copy link
Contributor

commented Feb 17, 2019

I am experiencing the same issue. This is one call stack:

 * thread #17, stop reason = signal SIGABRT
  * frame #0: 0x00007fff63c14b66 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff63ddf080 libsystem_pthread.dylib`pthread_kill + 333
    frame #2: 0x00007fff63b701ae libsystem_c.dylib`abort + 127
    frame #3: 0x00007fff63c6e822 libsystem_malloc.dylib`free + 521
    frame #4: 0x0000000101023d3c Regard3D`std::__1::__hash_table<std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::__unordered_map_hasher<unsigned int, std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::hash<unsigned int>, true>, std::__1::__unordered_map_equal<unsigned int, std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::equal_to<unsigned int>, true>, std::__1::allocator<std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> > > >::__rehash(unsigned long) + 60
    frame #5: 0x0000000101106d22 Regard3D`void cereal::load<cereal::PortableBinaryInputArchive, std::__1::unordered_map, unsigned int, std::__1::shared_ptr<openMVG::sfm::View>, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, std::__1::shared_ptr<openMVG::sfm::View> > >, std::__1::shared_ptr<openMVG::sfm::View> >(cereal::PortableBinaryInputArchive&, std::__1::unordered_map<unsigned int, std::__1::shared_ptr<openMVG::sfm::View>, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, std::__1::shared_ptr<openMVG::sfm::View> > > >&) + 930
    frame #6: 0x0000000101092f59 Regard3D`bool openMVG::sfm::Load_Cereal<cereal::PortableBinaryInputArchive>(openMVG::sfm::SfM_Data&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, openMVG::sfm::ESfM_Data) + 1289
    frame #7: 0x000000010108f57e Regard3D`openMVG::sfm::Load(openMVG::sfm::SfM_Data&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, openMVG::sfm::ESfM_Data) + 478
    frame #8: 0x000000010012f36b Regard3D`OpenMVGHelper::exportToPMVS(pDensification=0x00000001061f8e30) at OpenMVGHelper.cpp:970 [opt]

This is another:

* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x00007fff63c14b66 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff63ddf080 libsystem_pthread.dylib`pthread_kill + 333
    frame #2: 0x00007fff63b701ae libsystem_c.dylib`abort + 127
    frame #3: 0x00007fff63c6e822 libsystem_malloc.dylib`free + 521
    frame #4: 0x0000000101023d3c Regard3D`std::__1::__hash_table<std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::__unordered_map_hasher<unsigned int, std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::hash<unsigned int>, true>, std::__1::__unordered_map_equal<unsigned int, std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> >, std::__1::equal_to<unsigned int>, true>, std::__1::allocator<std::__1::__hash_value_type<unsigned int, std::__1::shared_ptr<openMVG::sfm::View> > > >::__rehash(unsigned long) + 60
    frame #5: 0x000000010109295c Regard3D`std::__1::unordered_map<unsigned int, std::__1::shared_ptr<openMVG::sfm::View>, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, std::__1::shared_ptr<openMVG::sfm::View> > > >::operator[](unsigned int const&) + 476
    frame #6: 0x0000000101093178 Regard3D`bool openMVG::sfm::Load_Cereal<cereal::PortableBinaryInputArchive>(openMVG::sfm::SfM_Data&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, openMVG::sfm::ESfM_Data) + 1832
    frame #7: 0x000000010108f57e Regard3D`openMVG::sfm::Load(openMVG::sfm::SfM_Data&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, openMVG::sfm::ESfM_Data) + 478
    frame #8: 0x0000000100124d33 Regard3D`OpenMVGHelper::getBestValidatedPairs(paths=<unavailable>, matchesfilename="pictureset_0/matching_0/matches/matches.f.txt", pairCount=0) at OpenMVGHelper.cpp:352 [opt]

I tried to create a small test program (only with cereal) to reproduce the error, but failed until now. My configuration:
Mac OS 10.13.6, Clang 7.0 (from http://clang.llvm.org/), libc++ 7.0.
I doubt that this is an issue with OpenMVG, I am pretty sure it's an incompatibility between cereal and libc++ and/or Clang.

@brejchajan

This comment has been minimized.

Copy link
Author

commented Feb 17, 2019

@rhiestan It also came on my mind that it could be some compatibility problem with cereal, however, I did not know the library until now. Could you share your small test program? I can try it on my system, maybe I will have more luck to reproduce the error.

@rhiestan

This comment has been minimized.

Copy link
Contributor

commented Feb 17, 2019

Of course, here you go:

// cmake -G Ninja -DCEREAL_INCLUDE_DIR=.../include/openMVG_dependencies/cereal/include/ ../src

#include <cereal/archives/portable_binary.hpp>
#include <cereal/types/unordered_map.hpp>
#include <cereal/types/memory.hpp>

#include <sstream>
#include <unordered_map>
#include <memory>

struct OtherClass
{
	int i;

	template<class Archive>
	void serialize(Archive & archive)
	{
		archive( cereal::make_nvp("i", i ) );
	}
};
struct TestClass
{
	std::map<int, std::shared_ptr<OtherClass>> otherClassMap;

	template<class Archive>
	void serialize(Archive & archive)
	{
		archive( cereal::make_nvp("otherClassMap", otherClassMap ) );
	}
};

int main(int argc, char **argv)
{
	std::cout << "Libc++ version: " << _LIBCPP_VERSION << std::endl;
	std::stringstream ss;

	{
		cereal::PortableBinaryOutputArchive oarchive(ss);
		TestClass tc;
		tc.otherClassMap[1] = std::shared_ptr<OtherClass>(new OtherClass);
		tc.otherClassMap[2] = std::shared_ptr<OtherClass>(new OtherClass);
		oarchive(cereal::make_nvp("views", tc));
	}

	{
		cereal::PortableBinaryInputArchive iarchive(ss);
		TestClass tc;
		iarchive(cereal::make_nvp("views", tc));
	}

	return 0;
}

and the CMakeLists.txt:

cmake_minimum_required(VERSION 3.2)
project(cereal_test CXX)


ADD_EXECUTABLE(cereal_test cereal_test.cpp)

TARGET_LINK_LIBRARIES(cereal_test ${REGARD3D_LINK_LIBRARIES})
TARGET_INCLUDE_DIRECTORIES(cereal_test PRIVATE ${CEREAL_INCLUDE_DIR})
target_compile_features(cereal_test PUBLIC cxx_std_11)

@brejchajan

This comment has been minimized.

Copy link
Author

commented Feb 24, 2019

I tried the test program to test the Cereal and it works fine.

@brejchajan

This comment has been minimized.

Copy link
Author

commented Feb 24, 2019

@pmoulon @rhiestan I tried to build OpenMVG and my test application using gcc-8 from Homebrew and now it works. Do you have an idea what could be the reason? Could there be some compatibility issue with cereal and clang? Or maybe compatibility with libc++ (I believe gcc uses libstdc++).

@rhiestan

This comment has been minimized.

Copy link
Contributor

commented Mar 3, 2019

Looking at the call stack, I would guess it's a incompatibility of cereal with libc++. I don't think it's a multithreading issue, as at least in Regard3D (I also think it is the case in OpenMVG) serializations do not occur multithreaded.
The difficult thing, however, is to create a small test program to reproduce the error.

@brejchajan

This comment has been minimized.

Copy link
Author

commented Mar 9, 2019

@pmoulon, @rhiestan I had some time to dig in and I believe I found the problem. For MacOS clang, openMVG defines at the root CMakeLists.txt to use unordered map: register_definitions(-DOPENMVG_STD_UNORDERED_MAP). It seems, that this is NOT propagated using the CMake transitivity, so this is not defined in third party project. When I added this definition to my CMakeLists.txt, the code started working as expected. See my CMakeLists.txt in my minimal test project.

Can this information be added into some readme? Or do you prefer to fix it somehow using CMake so that users don’t need to define this themselves? I can contribute a fix, but I need some guidance how it should be done.

@rhiestan

This comment has been minimized.

Copy link
Contributor

commented Mar 10, 2019

Thank you very much for the investigation! I can confirm that after adding -DOPENMVG_STD_UNORDERED_MAP I did not experience those crashes anymore!
I am not familiar with OpenMVG CMakeListst.txt structure, but I suggest adding something like
target_compile_definitions(... PUBLIC ...) in the OpenMVG CMakeLists.txt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.