openMVG compiled with Visual Studio 2015, 64 bits, crashes for some projects

[ShinNoNoir]

For certain (generally larger) projects, openMVG crashes on Windows. The same projects run fine under Linux.

Compiled with Visual Studio 2015, 64 bits.

Executable that crashes:
openMVG_main_IncrementalSfM -i D:\someproject\matches\sfm_data.json -m D:\someproject\matches -o D:\someproject\reconstruction_sequential

In release mode, the crash is silent and the executable will run forever using one full core, never finishing the process. In Debug mode, an assertion is violated:

---------------------------
Microsoft Visual C++ Runtime Library
---------------------------
Debug Assertion Failed!

Program: C:\WINDOWS\SYSTEM32\MSVCP140D.DLL
File: C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\xtree
Line: 262

Expression: map/set iterator not incrementable

For information on how your program can cause an assertion
failure, see the Visual C++ documentation on asserts.

(Press Retry to debug the application)

---------------------------
Abort   Retry   Ignore   
---------------------------

These are the relevant stackframes:

/* STACK FRAMES (most recent first) */
// FRAME 0: xtree
    _Myiter& operator++()
        {   // preincrement
 #if _ITERATOR_DEBUG_LEVEL == 2
        if (this->_Getcont() == 0
            || this->_Ptr == 0
            || _Mytree::_Isnil(this->_Ptr))
            {   // report error
            _DEBUG_ERROR("map/set iterator not incrementable");
            _SCL_SECURE_OUT_OF_RANGE;     // <-- break
            }

// FRAME 1: xutility
template<class _BidIt,
    class _Diff> inline
        void _Distance2(_BidIt _First, _BidIt _Last, _Diff& _Off,
            bidirectional_iterator_tag)
    {   // add to _Off distance between bidirectional iterators (redundant)
    for (; _First != _Last; ++_First)             // <-- break
        ++_Off;
    }

// FRAME 2: xutility
template<class _InIt,
    class _Diff> inline
        void _Distance(_InIt _First, _InIt _Last, _Diff& _Off)
    {   // add to _Off distance between iterators
    _Distance2(_First, _Last, _Off, _Iter_cat(_First));  // <-- break
    }

// FRAME 3: xtree
    size_type count(const key_type& _Keyval) const
        {   // count all elements that match _Keyval
        _Paircc _Ans = equal_range(_Keyval);
        size_type _Num = 0;
        _Distance(_Ans.first, _Ans.second, _Num);
        return (_Num);                        // <-- break
        }

// FRAME 4: sequential_SfM.cpp (L:1120)
          if (sfm_data_.structure.count(trackId) != 0) // <-- break

I do not know how I can provide any further information that would help track down this bug. I may not be able to provide the actual source photos due to usage rights, but I might be able to send the openMVG files if that is of any use (it does weigh in at 955 MB though).

[pmoulon]

Hi @ShinNoNoir,
Thanks for the report.
I think about a openMP/STL related error.
I never experiment crash on the sequential pipeline on Linux.
On windows, every scene I have tried worked fine.
But some user report to me the same error as you (sort of infinite loop) on the sequential pipeline on some large scene.

It would be interesting to run the process with one thread (just to know if it's related to parallelism or to STL usage).
Would you be kind to run it with a single thread?

If necessary, you can provide me your file by mail (wetransfer or other) (only the .feat, matches.X + sfm_data.json are necessary) (.desc files are not necessary).

[ShinNoNoir]

Thanks for the response. I'll try running it on a single thread (OMP_NUM_THREADS=1). I cannot send you the files right now since I'm on a connection with limited upload bandwidth, but I will when I'm on a faster connection.

[ShinNoNoir]

@pmoulon, it seems you're right. It is most likely a parallelization issue. With OMP_NUM_THREADS=1, openMVG_main_IncrementalSfM runs fine. I'll try sending the files this evening so you'll have some test data.

[ShinNoNoir]

OK, I've used WeTransfer to send the files.

[pmoulon]

Than you @ShinNoNoir I will try to have a look on it this week.

[pmoulon]

@ShinNoNoir
The problem comes from a multithread access to structure of the SfM scene.
I have tried a fix on the branch called develop_multithread_issue_589_incremental_sfm. It worked for me on your scene.

Can you:

• try to checkout this branch, build it?
• test this fix on the problematic scene?
• test some other to confirm that there is no border effect?

Looking towards your comments.

[ShinNoNoir]

Hmm, I have built that particular branch, but the process does not seem to finish. I'll try to restart the whole pipeline from scratch.

[pmoulon]

How many thread do you have on your machine?

[ShinNoNoir]

Here's my CPU info:
Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz, 3201 Mhz, 6 Core(s), 12 Logical Processor(s)

[pmoulon]

Ok, the scene was blocking at resection 4 in my case before the fix, and after the fix it was working fine.
According the fix, (be sure you build from scratch), at which step it is blocking on your side?

[ShinNoNoir]

I killed the process (that I started yesterday) this morning around this step:

-------------------------------
-- Robust Resection of view: 64
  nfa=14.8883 inliers=131/11130 precisionNormalized=0.00136279 precision=987.051 (iter=0 ,sample=328,6196,6909,)
  nfa=-35679.3 inliers=10872/11130 precisionNormalized=0.000147746 precision=325.001 (iter=0 ,sample=328,6196,6909,)
  nfa=-52644.3 inliers=10947/11130 precisionNormalized=4.39059e-06 precision=56.0257 (iter=0 ,sample=328,6196,6909,)
  nfa=-63265 inliers=11038/11130 precisionNormalized=4.39819e-07 precision=17.7322 (iter=1 ,sample=2041,2430,7529,)
  nfa=-66878.4 inliers=10976/11130 precisionNormalized=1.17695e-07 precision=9.17284 (iter=2 ,sample=8771,3738,9391,)
  nfa=-67573.1 inliers=10910/11130 precisionNormalized=6.40604e-08 precision=6.76738 (iter=6 ,sample=10627,3593,5485,)
  nfa=-67690.3 inliers=10915/11130 precisionNormalized=6.10898e-08 precision=6.60861 (iter=14 ,sample=6874,2566,6724,)
  nfa=-67809.4 inliers=10916/11130 precisionNormalized=5.69086e-08 precision=6.37844 (iter=227 ,sample=1277,2543,8008,)

-------------------------------
-- Robust Resection
-- Resection status: 1
-- #Points used for Resection: 11130
-- #Points validated by robust Resection: 10916
-- Threshold: 6.37844
-------------------------------

[pmoulon]

Have you tried to run it many times?

[e-vlad]

I have the same issue with master and develop_multithread_issue_589_incremental_sfm
in both release and debug.

BTW, why you not have compiled files for windows or mac? There are many designers that probably can use command line interface but installing and compiling is much harder? Especially if you don't write all required steps.
Also so many warnings or errors when you try compile. And this is not easy for me designer and system admin with 20 years of experience. But how about less experienced users?

[pmoulon]

@e-vlad
Thank for the test, it seems that the multithread error comes from another place. Perhaps I don't have a sufficient thread count on my windows machine.
It would be interesting if you could share my your dataset to me (I just need from the matches folder the *.feat, matches.f.bin & sfm_data.json, the *.desc are not required).

Regarding your additional questions:

• precompiled binaries (any help is welcome to enhance the project ;-) ):
  • They are existing for windows https://github.com/openMVG/openMVG/releases/download/v1.0/ReleaseV1.0.Pangio.kuhlii.WindowsBinaries_VS2015.zip
• build instruction:
  • they are provided here for three platforms https://github.com/openMVG/openMVG/blob/master/BUILD
• compilation warnings, they are dependent of your compiler, on visual studio you got more than gcc/clang, mainly for the third parties (ceres & osi clp). I mean OpenMVG core does not have so much warnings.

As OpenMVG is an opensource project so any user , regardless of his skills can help!

[e-vlad]

Oh, sorry. Now i see release.
BTW, it work correctly on 64bit windows 10.

[pmoulon]

@e-vlad, when you said "I have the same issue with master and develop_multithread_issue_589_incremental_sfm
in both release and debug." It was on which platform/compiler?

[e-vlad]

Windows 10, Visual Studio 2015 64bit.
32Gb RAM, Nvidia GTX960,.

[pmoulon]

So your comment "BTW, it work correctly on 64bit windows 10." is not true.
Do you still have an issue, or does it worked correctly (and with which code/version)?

[e-vlad]

Ah Sorry.
Compiled from master (release) and develop_multithread_issue_589_incremental_sfm branches (release and debug) is crashed on ComputeMatches.
Prebuild windows release work without problem.

Only one problem is opensource documentation. Absolutely correct but useless without understanding inlay code :) At least i found tutorial from Piere Moulton. Still like puzzle but with some mistakes i probably can move to next steps.

[ShinNoNoir]

@pmoulon

Have you tried to run it many times?

I did try a few times, but that was before my last response to this thread (July 19th). I'm currently out of office, so I won't be able to run some tests again until Monday.

I haven't tried running develop_multithread_issue_589_incremental_sfm with a debugger yet (since it takes a long, long time to complete or crash), but I could give that a go.

[pmoulon]

@e-vlad
Here there is complete doc: http://openmvg.readthedocs.io/en/latest/software/SfM/SfM/
Note that you can run python script that will run the toolchain for you: http://openmvg.readthedocs.io/en/latest/software/SfM/SfM/#openmvg-sfm-pipelines-demo
I think that @e-vlad must start another issue thread since your have an error in a different place of the pipeline.

[e-vlad]

@pmoulon thanx. I seen this link. But no tumbler switch on that this is manual.
Now, after some investigation i understand basic pipeline.
BTW, this is not the best, but good tutorial
https://groups.google.com/forum/#!topic/vsfm/VcXGvP1HuMo
But may be not so easy to create similar, because OpenMVG have more tools than OpenMVS.

I don't see any killing feature in new branch (like 10x faster computation or perfect GPU utilisation), so i probably stop with release you already compiled. Sorry.

And thanx for a great tools!

[pmoulon]

Develop branch still bring improvements compare to the actual master branch.
BTW, if you want bring your expertise and help to make the lib faster or add new functionalities, you're welcome ;-)

[rperrot]

@e-vlad

I don't see any killing feature in new branch (like 10x faster computation or perfect GPU utilisation), so i probably stop with release you already compiled. Sorry.

Are you joking ?

I think you didn't understand how open source projects are made and works. openMVG like many project is developed on spare time with limited human and financial resources. The goals is to provide a reliable library with robust tools, well written code, if it doesn't fill your requirement, developers should feel sorry but you can't talk like that to them. Your sentence is really disrespectful about users that contribute to the project with the only goal to bring to the community some time and skill without anything in return.

because OpenMVG have more tools than OpenMVS.

This is not comparable, libraries work together and don't have the same goals, without SfM tools you can't use OpenMVS. The pipeline is :

Images -> OpenMVG -> (cameras + sparse point cloud) -> OpenMVS -> Mesh

[e-vlad]

@rperrot you understand me wrong.
I know how opensource work. I just told that because I not a coder, developer branch not work for me (if I try to compile) and I don't see big difference between stable and developer branch I probably stay with pre compiled release files. Sorry if I can't help with bugs.

[ShinNoNoir]

Hmm, I started the program with a debugger this morning and it crashed in exactly the same manner as I described on July 12th.

[pmoulon]

@ShinNoNoir
Sure that you have built from scratch (to ensure it's not old object file that are used)?
I will try to submit another possible patch during the afternoon.

[pmoulon]

@ShinNoNoir
I pushed a commit. Just tried to run the thread in a different way.

Since the problem does not longer occur on my pc...(since the previous and this change), and since the problem does not happen on unix platform, I don't understand what is going on.

On my windows pc, I have 4 threads, on yours you have 12.
Since your have more threads, you have more chance to create threading conflict effects.

If the problem still occurs can you try to run in release with:

• OMP_NUM_THREADS=1 -> run fines
• OMP_NUM_THREADS=2 -> ?
• OMP_NUM_THREADS=4 -> ?
• OMP_NUM_THREADS=8 -> ?
• OMP_NUM_THREADS=12 -> crash in release & debug

and tell me the results.

[rperrot]

@pmoulon

Did you discover exactly where is the segfault ?

@e-vlad

Could you give exactly the compilation toolchain used (32 or 64bits, Debug or Release ?), what's your VS environment (Community ?, are there any service pack ?) Did you encounter the same behavior running with debugging and without it ? ( @pmoulon if it's true, maybe there's an initialized variable)

What's your processor ? ( @pmoulon is it possible that a bug in openMP could lead to a wrong atomic operation on a given processor ?)

[ShinNoNoir]

@pmoulon
I have pulled the new commit, re-run CMake, recompiled the project (Release) and re-run it on the dataset I have here. It seems it works.

Just in case, I'll re-run it again from scratch (by deleting any previous computed data) to confirm the effectiveness of the code changes.

[pmoulon]

Thanks @ShinNoNoir, redo it from scratch will be good to to assert that there is no border effect.
I think the problem is related to the visual studio compiler and some openMP thread things.
Because here I just changed the way the openMP thread are called.

[ShinNoNoir]

I just noticed it finished computing, so it seems the fix works. :)

[pmoulon]

Thank you @ShinNoNoir for your feedback.
Modification is now merged in the develop branch ad108c0