Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dense mapper problem #107

Closed
alexsmartens opened this issue Apr 15, 2017 · 59 comments
Closed

Dense mapper problem #107

alexsmartens opened this issue Apr 15, 2017 · 59 comments

Comments

@alexsmartens
Copy link

alexsmartens commented Apr 15, 2017 •
edited

Hey,

I have a problem with dense mapper. Here is my log:

alex@alex-Alienware-Aurora-R6:~/Downloads/libraries/colmap/build$ ./src/exe/dense_mapper \
>     --workspace_path /media/alex/...my_path../dense \
>     --workspace_format COLMAP \
>     --DenseMapperOptions.max_image_size 0 \
>     --DenseMapperOptions.patch_match_filter true \
>     --DenseMapperOptions.patch_match_geom_consistency true
Reading model...
Reading configuration...
Reading inputs...
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
*** Aborted at 1492212490 (unix time) try "date -d @1492212490" if you are using GNU date ***
PC: @     0x7fb2a9ddf428 gsignal
*** SIGABRT (@0x3e800001fbe) received by PID 8126 (TID 0x7fb2a550e700) from PID 8126; stack trace: ***
    @     0x7fb2abf55390 (unknown)
    @     0x7fb2a9ddf428 gsignal
    @     0x7fb2a9de102a abort
    @     0x7fb2aa72184d __gnu_cxx::__verbose_terminate_handler()
    @     0x7fb2aa71f6b6 (unknown)
    @     0x7fb2aa71f701 std::terminate()
    @     0x7fb2aa74ad38 (unknown)
    @     0x7fb2abf4b6ba start_thread
    @     0x7fb2a9eb082d clone
    @                0x0 (unknown)
Aborted (core dumped)

What do you think Johannes ?
@alexsmartens alexsmartens mentioned this issue Apr 15, 2017
Closed
@OlafHaag
Copy link

OlafHaag commented May 9, 2017 •
edited

On windows 10, the GUI application just terminates when attempting the dense reconstruction (also auto reconstruct) somewhere after "reading inputs" in the stereo step. When I start colmap.exe from a terminal I get this error message:

F0509 07:35:31.095772 7672 image.cc:41] Check failed: width_ == bitmap_.Width() (4327 vs. 4326)
*** Check failure stack trace: ***

@ahojnnes
Copy link
Contributor

ahojnnes commented May 9, 2017

Can you run the dense_stereo binary from the command-line? If you downloaded the pre-built binaries from the website, you can find them in the bin directory.

@alexsmartens
Copy link
Author

alexsmartens commented May 9, 2017 •
edited

  1. I also have the same problem with the windows 10 GIU dense reconstruction.

  2. The same problem during automatic reconstruction with the latest version of Colmap binaries (Ubuntu 16.04):

...
Reading model...
Reading configuration...
Reading inputs...
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
*** Aborted at 1494293862 (unix time) try "date -d @1494293862" if you are using GNU date ***
PC: @     0x7fc569523428 gsignal
*** SIGABRT (@0x3e8000055f8) received by PID 22008 (TID 0x7fc52f7fe700) from PID 22008; stack trace: ***
    @     0x7fc56e29f390 (unknown)
    @     0x7fc569523428 gsignal
    @     0x7fc56952502a abort
    @     0x7fc569e6584d __gnu_cxx::__verbose_terminate_handler()
    @     0x7fc569e636b6 (unknown)
    @     0x7fc569e63701 std::terminate()
    @     0x7fc569e8ed38 (unknown)
    @     0x7fc56e2956ba start_thread
    @     0x7fc5695f482d clone
    @                0x0 (unknown)
Aborted (core dumped)
  1. My attempt to run dense_stereo faced the following problem (Ubuntu 16.04):
$ ./src/exe/dense_stereo \
>     --workspace_path /media/alex/...my_path../dense \
>     --workspace_format COLMAP \
>     --DenseStereo.max_image_size 0 \
>     --DenseStereo.filter false \
>     --DenseStereo.geom_consistency false \
>     --DenseStereo.num_iterations 4
Reading model...
F0509 09:25:00.597965  3147 reconstruction.cc:1591] Check failed: file.is_open() 
*** Check failure stack trace: ***
    @     0x7f83c492a5cd  google::LogMessage::Fail()
    @     0x7f83c492c433  google::LogMessage::SendToLog()
    @     0x7f83c492a15b  google::LogMessage::Flush()
    @     0x7f83c492ce1e  google::LogMessageFatal::~LogMessageFatal()
    @           0x61fee6  colmap::Reconstruction::ReadCameras()
    @           0x6231ab  colmap::Reconstruction::Read()
    @           0x5ffd47  colmap::mvs::Model::ReadFromCOLMAP()
    @           0x601a0b  colmap::mvs::Model::Read()
    @           0x567448  colmap::mvs::(anonymous namespace)::ReadPatchMatchProblems()
    @           0x56ab45  colmap::mvs::PatchMatchController::Run()
    @           0x6746dc  colmap::Thread::RunFunc()
    @     0x7f83bfdbfc80  (unknown)
    @     0x7f83c41c66ba  start_thread
    @     0x7f83bf52582d  clone
    @              (nil)  (unknown)
Aborted (core dumped)

@alexsmartens
Copy link
Author

Hi,
is there any possibility this issue is solvable?

@ahojnnes
Copy link
Contributor

ahojnnes commented May 16, 2017 via email

@alexsmartens
Copy link
Author

Sounds good,
have a great time!

@LenaSailor
Copy link

Hi fellow Colmap 'activists'.

Similar to some other contributors I'm having a problem running dense_stereo in colmap under Win10. I'm using a GTX970 and have eliminated all timeout issues. Problem exists whether using the GUI or
command line.

Typical log instances here:

**F0519 21:50:29.863973 12236 image.cc:41] Check failed: width_ == bitmap_.Width() (3009 vs. 2999)
*** Check failure stack trace: ***

F0519 23:31:33.841164 6464 image.cc:41] Check failed: width_ == bitmap_.Width() (2998 vs. 2999)
*** Check failure stack trace: *****

Help would be much appreciated at your convenience (holidays are no excuse!).

@ahojnnes
Copy link
Contributor

Is it possible for you to share your dense workspace folder?

@LenaSailor
Copy link

LenaSailor commented May 21, 2017 via email

@ahojnnes
Copy link
Contributor

ahojnnes commented May 21, 2017 via email

@LenaSailor
Copy link

LenaSailor commented May 21, 2017 via email

@ahojnnes
Copy link
Contributor

ahojnnes commented May 21, 2017 via email

@LenaSailor
Copy link

LenaSailor commented May 21, 2017 via email

@ahojnnes
Copy link
Contributor

I don't know what you are exactly doing here, but it works fine for me using the following commands:

./src/exe/image_undistorter --image_path ~/data/aerial/images --input_path ~/data/aerial/sparse/0 --output_path ~/data/aerial/dense
./src/exe/dense_stereo --workspace_path ~/data/aerial/dense  --workspace_format COLMAP

I suspect you mixed up the undistorted and distorted sparse reconstruction and/or images.

@LenaSailor
Copy link

LenaSailor commented May 21, 2017 via email

@LenaSailor
Copy link

LenaSailor commented May 21, 2017 via email

@LenaSailor
Copy link

LenaSailor commented May 21, 2017 via email

@ahojnnes
Copy link
Contributor

ahojnnes commented May 22, 2017 via email

@LenaSailor
Copy link

LenaSailor commented May 22, 2017 via email

@ahojnnes
Copy link
Contributor

ahojnnes commented May 22, 2017 via email

@LenaSailor
Copy link

LenaSailor commented May 22, 2017 via email

@OlafHaag
Copy link

OlafHaag commented May 23, 2017 •
edited

I was able to reconstruct a mesh in one project by manually going through the steps involved, after setting HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\GraphicsDrivers TdrDelay to 60 (DWord, decimal). But in other projects the automated reconstruction using the GUI still crashes and terminates the GUI. I get the error:
out of memory in C:/Users/jschoenb/Development/colmap/src/.\mvs/gpu_mat.h at line 151
QObject::~QObject: Timers cannot be stopped from another thread
What to do about it?
With the manual steps I set max_image_size to 750.

Sometimes I also still get the Check failed: width_ == bitmap_.Width() (4327 vs. 4326)
*** Check failure stack trace: *** kind of crash when attempting the dense reconstruction, but it's not clear to me what's causing it.

@LenaSailor
Copy link

LenaSailor commented May 23, 2017 via email

@KimRoos
Copy link

KimRoos commented May 25, 2017

I can not get the stereo function to work on the dev och 3.0 version, but for some reason it works great on the 2.1 version. I am using Windows 10.

@ahojnnes
Copy link
Contributor

What error do you see?

@KimRoos
Copy link

KimRoos commented May 25, 2017

I am using the GUI and it it loads the next image and then it writes:

PatchMatch::Run

After that the hole program crashes after 2-3 minutes.

@ahojnnes
Copy link
Contributor

@KimRoos Sounds like you are experiencing this issue here: https://colmap.github.io/faq.html#fix-gpu-freezes-and-timeouts-during-dense-reconstruction - did you increase the timeout?

@KimRoos
Copy link

KimRoos commented May 25, 2017

I am now using 1000 for the WDDM TDR Delay, is it too small?

I started this time the program from the dos prompt and got the following error:

D:\COLMAP-dev-windows\bin>colmap
Initialization: 2.3675s
Sweep 1: 13.5357s
Sweep 2: 7.5539s
Sweep 3: 9.2318s
Sweep 4: 7.1307s
Iteration 1: 37.4539s
Sweep 1: 8.3213s
Sweep 2: 7.1348s
Sweep 3: 8.2788s
Sweep 4: 7.1922s
Iteration 2: 30.9288s
Sweep 1: 8.2902s
Sweep 2: 7.2479s
Sweep 3: 8.2074s
Sweep 4: 7.1432s
Iteration 3: 30.8904s
Sweep 1: 8.2308s
Sweep 2: 6.9007s
Sweep 3: 8.2551s
Sweep 4: 7.3889s
Iteration 4: 30.7773s
Sweep 1: 8.2543s
Sweep 2: 7.2512s
Sweep 3: 8.2099s
out of memory in C:/Users/jschoenb/Development/colmap/src/.\mvs/gpu_mat.h at line 151
QObject::~QObject: Timers cannot be stopped from another thread

@ahojnnes
Copy link
Contributor

@KimRoos The error says "out of memory". Please refer to https://colmap.github.io/faq.html#reduce-memory-usage-during-dense-reconstruction for info on how to solve this issue.

@ahojnnes
Copy link
Contributor

Sorry, but it appears there is another small bug that I'll fix over the weekend. I'll let you know once it is committed.

@ahojnnes
Copy link
Contributor

The errors should be fixed now and the memory usage even lower. In addition to the previously described changes, you can now also set DenseFusion.max_image_size, which down-samples your images, depth maps, normal maps for fusion. This works even if you computed them in higher resolution. No need to recompute them. Moreover, there is a new "medium" quality mode in the automatic reconstructor. Let me know how this works for you. If I get positive feedback, I will push a 3.1 version soon.

@alexsmartens
Copy link
Author

Sounds very good,
will test the updated version and get back to you with the feetback soon

@LenaSailor
Copy link

LenaSailor commented May 29, 2017 via email

@KimRoos
Copy link

KimRoos commented May 30, 2017

With down sampling the image from max_image_size 2000 to 1500 it works great for me, without down sampling I get the same out of memory error as earlier.

@alexsmartens
Copy link
Author

I am working with the updated version of the master brunch.
Installation again takes hours (Ubuntu 16.04) ..

Testing dense_stereo on the data obtained by the previous version of Colmap (sparse model, etc.) still gives the similar error

./src/exe/dense_stereo \
>     --workspace_path /media/alex/...my_path../dense \
>     --workspace_format COLMAP \
>     --DenseStereo.max_image_size 0 \
>     --DenseStereo.geom_consistency true
Reading workspace...
F0530 16:52:56.841083  4589 reconstruction.cc:1591] Check failed: file.is_open() /media/alex/...my_path../dense/sparse/cameras.txt
*** Check failure stack trace: ***
    @     0x7fbec3df35cd  google::LogMessage::Fail()
    @     0x7fbec3df5433  google::LogMessage::SendToLog()
    @     0x7fbec3df315b  google::LogMessage::Flush()
    @     0x7fbec3df5e1e  google::LogMessageFatal::~LogMessageFatal()
    @           0x628928  colmap::Reconstruction::ReadCameras()
    @           0x62bc1b  colmap::Reconstruction::Read()
    @           0x56e517  colmap::mvs::Model::ReadFromCOLMAP()
    @           0x57001b  colmap::mvs::Model::Read()
    @           0x569050  colmap::mvs::Workspace::Workspace()
    @           0x57672e  colmap::mvs::PatchMatchController::ReadWorkspace()
    @           0x57b74c  colmap::mvs::PatchMatchController::Run()
    @           0x67f2fc  colmap::Thread::RunFunc()
    @     0x7fbebf478c80  (unknown)
    @     0x7fbec367b6ba  start_thread
    @     0x7fbebebde82d  clone
    @              (nil)  (unknown)
Aborted (core dumped)

Any suggestions?

@ahojnnes
Copy link
Contributor

@alexsmartens If the installation takes hours, then your machine must be really slow or you are compiling on a single thread, etc. The error message is pretty clear, in that it says that the file /media/alex/...my_path../dense/sparse/cameras.txt could not be opened. Make sure that the file exists and that it has the correct access rights.

@alexsmartens
Copy link
Author

  • I seem to have a decent machine Intel 4.2 GHz, Nvidia Titan X, 16 GB RAM. I checked the Nvidia and cuda drivers:
$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  375.66  Mon May  1 15:29:16 PDT 2017
GCC version:  gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 

$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17

It looks like I have cuda enabled. How can I check if I have multithreading enabled while compiling colmap?

  • this error was pretty obvious to me, this is what I checked first that the file exists and it's access rights. Nothing helped, that's why I reported the problem here. Anything else I can try? If there's nothing else to check, I will try to run all the commands again after we clarify the multi-threading issue

@alexsmartens
Copy link
Author

Update:

  1. I installed CUDA 8.0 toolkit. The compilation still takes hours.
    Is there anything could be done to compile and run Colmap faster?

There are my CUDA samples output:

$ ./deviceQuery
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "TITAN X (Pascal)"
  CUDA Driver Version / Runtime Version          8.0 / 8.0
  CUDA Capability Major/Minor version number:    6.1
  Total amount of global memory:                 12188 MBytes (12779978752 bytes)
  (28) Multiprocessors, (128) CUDA Cores/MP:     3584 CUDA Cores
  GPU Max Clock rate:                            1531 MHz (1.53 GHz)
  Memory Clock rate:                             5005 Mhz
  Memory Bus Width:                              384-bit
  L2 Cache Size:                                 3145728 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 1 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 8.0, CUDA Runtime Version = 8.0, NumDevs = 1, Device0 = TITAN X (Pascal)
Result = PASS


$ ./bandwidthTest
[CUDA Bandwidth Test] - Starting...
Running on...

 Device 0: TITAN X (Pascal)
 Quick Mode

 Host to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)	Bandwidth(MB/s)
   33554432			6355.6

 Device to Host Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)	Bandwidth(MB/s)
   33554432			6434.7

 Device to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)	Bandwidth(MB/s)
   33554432			340479.8

Result = PASS

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
  1. I recalled all the colmap versions installation time till Oct 16, and only the previous version compiled quickly in my case

@ahojnnes
Copy link
Contributor

ahojnnes commented Jun 2, 2017

Are you compiling in release or relwithdeb mode?

@alexsmartens
Copy link
Author

Sorry, I did not quite get you. What peace of the software are you talking about?
In fact, I usually do not specify CMAKE_BUILD_TYPE.

One more answer on the question you asked before, I compile Colmap with multi threads (make -j).

What would you recommend to be the next step?

@ahojnnes
Copy link
Contributor

ahojnnes commented Jun 3, 2017

I have no idea why your build takes so long. Several hours is definitely too long... maybe you're compiling from a network disk?

@alexsmartens
Copy link
Author

Nope, I'm compiling form a local disk

@ahojnnes
Copy link
Contributor

ahojnnes commented Jun 3, 2017 via email

@alexsmartens
Copy link
Author

I report successful run of the reconstruction cycle.

My photo set: 400 images 3264 x 2448
My PC: Intel 4.2 GHz, Nvidia Titan X, 16 GB RAM

My timing:
feature_extractor - 0 min
exhaustive_matcher - 52 min
mapper - 37 min
image_undistorter - 0 min *(max_image_size 2000)
dense_stereo - 372 min
dense_fuser - 350 min
TOTAL TIME - 811 min (13.5 hr)

Does it sound like reasonable time?

@ahojnnes
Copy link
Contributor

ahojnnes commented Jun 5, 2017

Sounds reasonable and I am glad you could finally run the pipeline successfully. Fusion could probably be speeded up with more RAM. Thanks for the feedback!

@ahojnnes
Copy link
Contributor

I assume these issues are solved with the latest version, closing this now. Feel free to reopen. Thanks to everyone for the feedback.

@alexsmartens
Copy link
Author

Hi @ahojnnes,
can we reopen this issue, please

I have an analogous issue on a different data set

$ ./src/exe/dense_stereo    
 --workspace_path /media/alex/...my_path../dense     
--workspace_format COLMAP     
--DenseStereo.max_image_size 0    
--DenseStereo.geom_consistency true

Reading workspace...
Reading configuration...

...

==============================================================================
Processing view 123 / 263
==============================================================================

Reading inputs...
F0718 12:01:47.104317   915 image.cc:41] Check failed: width_ == bitmap_.Width() (1000 vs. 0) 
*** Check failure stack trace: ***
    @     0x7f3cb74c45cd  google::LogMessage::Fail()
    @     0x7f3cb74c6433  google::LogMessage::SendToLog()
    @     0x7f3cb74c415b  google::LogMessage::Flush()
    @     0x7f3cb74c6e1e  google::LogMessageFatal::~LogMessageFatal()
    @           0x56db64  colmap::mvs::Image::SetBitmap()
    @           0x57ab29  colmap::mvs::PatchMatchController::ProcessProblem()
    @           0x57c0fc  std::_Function_handler<>::_M_invoke()
    @           0x57c739  std::__future_base::_State_baseV2::_M_do_set()
    @     0x7f3cb6d53a99  __pthread_once_slow
    @           0x57d116  std::call_once<>()
    @           0x57d340  _ZNSt17_Function_handlerIFvvEZN6colmap10ThreadPool7AddTaskIMNS1_3mvs20PatchMatchControllerEFvRKNS4_10PatchMatch7OptionsEmEIPS5_RS7_RmEEESt6futureINSt9result_ofIFT_DpT0_EE4typeEEOSH_DpOSI_EUlvE_E9_M_invokeERKSt9_Any_data
    @           0x67ff22  colmap::ThreadPool::WorkerFunc()
    @     0x7f3cb2b49c80  (unknown)
    @     0x7f3cb6d4c6ba  start_thread
    @     0x7f3cb22af3dd  clone
    @              (nil)  (unknown)
Aborted (core dumped)

@ahojnnes
Copy link
Contributor

ahojnnes commented Jul 19, 2017 via email

@alexsmartens
Copy link
Author

Sorry for my late reply, I was out of city for some time.
Meanwhile, I saw that you made some updates to COLMAP just recently. So, I compiled COLMAP with the last updates and this solved my problem. Thank you

@alexsmartens
Copy link
Author

Hi @ahojnnes ,

I'm reporting that the last updates which solved my recent problem simultaneously increased computational time dramatically (from 36 to 112 min in my case). I got this time difference by running dense_stereo . Is there anything can be done about it?

@ahojnnes
Copy link
Contributor

ahojnnes commented Jul 25, 2017 via email

@alexsmartens
Copy link
Author

I have an overwhelming amount of work in the next two weeks. Can try to work it out later
So far, I can tell you that the first COLMAP version (FAST) I downloaded and compiled on 31/05/17 and the other version (SLOW) was downloaded and compiled on 24/07/17

@yonasteodros
Copy link

Hi @ahojnnes and @alexsmartens ,

I am having the same issue on the latest commit COLMAP 3.2 for images from two different cameras with similar resolution. could you please tell me what is the reason behind it ?

==============================================================================
Processing view 1 / 698

Reading inputs...
F1005 23:33:33.636186 5656 image.cc:41] Check failed: width_ == bitmap_.Width() (1919 vs. 0)
*** Check failure stack trace: ***

@ahojnnes
Copy link
Contributor

ahojnnes commented Oct 7, 2017

Are some of the undistorted images in the workspace missing or are you using a non-standard bitmap format?

@yonasteodros
Copy link

Hi @ahojnnes

Thanks for the reply, today I found out that the undistort step needs all the images to be written with there extension like "Lena.png" . if the pictures dosen't have the extension appended like this "Lena", it will fail to create the undistorted images.

@ahojnnes
Copy link
Contributor

ahojnnes commented Oct 8, 2017

@yonasteo Thanks for the feedback. This should be fixed with 4d96c9f - can you check whether it works for you and let me know? Thanks.

@yonasteodros
Copy link

@ahojnnes Thanks for the fix it works verywell !

@mattbrown11
Copy link

I also ran into this error message when one of my .photometric.bin files had size 0 because of a previous crash.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@ahojnnes @yonasteodros @OlafHaag @LenaSailor @KimRoos @mattbrown11 @alexsmartens