DepthMap step fails in 2019.2 #593
Comments
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
@MightyBOBcnc : Could you set the verbose to "trace" and send the log with the 2019.2 release? |
|
@fabiencastan The log I included in the issue report is already set to trace. That's the full trace log for the DepthMap step in 2019.2. The last line in the log is: I have taken a look in the Windows Event Viewer and I have found that aliceVision_depthMapEstimation.exe appears to be crashing. There is 1 "Error" level events and 1 "Information" level events that appear to be connected to the crash: and Here is the associated Windows error report: The same thing happens in release 2019.1, which is to say that aliceVision_depthMapEstimation.exe appears to be crashing and in the Windows Event Viewer there is a crash that implicates ucrtbase.dll I am searching Google for errors related to ucrtbase.dll and I have found this so far: |
|
@fabiencastan another update: I created a dump of the crash. I can send it to you if you would like. It is 38MB. I ran the dump through the Debugging Tools for Windows (WinDbg). This is the analysis: What stands out to me is this: |
|
Just a check: Did you make sure your Antivirus software does not interfere? |
|
I have disabled the antivirus software (and also disabled its services in Services.msc) so I do not believe that it is interfering. The antivirus log files also do not contain any alerts for any Meshroom/Alicevision executables. |
|
Just to add more data points to the discussion... I also have a program termination "aliceVision_depthMapEstimation - AliceVision 2.1.0 has stopped working" for ver 2019.1 , and a program termination "aliceVision_depthMapEstimation - AliceVision 2.2.0 has stopped working" along with a kernel fault in the Nvidia display driver for ver 2019.2 . I have this error on two different machines, both with Nvidia GT730 graphics cards. I am using the MonsterTree sample files. The 2018.1 version will finish the mesh correctly. |
|
Meshroom works fine for me, but I also have this in my system log: P1: aliceVision_incrementalSfM.exe same with aliceVision_featureMatching.exe aliceVision_depthMapFiltering.exe This crash info is in my system log (event viewer), pointing to the Meshroom path of my 2019.1 installation. There are app crash reports for today even tough I did not start MR... For aliceVision_depthMapEstimation.exe I found this entry from an actual MR 2019.2 crash two days ago: Event name: RADAR_PRE_LEAK_64 with no other useful information. |
|
I forgot to add.. |
|
I run into problems with meshroom … if i save the project in my (user) documents Folder ... but if i save the project on my desktop or on any other drive or folder it works fine. |
|
@natowi … no I use a "mklink" for my user documents folder!
=
(C:\Users\Name\Documents) is linked to (D:\Documents).
Sorry for confusing everybody! |
|
This sounds specific to symbolic link junctions. Maybe reopen #599. |
|
Update: The aliceVision_depthMapEstimation.exe crash occurs when downscale=1. I changed to downscale=2 and it completed the task but this is a severe hit to quality of the mesh that can be extracted in the subsequent meshing steps. By comparison, release 2018.1 does not crash with downscale=1. |
|
Can confirm the windows event log error; for some projects downscale is > 1 works, for others we had to increase it to 4. <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>100</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2019-08-26T04:15:39.307564400Z" />
<EventRecordID>70692</EventRecordID>
<Channel>Application</Channel>
<Computer>DESKTOP-TR6MJER</Computer>
<Security />
</System>
<EventData>
<Data>aliceVision_depthMapEstimation.exe</Data>
<Data>2.0.0.0</Data>
<Data>5d4c28ec</Data>
<Data>ucrtbase.dll</Data>
<Data>10.0.17763.404</Data>
<Data>490b0aeb</Data>
<Data>c0000409</Data>
<Data>000000000006e91e</Data>
<Data>410</Data>
<Data>01d55bc4a031309d</Data>
<Data>D:\Mijn Downloads\Meshroom-2019.2.0-win64\aliceVision\bin\aliceVision_depthMapEstimation.exe</Data>
<Data>C:\WINDOWS\System32\ucrtbase.dll</Data>
<Data>63147727-4525-4377-a9ce-c4192c6823c1</Data>
<Data />
<Data />
</EventData>
</Event> |
This depends on the available memory. 8GB is the bare minimum for ~30-50 images (depending on resolution). 16gb are recommended for small scale reconstructions, 32gb+ for 100+ images. If you don´t have enough memory, you need to increase the downscale factor accordingly. |
|
I don't believe that system RAM is the bottleneck here. I have 32GB of RAM and aliceVision_depthMapEstimation.exe 2019.2 crashes on datasets of even only 40-50 images. I've tested the Buddha and Monstree datasets and they both crash when downscale=1. Buddha crashes on the 19th chunk. Monstree crashes immediately on the 1st chunk. 2018.1 does not crash on any dataset that I throw at it including datasets that are 3x the size of Buddha and Monstree. |
|
Same problem here, same photos in 2018.1 worked fine... updated to 2019.2 depthMap crashes. I tried all the suggested fixes but the only increasing downscale seems to do something. Doesn't seem a RAM problem since my ram was always at 50%/max 70% all the time with all downscales factors; including the fact that with 2018.1 no problems occurred. |
|
I need to reproduce it to be able to fix it. But I never managed to reproduce a crash with the 2019.2 and I have tested lots of data sets. |
|
@fabiencastan You can download my data set (~912MB). However as I mentioned previously the Monstree data set and the Buddha data set also crash. Curiously, though, the Buddha set does not crash until chunk 19 of the DepthMap stage and I'm struggling to understand why that would be the case. All of the images in the Buddha set have the exact same dimensions so I don't understand why it may crash on any individual image. If all of the images before chunk 19 work I would intuitively think that all the images should work all the way through. But I don't know how the underlying process works so all I can do is make wild non-programmer guesses (I'm just a nerdy artist) like if something is going out of bounds with the memory allocation or the garbage collector isn't deleting something. I'm trying to uncover a pattern in the crashes but so far I don't see one. At first I thought it might be crashing if the images are larger than a certain number of megapixels but as we can see below the Buddha set crashes despite having a smaller number of megapixels than when my own data set has downscale=2 enabled, which does not crash. I did some more googling about the specific errors involved (0xc0000409 STATUS_STACK_BUFFER_OVERRUN and 0x7 FAST_FAIL_FATAL_APP_EXIT) and came across these articles here and here. I don't know if that's helpful in any way.
I also have the crashdump .dmp files available to share if those would be of any help. |
|
Small update: I installed Linux Mint 19.2 on the same laptop where I was having the DepthMap crashes and did not experience a crash when using the linux version of Meshroom 2019.2.0. So it looks like the crash is likely exclusive to the Windows binaries. |
|
@MightyBOBcnc Thanks for the feedback. |
|
same issue for me , windows 10, crash at depthmap with low downscale 1 or 2 (needs to set at downscale 4 to avoid crash) with 2019 relases. 2018 version works ok with downscale1... |
|
same issue for me , windows 10 (32 Gb), K3100 (4Gb) crash at depthmap with low downscale 1 or 2 (needs to set at downscale 4 to avoid crash) with 2019 relases. Factory machine works well Xenon W-2123 (16 Gb), Driver 388.26, Quadro P620 (2 Gb) |
|
Same issue for me aswell Problem Machine (Laptop): Working Machine: No Hints for an out of memory case. RAM and VRAM don't go over 60% |
|
Same Issue for me here Maybe 2018.1 used cpu to calculate depthmap so 64GB Ram was used? Can we set 2019.2 to use CPU instead of GPU aswell for testing purpose? |
|
GPU has always been used to calculate depth maps, including in 2018.1. Something in the behavior changed between 2018.1 and 2019.1. Maybe related to AliceVision commits 671, 616, 642. |
|
Some comments here sound very simmilar to a problem i had with the DepthMapNode (and the low performance of my Quadro K3000M). I highly recommend to raise the "TdrDelay" value to several seconds (maybe even 1 or 2 minutes) while trying to find the root cause of this bug to eliminate the possible interference with the TDR feature! |
|
Oh gosh I had never considered that the Windows TDR may be interfering with Meshroom. I've heard of TDR before because Substance Painter explicitly calls it out as part of their recommended program setup and documentation. They have a good tutorial for changing it if anyone wants to give it a try: https://docs.substance3d.com/spdoc/gpu-drivers-crash-with-long-computations-128745489.html |
|
I never heard of the TDR feature before i stumbled across bug report for Battlefield 5 lest week. I had problems with the DepthMap node since i started using MeshRoom in 2018. I couldn't go lower than downscale 4 and even that crashed sometimes. I blamed my poor K3000M and decided to wait using Meshroom for serious work until a buy a beefier machine. After raising the TdrDelay to 20sec a had no problems with downscale 2. Maybe this should be mentioned in the MeshRoom FAQ? |
|
Came here to figure out same issue. I did a side by side test of Monstree dataset in 2019.2.0 with all defaults I could not get past chunk 0 of DepthMap with Downscale=2, worked ok with Downscale=4. Under same conditions, 2018.1.0 would work with Downscale=2. Interestingly I also have a GeForce GT 730 like another poster above and 16GB system memory Core i7-7700. Windows 10 Set registry values for TdrDelay and 2019.2 passes the first chunk now. Giving the registry settings here for others. Just copy paste into reg file and merge. Reboot required. Last few lines of log with is was 'crashing', it really just stops. |
|
What is this ? I have a laptop with 8GB RAM. It's happen in the Depthmap. |
|
and @davezap I don't understand where we have to do this. |
|
@Theaudorant Your error log suggest that you do not have a nvidia GPU or your drivers are not up do date. The solution by davezap is for the case that depthmap fails during computation. |
|
And if I don't a GPU, can I use Meshroom ? |
|
ok thank you |
|
but there is nothing |
|
@Theaudorant The link works. Please navigate to https://github.com/alicevision/meshroom/wiki You can navigate to the page in the right hand side menu. |
|
ok thank you I found |
|
I've never worked with Meshroom before and just downloaded version 2019.2.0 from the official website. I was watching this tutorial while letting it run with 22 jpg files and when it got to about 15% of "DepthMap", it just stopped. Pressing "Start" again doesn't work, it just stops again.
My PC: Win 10, Xeon E3-1220V2, GTX 710/GTX 1050Ti (drivers from November 2019), 8gb of RAM. Edit: I'm currently testing the same images in version 2018.1 and, even though now 12 images are marked as "not reconstructed", "DepthMap" is now already at 30% (working on chunk 2). |
|
For me updating GPU drivers & increasing tdr delay solved the issue Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers] |
|
Raising TDR values prevents the process from crashing, but the calculations never really finishes? Also, no calculations should make the GPU hang totally, should they? |
|
Same issue with my NVIDIA GeForce GT 630m on Windows 10. Not the same cuda dll are used :
Also when running [EDIT] [SOLUTION] : Increase the TDR delay - it fixed the problem : works fine and texture mesh is good (tip found in Meshroom github ) - but seems slow as the DeshMap took 55min for that sample https://rcdata.capturingreality.com/testset-ga.zip |
|
I confirm this solution for Nvidia GT 710 and Windows 7. Explicit setting of the TDR delay parameter helped to solve the problem (in my case, 8 seconds was enough). Everything is great now! |
|
On my Lenovo W530 with a Nvidia Quadro K1000M card, everything stalls if I change the TDR setting to 60. The machine becomes unresponsive several seconds at the time, and then I can move the mouse for 2 seconds and it stalls again as long as the reconstruction runs - and after hours it has still not finished. If I run the same on a new Laptop with a Quadro RTX4000 it finishes in a few minutes. If the same reconstructions runs fine on the 2018 release, there must be something wrong going on in the new versions. |
|
The registry edits worked for me, thanks, and my frustration with meshroom is replaced with enjoyment in getting some textured meshes! |
|
Hey together. Try it , maybe it works 4U 2 . |
I think the issue can be considered resolved. |
and others
Many users, including myself, ran into an issue in 2019.1 where the DepthMap step would frequently fail, but would not have this issue in 2018.1. See #409
Unfortunately this continues to happen in 2019.2. I am running a set of 52 photos that works perfectly fine in 2018.1, but failed in 2019.1 and now fail 2019.2 as well. 52/52 photos have been successfully matched but the DepthMap step immediately fails with no error message.
I'm so sad that I'm stuck on 2018.1. :( Let me know if I can provide any additional information to help.
The text was updated successfully, but these errors were encountered: