Troubles with 6.0-preview (Mogwai crash at runtime in Debug build)

[trsh]

(Info) Falcor 6.0 (commit: eb8f6916, branch: master)
(Info) Created GPU device 'NVIDIA GeForce RTX 3060' using 'Direct3D 12' API.
The specified module could not be found.

(Fatal) Mogwai crashed unexpectedly...
Failed to load plugin library from C:\Projects\Falcor\build\windows-vs2022\bin\Debug\plugins\USDImporter.dll. File is not a shared library.

Stacktrace:
 0# 0x0000002e9e3d8990 ( ?? )
 1# 0x00007ffa70ba83ba (RtlGetCurrentServiceSessionId)

[tomas-davidovic]

Can you confirm that the USDImporter.dll is there and has about 4.5MB? (Mine has 4,542,464 bytes, with another 33,648,640 bytes in the pdb file).

[trsh]

Can you confirm that the USDImporter.dll is there and has about 4.5MB? (Mine has 4,542,464 bytes, with another 33,648,640 bytes in the pdb file).

[skallweitNV]

I'm able to reproduce the issue here, but it's seems to be down to a corrupted packman cache. The way I can reproduce this is by deleting one of the depending DLLs of USDImporter.dll. I have removed usdShade.dll here and get the exact same error at runtime (the error message is unfortunate). The build process also quits (as documented in #340) as a consequence of this. Can you run WinDbg and look for where USDImporter.dll is loaded? In my case it looks something like this when manually deleting usdShade.dll:

ModLoad: 00007ffd`6b440000 00007ffd`6b8a6000   C:\projects\Falcor\build\windows-ninja-msvc\bin\Debug\plugins\USDImporter.dll
ModLoad: 00007ffd`6a860000 00007ffd`6b43b000   C:\projects\Falcor\build\windows-ninja-msvc\bin\Debug\usd.dll
ModLoad: 00007ffd`6a080000 00007ffd`6a85d000   C:\projects\Falcor\build\windows-ninja-msvc\bin\Debug\sdf.dll
ModLoad: 00007ffd`69e10000 00007ffd`6a078000   C:\projects\Falcor\build\windows-ninja-msvc\bin\Debug\usdGeom.dll
ModLoad: 00007ffd`69a00000 00007ffd`69e0b000   C:\projects\Falcor\build\windows-ninja-msvc\bin\Debug\usdImaging.dll
ModLoad: 00007ffd`69790000 00007ffd`699f1000   C:\projects\Falcor\build\windows-ninja-msvc\bin\Debug\usdSkel.dll
ModLoad: 00007ffd`69570000 00007ffd`69787000   C:\projects\Falcor\build\windows-ninja-msvc\bin\Debug\vt.dll
ModLoad: 00007ffd`ec560000 00007ffd`ec5db000   C:\projects\Falcor\build\windows-ninja-msvc\bin\Debug\ar.dll
ModLoad: 00007ffd`f7980000 00007ffd`f79a5000   C:\projects\Falcor\build\windows-ninja-msvc\bin\Debug\work.dll
ModLoad: 00007ffd`69260000 00007ffd`6956f000   C:\projects\Falcor\build\windows-ninja-msvc\bin\Debug\hd.dll
ModLoad: 00007ffd`eb9e0000 00007ffd`eba79000   C:\projects\Falcor\build\windows-ninja-msvc\bin\Debug\arch.dll
ModLoad: 00007ffd`68d60000 00007ffd`69255000   C:\projects\Falcor\build\windows-ninja-msvc\bin\Debug\tf.dll
ModLoad: 00007ffd`ee360000 00007ffd`ee3c6000   C:\projects\Falcor\build\windows-ninja-msvc\bin\Debug\usdLux.dll

(13cc.77cc): C++ EH exception - code e06d7363 (first chance)
(Fatal) Mogwai crashed unexpectedly...
Failed to load plugin library from C:\projects\Falcor\build\windows-ninja-msvc\bin\Debug\plugins\USDImporter.dll. File is not a shared library.

Stacktrace:
 0# 0x000002f81b2d0000 ( ?? )
 1# 0x000002f81b2d0000 ( ?? )

[skallweitNV]

Now, the packman cache can easily be broken as packman has no means of checking the package's integrity. It only checks the downloaded archive's checksum, but once it's extracted to a directory, there is no more production from corruption. I would suggest you remove the entire packman cache and try again. It's unfortunate that packman has that limitation, but this is out of our hands.

[trsh]

@skallweitNV how do we clean pack man cache?

[skallweitNV]

IIRC the default location is c:\packman-repo. Just remove that and rerun setup.bat to download all the dependencies again.

[trsh]

Ok I will try soon. And report back here.

[trsh]

Pulled latest, cleaned packman-repo, deleted build dir and run the bat file, builded solution and its same issues.

Build errors:

8>FalcorPython.vcxproj -> C:\Projects\Falcor\build\windows-vs2022\bin\Debug\python\falcor.cp37-win_amd64.pyd
8>INFO: Generating python stub files in 'C:/Projects/Falcor/build/windows-vs2022/bin/Debug/python'
8>The specified module could not be found.
8>
8>
8>Traceback (most recent call last):
8>  File "C:/Projects/Falcor/build_scripts/generate_stubs.py", line 377, in <module>
8>    import falcor
8>ImportError: Failed to load plugin library from C:\Projects\Falcor\build\windows-vs2022\bin\Debug\plugins\USDImporter.dll. File is not a shared library.
8>
8>Stacktrace:
8> 0# 0x000000477dfec8d0 ( ?? )
8> 1# 0x00007ffec4a083ba (RtlGetCurrentServiceSessionId)
8>
8>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: The command "setlocal
8>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: call ..\..\bin\Debug\setpath.bat & C:/Projects/Falcor/external/packman/python/python.exe C:/Projects/Falcor/build_scripts/generate_stubs.py C:/Projects/Falcor/build/windows-vs2022/bin/Debug/python
8>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: if %errorlevel% neq 0 goto :cmEnd
8>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: :cmEnd
8>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone
8>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: :cmErrorLevel
8>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: exit /b %1
8>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: :cmDone
8>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: if %errorlevel% neq 0 goto :VCEnd
8>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: :VCEnd" exited with code 1.
8>Done building project "FalcorPython.vcxproj" -- FAILED.

And when running Mogwai

(Info) Falcor 6.0 (commit: 5869ac64, branch: master)
(Info) Created GPU device 'NVIDIA GeForce RTX 3060' using 'Direct3D 12' API.
The specified module could not be found.

(Fatal) Mogwai crashed unexpectedly...
Failed to load plugin library from C:\Projects\Falcor\build\windows-vs2022\bin\Debug\plugins\USDImporter.dll. File is not a shared library.

Stacktrace:
 0# 0x0000003643d58bc0 ( ?? )
 1# 0x00007ffec4a083ba (RtlGetCurrentServiceSessionId)

[skallweitNV]

@trsh Thanks for trying this. It looks like you have the same issue still, something that USDImporter.dll depends on is not available. It could be that we depend on some default system library that is not available on your system for some reason. What version/build of Windows do you have installed?

To debug this further you could try running Mogwai with the gflags utility. It might show us what exact dependent DLL it cannot load.

Something like gflags /i Mogwai.exe +sls should do the trick.

[trsh]

@trsh Thanks for trying this. It looks like you have the same issue still, something that USDImporter.dll depends on is not available. It could be that we depend on some default system library that is not available on your system for some reason. What version/build of Windows do you have installed?

To debug this further you could try running Mogwai with the gflags utility. It might show us what exact dependent DLL it cannot load.

Something like gflags /i Mogwai.exe +sls should do the trick.

MS windows 11 Pro, 10.0.22000 Build 22000, x64.

C:\Projects\Falcor\build\windows-vs2022\bin\Debug>"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\gflags.exe" /i Mogwai.exe +sls
Current Registry Settings for Mogwai.exe executable are: 00000002
    sls - Show Loader Snaps

C:\Projects\Falcor\build\windows-vs2022\bin\Debug>Mogwai.exe
(Info) Falcor 6.0 (commit: 5869ac64, branch: master)
(Info) Created GPU device 'NVIDIA GeForce RTX 3060' using 'Direct3D 12' API.
The specified module could not be found.

(Fatal) Mogwai crashed unexpectedly...
Failed to load plugin library from C:\Projects\Falcor\build\windows-vs2022\bin\Debug\plugins\USDImporter.dll. File is not a shared library.

Stacktrace:
 0# 0x000000b950d088f0 ( ?? )
 1# 0x00007ffec4a083ba (RtlGetCurrentServiceSessionId)

[skallweitNV]

I tested locally now. You need to run gflags as you did to enable the sls feature on Mogwai.exe. Next, run Mogwai.exe in WinDbg to get the actual kernel messages. Here is what I get when deleting usdShade.dll that USDImporter.dll depends on:

4e50:4eb4 @ 02716515 - LdrpSearchPath - RETURN: Status: 0xc0000135
4e50:4eb4 @ 02716515 - LdrpProcessWork - ERROR: Unable to load DLL: "usdShade.dll", Parent Module: "C:\projects\Falcor\build\windows-ninja-msvc\bin\Debug\plugins\USDImporter.dll", Status: 0xc0000135
4e50:4eb4 @ 02716515 - LdrpLoadDllInternal - RETURN: Status: 0xc0000135
4e50:4eb4 @ 02716515 - LdrLoadDll - RETURN: Status: 0xc0000135
(Error) Error loading shared library 'C:\projects\Falcor\build\windows-ninja-msvc\bin\Debug\plugins\USDImporter.dll': The specified module could not be found.

This is like 100 lines above the end of the log ...

This clearly shows me that usdShade.dll cannot be loaded. It would be really helpful to understand what DLL it cannot load on your system.

[trsh]

This clearly shows me that usdShade.dll cannot be loaded. It would be really helpful to understand what DLL it cannot load on your system.

2eb0:83c8 @ 107215656 - LdrpProcessWork - ERROR: Unable to load DLL: "tbb.dll", Parent Module: "C:\Projects\Falcor\build\windows-vs2022\bin\Debug\plugins\USDImporter.dll", Status: 0xc0000135

[skallweitNV]

Now we are getting somewhere. Still weird because on our machines it doesn't try to load tbb.dll (which isn't available in debug builds, only tbb_debug.dll) is. So the next question is why USDImporter.dll depends on that ..

[skallweitNV]

Looks like the issues is 2 competing version of TBB. It's weird that this triggers an error on your system but not here. Can you try copying the tbb.dll and tbb_debug.dll files from external/packman/nv-usd-debug/lib over to the bin folder and then run Mogwai.exe again? Do not run Mogwai from VS as it will otherwise overwrite the dll again, just start it from the folder.

[trsh]

Seems to work like this with some msgs:

(Error) GFX Error: IDevice::getAccelerationStructurePrebuildInfo: IAccelerationStructure::BuildInputs::instanceDescs cannot be null when creating a top-level acceleration structure.
(Error) GFX Error: IRayTracingCommandEncoder::buildAccelerationStructure: IAccelerationStructure::BuildInputs::instanceDescs cannot be null when creating a top-level acceleration structure.

tbb_debug.dll was already present, but I overwrote it. tbb.dll was not.

Opened SDF scene and it now works but still looks bad. There is another issue from me on this #321. I remember having something similar with on of mine projects, I was offsetting the shadow ray origin in the direction of ray (by epsilon), but should do in direction of hit normal.

[skallweitNV]

I hope all of this is resolved with the 6.0 release.