Renderdoc closes client application before main function without generating an error

[simonvanbernem]

Description

When I try to start my application using the Launch Application dialog in the Renderdoc UI, a console shows up and closes after a brief moment. After that my application is nowhere to be seen (terminated before entering main), and no capturing session is active. Using the Renderdoc v1.7 release, the log contains no information about an error (log v1_7.log). Attaching via "Inject to Process" is working as expected and creates a capturing session.

To investigate this bug further, I built the current version (unstable 1.9) of the repo: Doing the exact same thing with RenderDoc running from within Visual Studio debugger yields an error (log v1_9dev no breakpoints.log):

RDOC 027624: [22:11:17]    win32_network.cpp( 474) - Debug   - Unknown error 10049
RDOC 027624: [22:11:17]    win32_network.cpp( 490) - Debug   - Failed to connect to localhost:0

After some more debugging of RenderDoc I found that a port of 0 was passed to the socket connect function (win32_network.cpp:436) which makes no sense. I found that this value propagates through LiveCapture::m_RemoteIdent from a call to RENDERDOC_ExecuteAndInject (ReplayManager.cpp:400), which returns an ExecuteResult of {status=Succeeded (0) ident=0 } (which may be an internally inconsistent application state?). While chasing this value, I stepped through Process::InjectIntoProcess and by doing so, RenderDoc logged different error messages (log v1_9dev with breakpoints.log):

RDOC 025752: [21:50:14]    win32_process.cpp( 514) - Log     - Running process B:\spaced\build\spaced.exe
RDOC 025752: [21:54:14]    win32_process.cpp( 597) - Log     - Injecting renderdoc into process 26856
RDOC 025752: [21:56:02]    win32_process.cpp( 310) - Warning - CreateToolhelp32Snapshot(26856) -> 0x0000012b
RDOC 025752: [21:56:02]    win32_process.cpp( 323) - Error   - Couldn't create toolhelp dump of modules in process 26856
RDOC 025752: [21:57:53]    win32_process.cpp( 940) - Error   - Can't locate renderdoc.dll in remote PID 26856

So the timing of Process::InjectIntoProcess seems to be relevant for the error. At this point I called it quits because I don't really think I will be able to do much about possibly timing related issues in a foreign codebase.

I also compiled another dummy program (simple while loop with printf and sleep) to test that this is not an issue with my machine. This test application didn't experience any error, so it seems that this bug is triggered by my main application.

Steps to reproduce

To reproduce, simply try to start my application via the Launch Application Dialog, that I sent you via email to baldurk@baldurk.org. It should output a message on the console and close after 5 seconds, but will be instantly closed by renderdoc.

Environment

Versions are:
RenderDoc v1.7
RenderDoc v1.9 Windows 64-bit Development (efafd0b)
Windows 10 Education x64 v1903
Directx11
GTX 1070 Driver 451.48

I hope this is enough information to get you started.

[baldurk]

This crashes somewhere when entering a critical section from your code, but unfortunately without symbols I can't tell anything useful other than that:

>	ntdll.dll!RtlpWaitOnCriticalSection()	Unknown
 	ntdll.dll!RtlpEnterCriticalSectionContended()	Unknown
 	ntdll.dll!RtlEnterCriticalSection()	Unknown
 	my_problematic_application.exe!00007ff7640339e7()	Unknown
 	my_problematic_application.exe!00007ff763faecca()	Unknown
 	my_problematic_application.exe!00007ff763faec1d()	Unknown
 	my_problematic_application.exe!00007ff763fb283f()	Unknown
 	my_problematic_application.exe!00007ff763e6e6b5()	Unknown

This should be possible to reproduce yourself. When launching your program set the "Debugger Delay" to 30s or so, and then attach visual studio once the program has launched. You'll hit one debugbreak early in ntdll.dll!LdrpDoDebuggerBreak() which can be ignored/continued, and then hit this crash above.

If I had to guess I'd say that your code enters a critical section in some static initialiser that runs on thread attach, but the critical section isn't initialised until after main() is entered. RenderDoc injects a thread to insert its hooks which will run before main(), so any global initialisers that are fragile like this can break.

[simonvanbernem]

That was quick, thanks! You are right, I am able to reproduce it, it crashes in a library I use (Tracy Profiler). Thanks for pointing this out, I wasn't really aware of the debugger delay feature.

I will try to fix this/ask the library maintainer about it, but it certainly looks like this is not an issue with RenderDoc (apart from maybe providing more information about the application crash in the logs).