System.DllNotFoundException

[ppekrol]

Hi,

we are using ClrMD in RavenDB to get stacktraces of our process. In most cases it works properly (had some issues with .NET 4.6 but latest version resolved that problem - thanks), but on some we are getting following exception:

System.DllNotFoundException: Unable to load DLL 'dbgeng.dll': The specified procedure could not be found. (Exception from HRESULT: 0x8007007F) at 
Microsoft.Diagnostics.Runtime.Desktop.NativeMethods.DebugCreate(Guid& InterfaceId, Object& Interface) at 
Microsoft.Diagnostics.Runtime.DbgEngDataReader.CreateIDebugClient() at 
Microsoft.Diagnostics.Runtime.DbgEngDataReader..ctor(Int32 pid, AttachFlag flags, UInt32 msecTimeout) at 
Microsoft.Diagnostics.Runtime.DataTarget.AttachToProcess(Int32 pid, UInt32 msecTimeout, AttachFlag attachFlag) at 
Raven.Debug.Program.ShowStackTrace(Int32 processId, UInt32 attachTimeout, String outputPath) at 
Raven.Debug.Program.Main(String[] args) at 
Raven.Database.Server.Controllers.Admin.AdminController.DumpStacktrace(ZipArchive package) in c:\workspaces\HR\ravendb\Raven.Database\Server\Controllers\Admin\AdminController.cs:line 849

Not sure what is the reason and why it only happens on some machines.

I would appreciate your help/feedback.

This is how it works on our side:

1. We are sending a request to a WebAPI controller
2. Controller spawns new process to get the stacktraces (extracts the program from embedded resource):
   https://github.com/ayende/ravendb/blob/master/Raven.Database/Server/Controllers/Admin/AdminController.cs#L857
3. The program is called Raven.Debug and can be found here: https://github.com/ppekrol/ravendb/blob/master/Raven.Debug/Program.cs#L53

Not sure if this can help, but on my machine I have a pretty fresh (1 month) Windows 10 Professional with VS2015 installed.

[leculver]

Hmm. That's an interesting one. Does it fail 100% of the time on those machines, or does it happen intermittently?

dngeng.dll is a windows component, it should be located on every machine in "C:\Windows\System32\dbgeng.dll". For the machines this does not work on, can you check to see that this file exists and isn't corrupted? (Hard to check the latter but at least make sure the file isn't of 0 length.)

If the file exists I would suggest trying two things:

1. Build ClrMD from source yourself (maybe use the v0.8.31.0 tag) and remove the DefaultDllSearchPaths attribute: https://github.com/Microsoft/clrmd/blob/v0.8.31.0/src/Microsoft.Diagnostics.Runtime/Desktop/helpers.cs#L364 Then see if the problem still reproduces. If the issue is fixed by removing this line, please let me know. That means there's an issue with that attribute and something I need to fix.
2. If that does not fix the issue then I'm not sure what the problem is. You can try to load dbgeng.dll yourself into the process as the first thing in your main method. To do that add a PInvoke to LoadLibrary and call LoadLibrary(@"c:\windows\system32\dbgeng.dll") (you should use the system directory instead of hard coding C:\windows, but you get the idea).

Please let me know what you find. It's really tough for me to know what the problem is without being able to reproduce the issue myself, but if you have some time to do the above investigation I'll try to get this taken care of asap.

Thanks for the bug report!

[ppekrol]

Thanks for quick reply. Here are my findings:

1. It fails 100% of times when ran from RavenDB on those machines and 0% of times when run manually from CMD or console application (strange?)
2. The DLL exists in C:\Windows\System32
3. After removing the DefaultDllSearchPaths it still fails
4. LoadLibrary returns success

Maybe the problem is not that DLL cannot be found but 'The specified procedure could not be found'?

[lasherer]

It looks like a path lacking in the account running RavenDB. Is there a profile created for such account? It can also be lack of rights in RavenDB account.
Try procmon to verify dll loading errors.

Sent from my Windows Phone

---

From: Paweł Pekrólmailto:notifications@github.com
Sent: ‎10/‎27/‎2015 2:42 AM
To: Microsoft/clrmdmailto:clrmd@noreply.github.com
Subject: Re: [clrmd] System.DllNotFoundException (#29)

Thanks for quick reply. Here are my findings:

1. It fails 100% of times when ran from RavenDB on those machines and 0% of times when run manually from CMD or console application (strange?)
2. The DLL exists in C:\Windows\System32
3. After removing the DefaultDllSearchPaths it still fails
4. LoadLibrary returns success

Maybe the problem is not that DLL cannot be found but 'The specified procedure could not be found'?

---

Reply to this email directly or view it on GitHub:
#29 (comment)

[ppekrol]

I do not see anything out of order in procmon (screen: http://1drv.ms/1GFDLb1) and the account I'm using have Administrative privileges (Win 10 x64 Pro).

[ayende]

@ppekrol What happens if you try to do a File.Read on tha tpath

[ppekrol]

I've read the dbgeng.dll without any problems using Raven.Debug

using (var stream = File.OpenRead(@"C:\Windows\System32\dbgeng.dll"))
using (var reader = new StreamReader(stream))
{
    var dll = reader.ReadToEnd();
}

[leculver]

One last thing to try. Can you successfully run this program on the machines where this is failing?

https://gist.github.com/leculver/b48d6c7eb5a99f94af5a

[ppekrol]

Here are my findings from running your code:

I. Running from CMD

1. Compiled as x64: works
2. Compiled as x86: works

II. Spawning process from RavenDB WebAPI controller (Environment.Is64BitProcess returns true)

1. Spawning x64 compiled code: error (see below)
2. Spawning x86 compiled code: works

Error is:
"The procedure entry point SetSymLoadError could not be located in the dynamic link library C:\Windows\system32\dbgeng.dll"

line that throws:

IntPtr debugCreate = GetProcAddress(dbgeng, "DebugCreate");

[JeffCyr]

@ppekrol
It probably has nothing to do with the issue, but why are you setting LoadUserProfile = true when you spawn the sub process?

I would also be curious to see if it works when you set UseShellExecute = true.

[ppekrol]

@JeffCyr

No good reason I guess for setting LoadUserProfile to true, will remove it, thanks!

Checked with UseShellExecute = true with same results, still fails.

[leculver]

I have asked a few experts around here. They said dbgcore.dll needs to be in the folder/search path. Can you check to see if that file exists in your search path? (c:\windows\system32). I don't know why it wouldn't be though...

You could probably check by calling LoadLibrary("dbgcore.dll") in the test program and checking the return value against IntPtr.Zero.

[ppekrol]

It is there, next to dbgeng.dll. This is really strange, because when I run this program from CMD then it all works. It just doesn't when I spawn process from WebAPI controller.

LoadLibrary("dbgcore.dll") loads correctly.

Can I assume that this message

"The procedure entry point SetSymLoadError could not be located in the dynamic link library
C:\Windows\system32\dbgeng.dll"

means that the dbgeng.dll exists and is loaded correctly, but SetSymLoadError is not there (for unknown reason)?

Do you have an idea why it is not working when I compile program as x64?

[ayende]

Maybe it is a compact shim?
On Oct 29, 2015 09:00, "Paweł Pekról" notifications@github.com wrote:

It is there, next to dbgeng.dll. This is really strange, because when I
run this program from CMD then it all works. It just doesn't when I spawn
process from WebAPI controller.

LoadLibrary("dbgcore.dll") loads correctly.

Can I assume that this message

"The procedure entry point SetSymLoadError could not be located in the dynamic link library
C:\Windows\system32\dbgeng.dll"

means that the dbgeng.dll exists and is loaded correctly, but
SetSymLoadError is not there (for unknown reason)?

Do you have an idea why it is not working when I compile program as x64?

—
Reply to this email directly or view it on GitHub
#29 (comment).

[JeffCyr]

I found some references that says SetSymLoadError is located in dbghelp.dll. Maybe your WebAPI host exe in x64 has a PATH which include a wrong version of dbghelp.dll. You could validate this by spawning cmd.exe with args where dbghelp.dll in the WebAPI process.

[ayende]

On my machine, I get this error:

  The procedure entry point SetCheckUserInterruptShared could not be located in the dynamic link library C:\windows\SYSTEM32\dbgeng.dll. 

There is a similar error here VerySleepy/verysleepy#7 (comment)

With the core of the issue:

 we load the system dbgeng, which then tries to load dbghelp, but instead of loading the system one, it's loading Sleepy's version, which is too old for the system dbgeng

And that probably explains it. Are we (either ravendb or clrmd) packaging dbghelp?

This error would only occur on specific versions of debugging tools for windows, I'm guessing.

[leculver]

Haven't heard back on this one in a while, but it sounds like ayende's analysis is correct here. Please feel free to reply here if this is still an issue for you.

[idofl]

Not sure if it's the same thing, but I also encountered the dbgeng exception, also on a machine running .NET 4.6.1.
In my case, the solution was to remove the "prefer 32-bit" flag in the project's build settings. Once it ran as a 64-bit console app, it managed to load the dll.

[cklutz]

FWIW, the following code (invoked before the first code of ClrMD is called) helped me with the issue:

        [DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
        private static extern IntPtr LoadLibraryEx(string lpFileName, IntPtr hReservedNull, int dwFlags);
        private const int LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR = 0x00000100;

        private static void EnsureProperDebugEngineIsLoaded()
        {
            var sysdir = Environment.GetFolderPath(Environment.SpecialFolder.System);
            var res = LoadLibraryEx(Path.Combine(sysdir, "dbgeng.dll"), IntPtr.Zero, LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR);
            if (res == IntPtr.Zero)
                throw new Win32Exception(Marshal.GetLastWin32Error());
        }

Essentially, when called before ClrMD's DllImport of dbgeng.dll has a chance to run, it not only loads dbgeng.dll from the Windows system directory, but also force it to look for its dependencies only there.
That should prevent such dependencies to be loaded from other folders.

[ItielBeeri]

@cklutz, thank you very much! I've encountered the same situation as @ppekrol (debugger runs successfully from CMD, but fails when launched from the debuggee process). @cklutz's solution solved the problem completely.