System.OutOfMemoryException

[M00nBack]

Description

If too many assemblies are opened, a System.OutOfMemoryException error will be reported when searching, but in fact I still have a lot of free memory that is not used.

[an0ndev]

@M00nBack are you running 32-bit dnSpy instead of 64-bit? 32-bit processes on Windows are still capped at 2GB RAM*

Try using 64-bit dnSpy, you should only need the 32-bit binary for debugging 32-bit processes.

*can be 3GB-4GB depending on different conditions, see https://learn.microsoft.com/en-us/windows/win32/memory/memory-limits-for-windows-releases

[ElektroKill]

As mentioned, please use the 64-bit version of dnSpy. This allows dnSpy to use all of your available memory. Due to 32-bit addressing limitations, 32-bit dnSpy can only utilize 4GB of RAM.

If this issue still occurs when using the 64-bit version, please provide more exact steps to reproduce the problem. If you are analyzing a specific file(s), upload it here or send it to me at elektrokilldev@protonmail.com.

[Arinerron]

@ElektroKill @an0ndev The problem is that the 64 bit dnSpy doesn't allow you to run 32 bit binaries. It says, "Please use 32-bit dnSpy" at the bottom when you try to attach or open the binary.

It's very frustrating, because I have a large, 32-bit PE I want to debug. But if I try to search for literally any query, I get out of memory errors, and the entire debugger crashes. :(

It would be nice if it at least forked before searching (i.e., search in a new thread) so it doesn't kill the entire process if it runs out of memory.

[an0ndev]

@Arinerron I understand, as stated in my comment, dnSpy 64-bit can import 32-bit binaries, just not debug them. I assume there is some fundamental limitation with debugging a 32-bit process in a 64-bit host process that would require a complex kludge to work around.

[an0ndev]

@Arinerron fwiw I'm with you on wanting support for this, just trying to explain from how I imagine the developers pov :')

I have limited experience but I know WOW64 is an already kinda impressive/architecturally complex compatibility layer, so I wouldn't be surprised if the debugging API that is provided to userspace only supports debugging WOW64 processes from inside WOW64 processes, and vice versa -- so you would need to spawn a separate host process that attaches to the application, and marshal all communication with the debuggee through a channel to/from that host process. (I know atleast this is the only way to load a 32-bit library in a 64-bit process; I've had to build that architecture before, and it's not fun to maintain lol.)

This is just my best guess, based on the debugging window only showing processes of the same architecture. I could be wrong, maybe this is a limitation of dnSpy itself, I guess some internal data structure sizes could be tied directly to the register size of the host process, making it annoying to support debugging processes of the other architecture. I'm just betting against this; seeing how fully-featured and seemingly (at least decently) well thought-out the app is, I would be kinda surprised if it had such a major fatal design flaw.

Maybe you could try using WinDbg alongside dnSpy, with dnSpy only for static analysis -- I've had good experiences with WinDbg for native processes, and would be surprised if it doesn't also have good managed process support. Some seemingly relevant resources: 1 2

[ElektroKill]

@ElektroKill @an0ndev The problem is that the 64 bit dnSpy doesn't allow you to run 32 bit binaries. It says, "Please use 32-bit dnSpy" at the bottom when you try to attach or open the binary.

As @an0ndev has suggested, there is, in fact, a technical limitation for 64-bit dnSpy not being able to debug 32-bit processes. This limitation is not entirely dnSpy's fault, and it's a limitation of the debugging API provided by Microsoft.

The way that many IDEs do it (Visual Studio, JetBrains Rider) is that they spawn an entirely separate process from the IDE process, which then starts the debugging session and runs the target file. This separate process then communicates with the IDE process via some chosen form of cross-process communication so that the IDE can send some commands like "step over" to the process that is actually running the debugger. dnSpy does not do this and instead runs the debugger in the same process as dnSpy itself, hence the limitation to bitness. A similar approach is used to achieve remote debugging.

Implementing such a system would require major changes in the debugger code of dnSpy, together with coming up with a protocol for cross-process communication and all the stuff associated with it. There were no off-the-shelf debugging libraries when dnSpy was written, and it implements everything from the UI to the low-level, unmanaged, debugging API from Microsoft on its own.

It would be nice if it at least forked before searching (i.e., search in a new thread) so it doesn't kill the entire process if it runs out of memory.

dnSpy search is already multithreaded. I never experienced an OutOfMemoryException myself with the search. If you could provide a more specific way to reproduce it (files, search term, etc), I will take a look at it. OutOfMemoryException should be recoverable in .NET, so it's just a matter of catching it at the right point in the code.