Unable to create new AppDomain in EasyLoad.Loader resulting in error Code 15 #66

Open
spazzarama opened this Issue Jan 27, 2016 · 6 comments

Comments

Projects
None yet
3 participants
@spazzarama
Member

spazzarama commented Jan 27, 2016

The AppDomain.CreateDomain(...) is failing with an OutOfMemoryException within EasyLoad.Loader.Load under some circumstances. It appears to be related to stack commit sizes in the target.

If the stack commit size is > 0x3E000 (or 253952) the error occurs, this appears to be a bug with the .NET framework as outlined in this MSDN blog post
http://blogs.msdn.com/b/dsvc/archive/2012/12/30/unexpected-out-of-memory-exception-in-net-4-5-applications-that-uses-custom-stack-commit-size.aspx

A workaround for this scenario is that if an OutOfMemoryException does occur, then use the default AppDomain rather than creating a new one.

@sneusse

This comment has been minimized.

Show comment
Hide comment
@sneusse

sneusse Sep 12, 2016

Hi,

I encountered this issue and tracked it down a little bit.
It seems that inside clr.dll the stack size for internal AppDomain threads is "optimized" towards the allocation granularity provided by GetSystemInfo (kernel32).

My solution was to detour this winapi call during the creation of the new AppDomain and provide an allocation granularity that matches the stack commit size. This seems to fix the problem (at least on my machine, FW 4.6, x86 & x64).

Please find some sample code here: http://pastebin.com/23xxpyjt
Note: I used Process.NET to create the detour.

Regards

sneusse commented Sep 12, 2016

Hi,

I encountered this issue and tracked it down a little bit.
It seems that inside clr.dll the stack size for internal AppDomain threads is "optimized" towards the allocation granularity provided by GetSystemInfo (kernel32).

My solution was to detour this winapi call during the creation of the new AppDomain and provide an allocation granularity that matches the stack commit size. This seems to fix the problem (at least on my machine, FW 4.6, x86 & x64).

Please find some sample code here: http://pastebin.com/23xxpyjt
Note: I used Process.NET to create the detour.

Regards

@spazzarama

This comment has been minimized.

Show comment
Hide comment
@spazzarama

spazzarama Sep 12, 2016

Member

👍 Thanks @sneusse I guess the main issue now is to determine the correct size to use for any given process.

Member

spazzarama commented Sep 12, 2016

👍 Thanks @sneusse I guess the main issue now is to determine the correct size to use for any given process.

@sneusse

This comment has been minimized.

Show comment
Hide comment
@sneusse

sneusse Sep 12, 2016

You always could read the PE header for that (sorry just a quick writeup).
Also, this should only be necessary during the AppDomain creation, afterwards it's safe to uninstall the detour and continue as normal.

var dos = ps.Memory.Read<IMAGE_DOS_HEADER>(ps.ModuleFactory.MainModule.BaseAddress);
var ntaddr = ps.ModuleFactory.MainModule.BaseAddress + (int) dos.e_lfanew + 4;
var nt = ps.Memory.Read<IMAGE_FILE_HEADER>(ntaddr);

var optaddr = ntaddr + Marshal.SizeOf<IMAGE_FILE_HEADER>();
int stackCommitSize;
if (Kernel32.Is32BitProcess(ps.Native.Handle))
{
    var opt = ps.Memory.Read<IMAGE_OPTIONAL_HEADER32>(optaddr);
    stackCommitSize = opt.SizeOfStackCommit;
}
else
{
    var opt = ps.Memory.Read<IMAGE_OPTIONAL_HEADER64>(optaddr);
    stackCommitSize = opt.SizeOfStackCommit;
}

sneusse commented Sep 12, 2016

You always could read the PE header for that (sorry just a quick writeup).
Also, this should only be necessary during the AppDomain creation, afterwards it's safe to uninstall the detour and continue as normal.

var dos = ps.Memory.Read<IMAGE_DOS_HEADER>(ps.ModuleFactory.MainModule.BaseAddress);
var ntaddr = ps.ModuleFactory.MainModule.BaseAddress + (int) dos.e_lfanew + 4;
var nt = ps.Memory.Read<IMAGE_FILE_HEADER>(ntaddr);

var optaddr = ntaddr + Marshal.SizeOf<IMAGE_FILE_HEADER>();
int stackCommitSize;
if (Kernel32.Is32BitProcess(ps.Native.Handle))
{
    var opt = ps.Memory.Read<IMAGE_OPTIONAL_HEADER32>(optaddr);
    stackCommitSize = opt.SizeOfStackCommit;
}
else
{
    var opt = ps.Memory.Read<IMAGE_OPTIONAL_HEADER64>(optaddr);
    stackCommitSize = opt.SizeOfStackCommit;
}
@timothythompson89

This comment has been minimized.

Show comment
Hide comment
@timothythompson89

timothythompson89 Nov 21, 2016

I just logged in to say that I tried to use sneusse's method on UE4. When I tried setting allocationGranularity to my stack commit size I would still have the same out of memory exception. When I looked at the value of allocationGranularity it was 65536 (Int16.MaxSize + 1). I tried setting it to 2147483648 (Int32.MaxSize + 1) and everything worked without an issue. Just thought I'd throw that out there.

I just logged in to say that I tried to use sneusse's method on UE4. When I tried setting allocationGranularity to my stack commit size I would still have the same out of memory exception. When I looked at the value of allocationGranularity it was 65536 (Int16.MaxSize + 1). I tried setting it to 2147483648 (Int32.MaxSize + 1) and everything worked without an issue. Just thought I'd throw that out there.

@spazzarama

This comment has been minimized.

Show comment
Hide comment
@spazzarama

spazzarama Nov 22, 2016

Member

@timothythompson89 thanks for the input Timothy. I would suggest also trying some other smaller sizes as that value you used is quite large. Not sure if it has any adverse impact upon memory usage.

Member

spazzarama commented Nov 22, 2016

@timothythompson89 thanks for the input Timothy. I would suggest also trying some other smaller sizes as that value you used is quite large. Not sure if it has any adverse impact upon memory usage.

@spazzarama spazzarama modified the milestones: 2.8, 2.7 Stable Mar 2, 2017

@spazzarama

This comment has been minimized.

Show comment
Hide comment
@spazzarama

spazzarama Mar 2, 2017

Member

Keeping this here as an improvement for 2.8

Member

spazzarama commented Mar 2, 2017

Keeping this here as an improvement for 2.8

@spazzarama spazzarama reopened this Mar 2, 2017

@spazzarama spazzarama added enhancement and removed bug labels Mar 2, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment