Enable unloading of AssemblyLoadContext #18476

janvorli · 2018-06-14T23:14:28Z

This change is basically the work of @xoofx done some time ago, rebased
to the current master and with added fixes.

My contributions to it were:

Extensive testing - running all 11753 coreclr Pri1 tests loaded into
AssemblyLoadContext including their dependencies and ensuring
that they work (except the ones that use features not yet supported
for unloading, like thread local members, COM interop).
That uncovered issues listed below.
Fix few memory leaks
Enable unloading of assemblies with PInvokes
Fix virtual stub manager initialization
Fix issue with unwind info regions reporting for stub linker
Fix misplaced assert related to statics in collectible assemblies
Fix issue with SOS domain enumeration.
Fixed tests that were allocating, but not freeing GCHandle that
was preventing the tests from unloading.

I recommend viewing the commits separately.

There is still a lot of work to do to enable e.g. unloading of assemblies
with classes having thread local members and other stuff. But it seems
worth merging these changes in.

janvorli · 2018-06-14T23:14:50Z

CC: @xoofx, @sergiy-k

jkotas · 2018-06-15T04:54:48Z

tests/src/JIT/Directed/nullabletypes/Desktop/StructDefinitions.cs

@@ -236,6 +236,11 @@ static Helper()
        GCHANDLE = GCHandle.Alloc(Console.Out);
    }

+    public static void Shutdown()


It would be better for this Helper to create object with finalizer to clean this up or register unload event to clean this up, rather than changing hundreds of tests to call the Shutdown method manually.

xoofx · 2018-06-15T07:16:59Z

Thanks @janvorli for bringing this back! And very happy to see also that "Enable unloading of assemblies with PInvokes" ❤️

janvorli · 2018-06-15T13:40:21Z

@jkotas I've updated the commit that fixes the test unloadability according to your feedback.

jkotas · 2018-06-16T06:20:23Z

src/System.Private.CoreLib/src/System/Runtime/Loader/AssemblyLoadContext.cs

        [DllImport(JitHelpers.QCall, CharSet = CharSet.Unicode)]
-        private static extern void LoadFromPath(IntPtr ptrNativeAssemblyLoadContext, string ilPath, string niPath, ObjectHandleOnStack retAssembly);
+        [SuppressUnmanagedCodeSecurity]


SuppressUnmanagedCodeSecurity annotation are not necessary in CoreCLR (we have stripped them a while ago).

(Multiple places)

jkotas · 2018-06-16T06:21:14Z

src/vm/assemblyspec.cpp

-
+    UPTR lookupKey = key;
+
+#if defined(FEATURE_CORECLR)


#if defined(FEATURE_CORECLR) is not needed

jkotas · 2018-06-16T06:49:27Z

src/vm/loaderallocator.inl

 {
    LIMITED_METHOD_CONTRACT;
    _ASSERTE(m_type == LAT_Assembly);
+
+	// Link domain assembly together


Nit: Indentation

jkotas · 2018-06-17T05:50:50Z

I have written a small test so I can step through some of the code. I am having troubles to make it unload properly. E.g. if I run the following - the memory will just keep growing (I have killed it when the working set was 2GB, on Windows x64 checked):

using System;
using System.Threading.Tasks;
using System.Reflection;
using System.Runtime.Loader;

class MyLoadContext : AssemblyLoadContext
{
    public MyLoadContext() : base(isCollectible: true)
    {
    }

    protected override Assembly Load(AssemblyName assemblyName)
    {
        return null;
    }

    static void Main()
    {
        for (;;)
        {
            var alc = new MyLoadContext();
            alc.Unload();
        }
    }
}

Edit: It looks like that there is a scalability problem. It does collect when I sprinkle it with sleeps. Still, it does not look right that it is possible to create these faster than the system is able to get rid of them.

jkotas · 2018-06-17T06:01:13Z

src/System.Private.CoreLib/src/System/Runtime/Loader/AssemblyLoadContext.cs

-            return loadedAssembly;
+        private void VerifyIsAlive()
+        {
+            if (state != InternalState.Alive)


When the unload event handlers are executing, their JITing may trigger loads of more assemblies into the AssemblyLoadContext. But this block will prevent those assemblies from being loaded because the state is not Alive anymore. I think we need to allow loading of more stuff into the assembly load context even when the unload event handlers are executing.

We should have a test for this.

jkotas · 2018-06-17T06:03:12Z

src/System.Private.CoreLib/src/System/Runtime/Loader/AssemblyLoadContext.cs

+#endif
+        }
+
+        public void Unload()


I forgot what contract for this method we agreed on: Is it required to call this method to unload the stuff? Or is the ALC going to auto-unload itself too when these are no references to it?

It may be worth a comment.

masonwheeler · 2018-06-17T11:59:00Z

@jkotas

Edit: It looks like that there is a scalability problem. It does collect when I sprinkle it with sleeps. Still, it does not look right that it is possible to create these faster than the system is able to get rid of them.

This isn't particularly surprising, considering that the amount of work that has to be done to verify a context is safe to unload is greater than the amount of work that has to be done to create one, and that (I can only assume, because if not then none of this makes any sense in the first place) it's happening asynchronously on a thread that occasionally has other things to do, while this is simply spinning forever creating contexts and dumping them as quickly as possible.

This doesn't feel like something that would happen in real-world usage. We'd definitely never see it with file-based assemblies, and even a script engine that's generating assemblies dynamically wouldn't approximate this level of turnover, because there would be space in between where it's compiling and then executing its generated code.

So basically what you've shown is that it's possible for deliberately malicious code to launch a DOS attack against this system, but that's a bit redundant because the hypothetical malicious code could be even more effective by not calling Unload in the first place. That being true, I don't think this scenario is something to worry about in and of itself.

jkotas · 2018-06-17T14:42:22Z

This doesn't feel like something that would happen in real-world usage

ok

jkotas · 2018-06-17T14:47:47Z

src/vm/assemblynative.cpp

@@ -1299,7 +1304,46 @@ INT_PTR QCALLTYPE AssemblyNative::InitializeAssemblyLoadContext(INT_PTR ptrManag
    {
        // Initialize a custom Assembly Load Context
        CLRPrivBinderAssemblyLoadContext *pBindContext = NULL;
+
+        // Create a new AssemblyLoaderAllocator for an AssemblyLoadContext
+        AssemblyLoaderAllocator* loaderAllocator = new AssemblyLoaderAllocator();


We should not need to do this for the existing regular non-unloadable load context. It is unnecessary overhead for them.

jkotas · 2018-06-17T14:52:06Z

CMakeLists.txt

@@ -584,6 +584,7 @@ add_subdirectory(src/pal/prebuilt/inc)

 add_subdirectory(src/debug/debug-pal)

+add_definitions(-DFEATURE_COLLECTIBLE_ALC)


I am not sure whether we need this define. It made sense when this started and the same code was still building for desktop. It is not the case anymore.

jkotas · 2018-06-17T14:53:26Z

src/vm/assemblyspec.cpp

@@ -1560,7 +1559,12 @@ BOOL AssemblySpecBindingCache::StoreAssembly(AssemblySpec *pSpec, DomainAssembly
    UPTR key = (UPTR)pSpec->Hash();

    // On CoreCLR, we will use the BinderID as the key 
-    ICLRPrivBinder* pBinderContextForLookup = pAssembly->GetFile()->GetBindingContext();
+    ICLRPrivBinder* pBinderContextForLookup = pSpec->GetBindingContext();


Why can't we just use pAssembly->GetFile()->GetBindingContext() as before?

I really don't remember exactly why I did this but I think I had cases where the binding context from the spec was the one to use instead of the real assembly loaded (this was the original commit c98620f )

It doesn't seem unreasonable that there may be a bug here, but I don't think its related to unloadability. This change is plenty complicated without adding possibly unrelated changes to it. @janvorli could you look into if this is necessary for the scenarios which you are testing?

Reverting this change doesn't have any impact on my testing scenarios.

jkotas · 2018-06-17T15:34:05Z

There is still a lot of work to do to enable e.g. unloading of assemblies with classes having thread local members and other stuff.

And adding tests...

jkotas · 2018-06-17T15:34:22Z

LGTM modulo comments.

xoofx · 2018-06-17T19:13:00Z

Edit: It looks like that there is a scalability problem. It does collect when I sprinkle it with sleeps. Still, it does not look right that it is possible to create these faster than the system is able to get rid of them.

Yeah, but I remember the problem was also on the original PR but afair, the ALC are delayed collected via SystemDomain::ProcessDelayedUnloadDomains which is indirectly trigerred via the finalizer of thread worker (via FinalizerThread::FinalizerThreadWorker), so unlikely to happen in the pathological case you described. So I agree with @masonwheeler.

jakobbotsch · 2018-06-17T21:11:26Z

How long would those sleeps need to be? Fuzzlyn loads a lot of assemblies, and because of no app domains in .NET core, it launches a new process for every 100 assemblies to load them. With unloadable AssemblyLoadContext I assume the new processes could be avoided? I just measured and it amounts to around 45 assemblies loaded per second.

xoofx · 2018-06-18T11:59:40Z

How long would those sleeps need to be?

There is something weird I can't remember exactly... but I'm vaguely remembering that there was a code that was actually called on some IDLE event happening on the app domain... but now can't find this code anymore... I also recall to have a brute force simple app, creating a bunch of ALC and unload them directly, it was not increasing in memory (even though there was two cases, one that hasn't loaded anything and one that loaded already something... maybe I tested only one that tested with at least one assembly loaded through it)... That's difficult to recall all the details after more than 1.5 years, I should have put more notes around 😓

jkotas · 2018-06-18T17:46:16Z

@dotnet-bot test Windows_NT x64 Checked corefx_baseline
@dotnet-bot test Ubuntu x64 Checked corefx_baseline

janvorli · 2018-06-18T22:53:09Z

Looks like one of the System.Reflection.Emit.Tests failed with assert in the runtime that I was not hitting locally. I'm looking into it.

masonwheeler · 2018-06-18T22:54:36Z

@janvorli A wild WOMM appears!

janvorli · 2018-06-22T00:25:15Z

@dotnet-bot test Windows_NT x64 Checked corefx_baseline
@dotnet-bot test Ubuntu x64 Checked corefx_baseline

The assert that pLoaderAllocator is not NULL in the CLRPrivBinderAssemblyLoadContext::SetupContext is now obsolete. The same in AppDomain::LoadDomainAssemblyInternal, where the pLoaderAllocator can also be NULL for non-collectible AssemblyLoadContext.

For collectible dynamic assemblies, we were not adding the DomainAssembly to the AssemblyLoaderAllocator. After fixing that, another issue surfaced. The AssemblyLoaderAllocator for dynamic assemblies doesn't have the m_binderToRelease set and we were asserting that it is not NULL.

* Let the the default global COMDelegate::m_pShuffleThunkCache use global stub heap instead of C++ heap for stub allocation. * Add separate instance of the shuffle thunk cache into collectible AssemblyLoaderAllocator.

In one of the recent commits, I have removed changing the managed AssemblyLoadContext handle from weak to strong in CLRPrivBinderAssemblyLoadContext::PrepareForLoadContextRelease. However I have not noticed that several tests from the CoreCLR test suite have started to throw NullReferenceException during unload. The issue was that SafeBCryptAlgorithmHandle finalizer was trying to PInvoke into a native library in order to close its handle and needed to get the AssemblyLoadContext in AssemblyLoadContext.ResolveUnmanagedDll.

janvorli · 2018-08-22T18:57:30Z

@davidwrighton, as we have already discussed offline, I've looked into your concerns related to the binder cache. When an AssemblyLoadContext.Load override gets an Assembly by calling another context's LoadFromAssemblyName name, it gets cached only for this other context. And this AssemblyLoadContext is held alive by the RuntimeAssembly's m_syncRoot.

I've also reverted back the way we get the binder for lookup in the AssemblySpecBindingCache::StoreAssembly and AssemblySpecBindingCache::StoreFile based on the feedback - my testing didn't show a need for that.

janvorli · 2018-08-22T20:37:43Z

I've apparently made some mistake when removing my instrumentation logging that I have added for my personal debugging purposes, that's why the tests are failing. I'm looking into it.

The code in AssemblySpecBindingCache::Store{File,Assembly} and in AppDomain::LoadDomainAssemblyInternal to get LoaderAllocator from a ICLRPrivBinder was previously assuming that the binder was always BINDER_SPACE::Assembly at that point. However, some new recently enabled tests have discovered that this assumption is wrong and that it can be CLRPrivAssemblyWinRT too. To fix that, I've added the GetLoaderAllocator method to ICLRPrivBinder interface and removed the ICollectibleAssemblyLoadContext. The binders that don't have LoaderAllocator return E_FAIL from this method.

Running tests with assembly loading in one AssemblyLoadContext delegated to another one has discovered an issue in selecting the LoaderAllocator for allocating shuffle thunks. The correct way is to get it from the pDelMT instead of the pTargetMeth.

The original set of changes modified the pBinderContextForLookup to be taken from pSpec and only if the spec had none, it would take it from the pFile as before the change. My testing hasn't shown this as necessary, so I am reverting this back based on the PR feedback.

janvorli · 2018-08-22T22:03:44Z

Found and fixed the issue in the "Add GetLoaderAllocator to ICLRPrivBinder" commit.
@davidwrighton can you please take a look and approve the PR if you are fine with it now?

masonwheeler · 2018-08-23T01:15:14Z

Woohoo! 🎉

TorVestergaard · 2018-08-23T09:02:06Z

Awesome job! So, what does this PR going through and being merged mean for the potential future release of AssemblyLoadContext unloading, exactly?

janvorli · 2018-08-23T18:19:09Z

@TorVestergaard this is the first step towards that goal. With the merged-in state, I am able to run coreclr tests using a wrapper tool that loads a test and its dependencies into an AssemblyLoadContext and unloads it after the test finishes.
There are still some tests that don't pass in such an environment due to four limitations that the unloadability now has. This is their list in the order of priority:

There can be no thread local member or static variables in the assemblies loaded into the unloadable AssemblyLoadContext
Delegate marshaling for types within collectible assemblies is not supported.
COM Interop is not supported for collectible types
FixedAddressValueTypeAttribute is not supported for fields in collectible types
If an attempt to load assemblies with these limitations into an unloadable AssemblyLoadContext is made, an exception is thrown.

I am currently working on investigating these limitations with the ultimate goal to get rid of them.

We also need more testing and also testing more complex scenarios like multiple AssemblyLoadContexts coexisting and interacting, etc.

jkotas · 2018-08-23T18:37:50Z

We also need:

Expose the collectible option in the public API surface: https://github.com/dotnet/corefx/issues/14724
Add related APIs that make it possible for libraries to do the right thing for collectible assemblies like: https://github.com/dotnet/corefx/issues/25671 .

janvorli added the area-VM label Jun 14, 2018

janvorli self-assigned this Jun 14, 2018

janvorli requested review from jkotas and davidwrighton June 14, 2018 23:14

benaadams mentioned this pull request Jun 14, 2018

[WIP] Collectible Assemblies and AssemblyLoadContext #8677

Closed

jkotas reviewed Jun 15, 2018

View reviewed changes

janvorli force-pushed the collectible-assemblies branch from 48915c5 to 4bc3ea1 Compare June 15, 2018 13:39

jkotas reviewed Jun 17, 2018

View reviewed changes

Apollo3zehn mentioned this pull request Jun 20, 2018

Nuget plugin manager: Remaining tasks OneDAS-Group/OneDAS-Core#140

Open

20 tasks

janvorli force-pushed the collectible-assemblies branch from 0683d09 to 3153f6b Compare June 22, 2018 01:07

janvorli added 10 commits August 15, 2018 17:52

Fix unloading event invocation

dfe2f21

Remove callback to AssemblyLoadContext after unload started

6d2ea2f

Refactored unloading into single common function

c282477

Remove obsolete asserts

cc8e5e8

The assert that pLoaderAllocator is not NULL in the CLRPrivBinderAssemblyLoadContext::SetupContext is now obsolete. The same in AppDomain::LoadDomainAssemblyInternal, where the pLoaderAllocator can also be NULL for non-collectible AssemblyLoadContext.

Re-enable Unloading event for non-collectible assemblies

675daa2

Make the StubLinker::Link always use LoaderHeap

a35190a

Fix two mistakes in the previous commit

16bf23b

Fix shuffle thunk cache for unloadability

2502992

* Let the the default global COMDelegate::m_pShuffleThunkCache use global stub heap instead of C++ heap for stub allocation. * Add separate instance of the shuffle thunk cache into collectible AssemblyLoaderAllocator.

janvorli force-pushed the collectible-assemblies branch 2 times, most recently from f299886 to 44ff704 Compare August 22, 2018 18:18

janvorli added 3 commits August 23, 2018 00:01

Fix shuffle thunk loader allocator

c33efe7

Running tests with assembly loading in one AssemblyLoadContext delegated to another one has discovered an issue in selecting the LoaderAllocator for allocating shuffle thunks. The correct way is to get it from the pDelMT instead of the pTargetMeth.

janvorli force-pushed the collectible-assemblies branch from 7ded8e6 to 5ad5feb Compare August 22, 2018 22:01

davidwrighton approved these changes Aug 22, 2018

View reviewed changes

janvorli merged commit 8cd4b39 into dotnet:master Aug 23, 2018

janvorli deleted the collectible-assemblies branch August 23, 2018 00:47

filipw mentioned this pull request Sep 11, 2018

Use AssemblyLoadContext rather than the AppDomain shim dotnet-script/dotnet-script#262

Open

natemcmaster mentioned this pull request Oct 25, 2018

Make plugins unloadable natemcmaster/DotNetCorePlugins#16

Closed

sandreenko mentioned this pull request Oct 30, 2018

Delete Free of the released GCHandle in tests/src/JIT/Regression/CLR-x86-JIT/V1-M09/b13621.cs #20701

Merged

rayrfan mentioned this pull request Apr 21, 2019

Plugins rayrfan/Fanray#303

Closed

		@@ -584,6 +584,7 @@ add_subdirectory(src/pal/prebuilt/inc)

		add_subdirectory(src/debug/debug-pal)

		add_definitions(-DFEATURE_COLLECTIBLE_ALC)

Enable unloading of AssemblyLoadContext #18476

Enable unloading of AssemblyLoadContext #18476

Conversation

janvorli commented Jun 14, 2018

janvorli commented Jun 14, 2018

Choose a reason for hiding this comment

xoofx commented Jun 15, 2018

janvorli commented Jun 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkotas commented Jun 17, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

masonwheeler commented Jun 17, 2018

jkotas commented Jun 17, 2018

jkotas Jun 17, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkotas commented Jun 17, 2018

jkotas commented Jun 17, 2018

xoofx commented Jun 17, 2018

jakobbotsch commented Jun 17, 2018

xoofx commented Jun 18, 2018

jkotas commented Jun 18, 2018

janvorli commented Jun 18, 2018

masonwheeler commented Jun 18, 2018

janvorli commented Jun 22, 2018

janvorli commented Aug 22, 2018

janvorli commented Aug 22, 2018

janvorli commented Aug 22, 2018

masonwheeler commented Aug 23, 2018

TorVestergaard commented Aug 23, 2018

janvorli commented Aug 23, 2018

jkotas commented Aug 23, 2018

jkotas commented Jun 17, 2018 •

edited

Loading

jkotas Jun 17, 2018 •

edited

Loading