Returning Assembly from collectible LoadAssemblyContext via AssemblyResolve event handler

[BaHXeLiSiHg]

Greetings! Using Core 3.0 with collectible AssemblyLoadContext, in order to unload/reload dynamically loaded assemblies on demand.

I'm not sure if it's bug or a misunderstanding of AssemblyResolve event handler, let me try to explain:

I have configured cache of dynamically loading/reloading/unloading assemblies, separate collectible AssemblyLoadContext per loaded assembly. Everything works perfect, but i have encountered issues with 3-rd party library.

This 3-rd party library have serialization logic in some parts of code flow, using kind of BinarySerializer extension which is trying to access types from any of mine dynamically loaded assembly by standard Type.GetType(string typeName) method. Such behavior throwing errors because types of dynamically loaded assembly are not in AppDomain.CurrentDomain but in separate collectible AssemblyLoadContext.

To save supporting of this 3-rd party library, i have added AssemblyResolve event handler, to guide system where it should looking for assembly:

AppDomain.CurrentDomain.AssemblyResolve += AssemblyResolveEventHandler;

Where handler looks like:

private static Assembly AssemblyResolveEventHandler(object sender, ResolveEventArgs resolveEvent)
{
    var dynamicPartsManager = StaticServiceProvider.Stash.GetService<IDynamicPartsManager>();

    if (dynamicPartsManager.DynamicAssemblyLoaded)
    {
        var assemblyFriendlyName = resolveEvent.Name.Substring(
            0,
            resolveEvent.Name.IndexOf(",", StringComparison.InvariantCultureIgnoreCase)
        );

        if (DynamicAssemblyLoader.HasLoadedDomain(assemblyFriendlyName, out AssemblyLoadContext domain))
        {
            var dynamicAssembly = domain.Assemblies.FirstOrDefault(assembly => assembly.FullName == resolveEvent.Name);

            return dynamicAssembly;
        }
    }

    return null;
}

Where DynamicAssemblyLoader is my cache of dynamically loaded assemblies per AssemblyLoadContext.

When looking assembly was successfully found and returned it throws exception:

System.IO.FileLoadException: Could not load file or assembly 'MyDynamicAssembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'. Operation is not supported. (0x80131515)
File name: 'MyDynamicAssembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'
 ---> System.NotSupportedException: Resolving to a collectible assembly is not supported.
   at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, StackCrawlMarkHandle stackMark, ObjectHandleOnStack assemblyLoadContext, Boolean loadTypeFromPartialName, ObjectHandleOnStack type, ObjectHandleOnStack keepalive)
   at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext, Boolean loadTypeFromPartialName)
   at System.RuntimeType.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
   at System.Type.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase)
   ...
   3-rd party library call stack

Probably it is not possible to "just return" found Assembly or am i doing something wrong?

If anybody have any suggestions - please, share them, cause i'm out of ideas :(

Tagging subscribers to this area: @vitek-karas, @agocke, @CoffeeFlux
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Greetings! Using Core 3.0 with collectible AssemblyLoadContext, in order to unload/reload dynamically loaded assemblies on demand.

I'm not sure if it's bug or a misunderstanding of AssemblyResolve event handler, let me try to explain:

I have configured cache of dynamically loading/reloading/unloading assemblies, separate collectible AssemblyLoadContext per loaded assembly. Everything works perfect, but i have encountered issues with 3-rd party library.

This 3-rd party library have serialization logic in some parts of code flow, using kind of BinarySerializer extension which is trying to access types from any of mine dynamically loaded assembly by standard Type.GetType(string typeName) method. Such behavior throwing errors because types of dynamically loaded assembly are not in AppDomain.CurrentDomain but in separate collectible AssemblyLoadContext.

To save supporting of this 3-rd party library, i have added AssemblyResolve event handler, to guide system where it should looking for assembly:

AppDomain.CurrentDomain.AssemblyResolve += AssemblyResolveEventHandler;

Where handler looks like:

private static Assembly AssemblyResolveEventHandler(object sender, ResolveEventArgs resolveEvent)
{
    var dynamicPartsManager = StaticServiceProvider.Stash.GetService<IDynamicPartsManager>();

    if (dynamicPartsManager.DynamicAssemblyLoaded)
    {
        var assemblyFriendlyName = resolveEvent.Name.Substring(
            0,
            resolveEvent.Name.IndexOf(",", StringComparison.InvariantCultureIgnoreCase)
        );

        if (DynamicAssemblyLoader.HasLoadedDomain(assemblyFriendlyName, out AssemblyLoadContext domain))
        {
            var dynamicAssembly = domain.Assemblies.FirstOrDefault(assembly => assembly.FullName == resolveEvent.Name);

            return dynamicAssembly;
        }
    }

    return null;
}

Where DynamicAssemblyLoader is my cache of dynamically loaded assemblies per AssemblyLoadContext.

When looking assembly was successfully found and returned it throws exception:

System.IO.FileLoadException: Could not load file or assembly 'MyDynamicAssembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'. Operation is not supported. (0x80131515)
File name: 'MyDynamicAssembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'
 ---> System.NotSupportedException: Resolving to a collectible assembly is not supported.
   at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, StackCrawlMarkHandle stackMark, ObjectHandleOnStack assemblyLoadContext, Boolean loadTypeFromPartialName, ObjectHandleOnStack type, ObjectHandleOnStack keepalive)
   at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext, Boolean loadTypeFromPartialName)
   at System.RuntimeType.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
   at System.Type.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase)
   ...
   3-rd party library call stack

Probably it is not possible to "just return" found Assembly or am i doing something wrong?

If anybody have any suggestions - please, share them, cause i'm out of ideas :(

Author:	BaHXeLiSiHg
Assignees:	-
Labels:	area-AssemblyLoader-coreclr, untriaged
Milestone:	-

[vitek-karas]

The global AssemblyResolve event handler can't return collectible assemblies because the returned assembly effectively becomes part of the "global" state - and so trying to unload it would be problematic. Collectible assemblies should be "bound" to their respective ALCs in this sense and should not "leak" out.

I think a better solution to this problem will be to use contextual reflection context: https://github.com/dotnet/coreclr/blob/master/Documentation/design-docs/AssemblyLoadContext.ContextualReflection.md

Before calling the 3-rd party library, set it to the load context to which the library belongs, or on which it should operate. The assembly resolution caused by Type.GetType will then go through that load context first. So assuming you implemented its Load method, it should go through that. It is perfectly valid to return collectible assemblies from that method.

(I haven't tried it though)

Ideally you should advocate the 3-rd party library to switch over to one of the overloads of Type.GetType which takes assembly resolver func and make it so that the library propagates that through its API. That way the caller of the library can set it up whichever way it wants to assembly resolution to work.

[BaHXeLiSiHg]

Thanks for guidance, will explore ContextualReflection usage.

3-rd party library owner currently declines any ideas of changes on library BinarySerializer extension, so i have to do something from my project.

Also, i noticed that my dynamically loaded assembly is shown in System.AppDomain.CurrentDomain.GetAssemblies() list. Does it must be like that or something "pinned" assembly to AppDomain???

But even if dynamic assembly is in general assemblies list and i have removed AssemblyResolve event handler, method Type.GetType(string typeName) does not work, getting exception:

System.IO.FileNotFoundException: Could not load file or assembly 'SysUserAccountPreferences, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified. 
File name: 'SysUserAccountPreferences, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'
   at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, StackCrawlMarkHandle stackMark, ObjectHandleOnStack assemblyLoadContext, Boolean loadTypeFromPartialName, ObjectHandleOnStack type, ObjectHandleOnStack keepalive)
   at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext, Boolean loadTypeFromPartialName)
   at System.RuntimeType.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
   at System.Type.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase)
   ...
   3-rd party library call stack

Does method System.RuntimeTypeHandle.GetTypeByName trying to load requested assembly cause it was leaked or even dynamically loaded assembly should be shown in assemblies list of AppDomain.CurrentDomain??

Just trying to figure out if did something wrong. Will be grateful for any hints!

[BaHXeLiSiHg]

@vitek-karas, does using of EnterContextualReflection assumes that not only assembly loading should be performed in scope of AssemblyLoadContext contextual reflection but even every operation with this dynamic assembly, loaded for certain AssemblyLoadContext, should be done inside reflection context of AssemblyLoadContext, created every time via using statement?

[vitek-karas]

AppDomain.GetAssemblies returns all assemblies in the process - across all load contexts. That API should basically not be used - it's a left-over from .NET Framework days and in apps which use load contexts can be very confusing. There's a AssemblyLoadContext.Assemblies which can be used to list all assemblies loaded into a given load context.

General comment on AppDomain - in .NET Core (and in .NET 5+) application domains don't really exist. There's always only one app domain - there's no way to create a second one. The APIs on AppDomain are basically just historical for compatibility with .NET Framework.

To diagnose assembly load issues you can use tracing: https://docs.microsoft.com/en-us/dotnet/core/dependency-loading/collect-details. There are many reasons why an assembly fails to load and unfortunately the exception messages are not very informative. The tracing should have much more info in it though.

With the contextual reflection load context set, Type.GetType should always first go to the set load context - so you should see the request to load SysUserAccountPreferences in your custom load context Load method. That is unless the assembly is already loaded there, but there's some other mismatch (like version - but that doesn't seem to be the case here).

The EnterContextualReflection sets an async-local variable (similar to thread-local) - and is used in all APIs which otherwise determine the load context to use based on their caller - like Type.GetType. So if you're using code which relies on these APIs, then all such code should be called under the EnterContextualReflection - unlike in .NET Framework there's no magical "current domain" variable which gets propagated throughout the code. The contextual reflection context is basically a workaround for that - hopefully all code will eventually use explicit APIs to correctly handle load contexts, but in the mean time...

[BaHXeLiSiHg]

Seems like EnterContextualReflection did the trick, AssemblyLoadContext-s were successfully unloaded without memory leaks (tested only with "simple" assemblies, without 3-rd party library, which caused type exception).

Going to wrap every dynamic instance invocation in reflection scope, to test behavior of 3-rd party library, but that's a lot of work...

What do you mean by "handle in the mean time..."?

[vitek-karas]

I assume you refer to:

The contextual reflection context is basically a workaround for that - hopefully all code will eventually use explicit APIs to correctly handle load contexts, but in the mean time...

Cleaner solution overall is to not have APIs which rely on "current context" - so for example Type.GetType(string) is a bad API in that sense. Ideally one day such APIs would be removed in favor of overloads which make load contexts explicit. At which point "current context" would not be needed anymore. I don't know if we can ever get there though. So "in the mean time", we're stuck with "Current context".

Going to wrap every dynamic instance invocation in reflection scope, to test behavior of 3-rd party library, but that's a lot of work...

Just curious - this means that you have a lot of methods which are exposed by the "plugin" to the host, right?

[BaHXeLiSiHg]

Got it.

Not so many methods) Generally, host only knows how to activate ruler-class within assembly (plugin), which can call any part of this assembly (plugin) described in interfaces (interfaces are implemented by assemblies (plugins), likely 10 or 12 interfaces that describes every dynamic assembly (plugins) behavior).

In short, invocation looks like this:

public static AssemblyLoadContext.ContextualReflectionScope ActivateLanguagesAssembly(out ILanguagesAssembly assembly)
{
    var contextualReflection = DynamicAssemblyLoader.ActivateDynamicInstance(out assembly);
    ...
    return contextualReflection;
}

And in required places allow to access dynamic assembly for something:

using (ActivateLanguagesAssembly(out var languages))
{
    LanguageMetadata metadata = languages.GetMetadata();
    ...
    return metadata;
}

But every dynamic assembly (plugin) could access any other assembly (plugin), depends on how it was generated. So i have a lot of places in those generators which should be replaced using invocation methods, described above.

[BaHXeLiSiHg]

@vitek-karas, done with minimal changes required for testing, but i think i have messed with something...

Expected ContextualReflectionScope in particular thread was changed by parallel request of another dynamic assembly (plugin).

Do i missing something? Does ContextualReflectionScope could be only one per application (threads)?

[vitek-karas]

The value is stores in an async-local field - so it should be per-thread (actually better on thread pool).
The using should take care of resetting it to original value once you leave the using block.
If the using blocks are correct I can't think of a case where they would "collide" the way you describe above.

[BaHXeLiSiHg]

@vitek-karas, i see. What would happen if i use it like this, if you any suggessions:

public AssemblyLoadContext.ContextualReflectionScope GetInnerAssembly(out SomeType innerAssembly)
{
    using (AssemblyLoadContext.ContextualReflectionScope scope = ActivateRegistryAssembly(out var registryAssembly))
    {
        registryAssembly.DoSomeStuff()
        ...
        AssemblyLoadContext.ContextualReflectionScope innerScope = registryAssembly.ActivateInnerAssembly(out innerAssembly);
        ...
        return innerScope ;
    }
}

Does such using may cause problems with ContextualReflectionScope? innerScope are used outside of activation method within using statement, to make whatever i need.

As i understand, disposing of scope scope will restore previous ContextualReflectionContext which is null. But created innerScope has scope as "parent" (previous ContextualReflectionContext) and after disposing of innerScope parent will be disposed already :\

[BaHXeLiSiHg]

Seems like i'm trying to use ContextrualReflectionScope a little bit out of recommended way :)

Well, my purpose is now to prevent restoring of predecessor CurrentContextualReflectionContext even if i'm out of predecessor using statement but still have something to do inside any created child ContextrualReflectionScope which is used as reference in some part of code...

To simplify, something like this:

ContextualReflectionScopeWrapper thirdScope;

var firstDomain = new AssemblyLoadContext("firstDomain", true);

using (var firstScope = firstDomain.WrapContextualReflection())
{
    var secondDomain = new AssemblyLoadContext("secondDomain", true);

    using (var secondScope = secondDomain.WrapContextualReflection(firstScope))
    {
        var thirdDomain = new AssemblyLoadContext("thirdDomain", true);

        thirdScope = thirdDomain.WrapContextualReflection(secondScope);
    }
}

var abc = 3;

using (thirdScope)
{
    abc = 4;
}

Have implemented AssemblyLoadContext extension with additional scope wrapper, it will prevent disposing of ContextrualReflectionScope when i'm out of using statement but not finished somethings within child scope.

Extension:

public static ContextualReflectionScopeWrapper WrapContextualReflection(
    this AssemblyLoadContext context,
    ContextualReflectionScopeWrapper predecessor = null)
{
    if (context == null)
    {
        throw new Exception("Context was not provided.")
    }

    var scope = context.EnterContextualReflection();

    var wrapper = new ContextualReflectionScopeWrapper(context, scope, predecessor);

    predecessor?.InviteReferral(wrapper);

    return wrapper;
}

And wrapper:

public sealed class ContextualReflectionScopeWrapper : IDisposable
{
    public EventHandler DisposePredecessor { get; private set; }

    public AssemblyLoadContext Context { get; }

    private bool _disposed;

    private AssemblyLoadContext.ContextualReflectionScope _scope;

    private ContextualReflectionScopeWrapper _referral;
    private ContextualReflectionScopeWrapper _predecessor;

    public ContextualReflectionScopeWrapper(
        AssemblyLoadContext context,
        AssemblyLoadContext.ContextualReflectionScope scope,
        ContextualReflectionScopeWrapper predecessor = null)
    {
        if (context == null)
        {
            throw CommonException.ArgumentNullException(nameof(AssemblyLoadContext));
        }

        Context = context;

        _scope = scope;
        _predecessor = predecessor;
    }

    public void InviteReferral(ContextualReflectionScopeWrapper referral)
    {
        if (_referral != null)
        {
            throw new Exception(
                $"Could not set referenced isolated scope wrapper for context \"{Context.Name}\". " +
                    $"Referenced context \"{_referral.Context.Name}\" already defined."
            );
        }

        _referral = referral ?? throw new Exception("Referenced context was not provided.");

        _referral.DisposePredecessor += OnReferralDisposing;
    }

    public static void OnReferralDisposing(object sender, EventArgs args)
    {
        if (!(sender is ContextualReflectionScopeWrapper wrapper))
        {
            throw new Exception("Referenced context should send his context predecessor in \"OnReferralDisposing\" event.");
        }

        wrapper._referral.DisposePredecessor = null;

        wrapper._referral._predecessor = null;

        wrapper._referral = null;

        if (!wrapper._disposed)
        {
            return;
        }

        wrapper.Dispose();
    }

    public void Dispose()
    {
        _disposed = true;

        if (_referral != null)
        {
            return;
        }

        _scope.Dispose();

        DisposePredecessor?.Invoke(_predecessor, null);
    }
}

Going to implement it in real code flow and see how it will maintain ContextualReflectionScope . Will be grateful for any hints about code mistakes or possible memory leaks on implementation like this :\