AssemblyLoadContext unloadability and 3rd party libraries

[Flohack74]

Hello, I have quite some issues with not being able to unload an ALC that uses the log4net dependency. I already attached a few remarks on #30656 but I feel its worth having a separate issue for that.

What it boils down to is that I have no control over whats happening in any 3rd party library, be it log4net, or, also used by us, DevExpress components. If I use those libs in an ALC and later on I cannot unload them it is a huge blocker for having any 3rd party libs in an ALC, because you will never know under which circumstances they could stay in memory or not.

I dont think that was the intention here, yet it seems to be a real-world problem. I went left and right with the mem profiler already and also with @raffaeler ´s tool PowerDiagnostic to at least understand which object reference keeps the ALC hot, but so far to no avail. Its not a thread/task, also most likely not an event subscription, but more than that I am not sure.

Note that we transitioned from an AppDomain concept into ALC when we migrated from 4.8 to net8 last year. Only now it became evident that this move created memleak issues in our backend. If this is not solveable for us, we need to make a huge refactoring on the code that ideally runs user/client requests in an isolation container.

I got a dump file on request to showcase the issue, and also https://github.com/Flohack74/ALCwithLog4Net holds an example of how the memory builds up with my custom ALC. Grateful for any advice! :)

[dotnet-policy-service]

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

[huoyaoyuan]

See discussions in #76864 and #103944.

[Flohack74]

@huoyaoyuan thanks for the references. Indeed I also agree that it is not really feasable to have support from the framework for unloading. I would rather ask into the direction of how to prevent stuck references, like e.g. preloading certain libraries into the ALC to prevent referencing types from the default load context.

So far that did not help me to solve the problem, but maybe I am also not preloading the right libs for that :)

Unfortunately it is really unlikely that a modern application works with almost no 3rd party libs, and if it is very likely that 3rd parties can keep an ALC in memory and the consumer has no way of cleaning them up then the usefulness of an ALC is limited, so to say.

For my company the discussions point into the direction to now investigate if we need to unload the ALCs around our backend service at all, or keep them as lomng-lived worker containers around. I am just afraid that they might lead to some degradation over time: For some reason e.g. Apache web server recycles its workers after a certain amount of requests. Nothing lives forever.

[agocke]

@janvorli who may be able to help trace down the issue with the dump.

To set expectations though, we do not plan to bring back AppDomain functionality that unsafely unloaded assemblies while references were still present into them. If this is a request for better tooling to try to find those references -- that's likely something we can help with.

[raffaeler]

For the records, the ALC is a cooperative feature, therefore there should not be any active references pointing to the secondary ALC to make unloading succeed. This is totally different from the AppDomain which forcibly unload the assemblies.
An example for the assembly loaded in the secondary ALC to be responsible is when it subscribe to an event in the default ALC. But of course there are many other possibilities.

When I initially worked to my tool PowerDiagnostics, @janvorli kindly added a precious API to make it possible to find the dependencies of the ALCs. This provided me the way to obtain the results that @Flohack74 documented here.

I resolved similar issues in the past by taking control over the Assembly Loading methods of the secondary ALC and loading a copy of the assemblies which had a strong dependency with my addin (loaded in the secondary ALC). This allowed me to unload the ALC with all its assemblies.
Anyway, finding the exact assemblies is not straightforward and depends on the assemblies being loaded in the secondary ALC.

@Flohack74 Now that you know that it is a ConcurrentDictionary, either you find a way to clear it or you have to load its assembly in the secondary ALC as well (with all the possible related issues).

HTH

[Flohack74]

Okay I will look into that. Thanks!

[Flohack74]

So I think I have loaded ALL possible DLLs that I can preload into the ALCs and yet they are still sitting in memory. To give an idea its like that:

        private static List<string> preloadSystemAssemblies = new List<string>
        {
            //"System.Private.CoreLib.dll",
            "Microsoft.CSharp.dll",
            "System.Collections.dll",
            "System.Collections.Concurrent.dll",
            "System.ComponentModel.EventBasedAsync.dll",
            "System.ComponentModel.Primitives.dll",
            "System.ComponentModel.TypeConverter.dll",
            "System.Console.dll",
            "System.IO.Pipes.dll",
            "System.Linq.dll",
            "System.Runtime.dll",
            "System.Runtime.InteropServices.dll",
            "System.Runtime.Loader.dll",
            "System.Security.AccessControl.dll",
            "System.Security.Claims.dll",
            "System.Security.Principal.Windows.dll",
            "System.Threading.dll",
            "System.Threading.Overlapped.dll",
            "System.Threading.Thread.dll",
            "System.Private.Xml.dll",
            "System.Private.Uri.dll",
            "System.Collections.Specialized.dll",
            "System.Diagnostics.Process.dll",
            "System.Configuration.ConfigurationManager.dll",
            "System.Xml.ReaderWriter.dll",
            "System.Net.WebClient.dll",
            "System.Net.Primitives.dll",
            "System.Memory.dll",
            "System.Text.Encoding.Extensions.dll"
        };

The only one which is commented rightfully at the beginning is the Corelib which I assume, is never accessible for user code. I also double-checked 2 times now that BEFORE the 1st ALC goes live, ALL assemblies used by the host process are also in that list. I have confirmed with the Modules view of VS that indeed I get now multiplication for ALL DLLs so that I see little chance that there is still a shared one that keeps a reference into each ALC.

And yet log4net seems to break the jail. @agocke yes it would now be very helpful to at least understand which type was loaded from / forwarded to the default ALC. Or some other tooling like that. Thanks!

[Flohack74]

WAIT! Can it be that when I replace my custom ALC with the default one, everything works? using var loadContext = new AssemblyLoadContext("DoSomeWork", true); seems to yield unloading in contrast to my var loadContext = new PluginLoadContext(assemblyLoadPaths, preloadSystemAssemblies);

So maybe the problem is in customizing an ALC, not the actual libraries being loaded? My ALC does not much except of overriding the Load method (which is necessary in our application for some complex blah)

Ok nevermind was obviously my tired eyes that did see smth what was not to be seen. It still happens also with the implemented one...