binderEquivalenceCache prevents unloading assembly

[Irrialite]

I am stuck on unloading an assembly because BinderEquivalence.cs caches ICSharpBinder which references that assembly's types.

This is the pinned handle:

HandleTable:
    000001F88CBC15C0 (pinned handle)
    -> 000001F89EA2C580 System.Object[]
    -> 000001F88F2D8F88 System.Collections.Concurrent.ConcurrentDictionary`2[[Microsoft.CSharp.RuntimeBinder.ICSharpBinder, Microsoft.CSharp],[Microsoft.CSharp.RuntimeBinder.ICSharpBinder, Microsoft.CSharp]]
    -> 000001F88F2D9140 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[Microsoft.CSharp.RuntimeBinder.ICSharpBinder, Microsoft.CSharp],[Microsoft.CSharp.RuntimeBinder.ICSharpBinder, Microsoft.CSharp]]
    -> 000001F88F2D9028 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[Microsoft.CSharp.RuntimeBinder.ICSharpBinder, Microsoft.CSharp],[Microsoft.CSharp.RuntimeBinder.ICSharpBinder, Microsoft.CSharp]][]
    -> 000001F88F2D9168 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[Microsoft.CSharp.RuntimeBinder.ICSharpBinder, Microsoft.CSharp],[Microsoft.CSharp.RuntimeBinder.ICSharpBinder, Microsoft.CSharp]]
    -> 000001F88F2D0B38 Microsoft.CSharp.RuntimeBinder.CSharpConvertBinder
    -> 000001F88F1301D8 System.RuntimeType
    -> 000001F88F12B398 System.Reflection.LoaderAllocator

This is the offending source code line:

private static readonly ConcurrentDictionary<ICSharpBinder, ICSharpBinder> binderEquivalenceCache =
            new ConcurrentDictionary<ICSharpBinder, ICSharpBinder>(concurrencyLevel: 2, capacity: 32, new BinderEqualityComparer()); 

It should be cleared before assembly unload or give us the ability to do so manually.

Edit: after further investigation it seems that a bunch of other stuff is kept as well. Mostly statics from namespace Microsoft.CSharp.RuntimeBinder.Semantics. I haven't gone much further yet. We use dynamic type in the unloadable assembly.

Edit2: This is the code that fixes it, however, the next dynamic type usage crashes with null value.

Assembly ass = ...;
var type1 = ass.GetType("Microsoft.CSharp.RuntimeBinder.BinderEquivalence");
if (type1 != null) {
   var cache_field = type1.GetField("binderEquivalenceCache", BindingFlags.NonPublic | BindingFlags.Static);
   var cache = cache_field.GetValue(null);
   var clear_method = cache.GetType().GetMethod("Clear");
   clear_method.Invoke(cache, null);
}
var type2 = ass.GetType("Microsoft.CSharp.RuntimeBinder.Semantics.PredefinedMembers");
if (type2 != null) {
   var field1 = type2.GetField("_methods", BindingFlags.NonPublic | BindingFlags.Static);
   var arr1 = field1.GetValue(null) as Array;
   var len1 = arr1.Length;
   for (int i = 0; i < len1; ++i) {
       arr1.SetValue(null, i);
   }

   var field2 = type2.GetField("_properties", BindingFlags.NonPublic | BindingFlags.Static);
   var arr2 = field2.GetValue(null) as Array;
   var len2 = arr2.Length;
   for (int i = 0; i < len2; ++i) {
       arr2.SetValue(null, i);
   }
}
var type3 = ass.GetType("Microsoft.CSharp.RuntimeBinder.Semantics.NamespaceSymbol");
if (type3 != null) {
   var field = type3.GetField("Root", BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Static);
   var root = field.GetValue(null);
   var fields = new List<FieldInfo>();
   ObjectUtil.GetFieldsForType(root.GetType(), fields);
   foreach (var f in fields) {
       if (f.Name == "firstChild" || f.Name == "_lastChild" ||
           f.Name == "nextChild" || f.Name == "nextSameName") {
           f.SetValue(root, null);
       }
   }
}
var type4 = ass.GetType("Microsoft.CSharp.RuntimeBinder.Semantics.SymbolStore");
if (type4 != null) {
   var field = type4.GetField("s_dictionary", BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Static);
   var dict = field.GetValue(null);
   var clear_method = dict.GetType().GetMethod("Clear");
   clear_method.Invoke(dict, null);
}
var type5 = ass.GetType("Microsoft.CSharp.RuntimeBinder.Semantics.TypeArray");
if (type5 != null) {
   var field = type5.GetField("s_tableTypeArrays", BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Static);
   var dict = field.GetValue(null);
   var clear_method = dict.GetType().GetMethod("Clear");
   clear_method.Invoke(dict, null);
}
var type6 = ass.GetType("Microsoft.CSharp.RuntimeBinder.Semantics.TypeTable");
if (type6 != null) {
   var fields = type6.GetFields(BindingFlags.Static | BindingFlags.NonPublic);
   foreach (var field in fields) {
       var dict = field.GetValue(null);
       var clear_method = dict.GetType().GetMethod("Clear");
       clear_method.Invoke(dict, null);
   }
}

[jaredpar]

Thanks for filing the detailed report here.

Unfortunately the dynamic infrastructure was designed in an era where we didn't have fine grained Assembly unloading. It was designed when we only had to worry about AppDomain unloading (for which this works). The way that caching works here ends up holding onto many types, at many different levels, that will essentially make an Assembly unloadable.

Fixing this is not a simple bug fix, it's essentially a feature request because it would require us to pretty substantially change our caching structure here. At the moment this area is only opening for servicing level bug fixes and is not taking feature requests. Hence I'm going to close this out. Can reconsider if we fund feature work in this area in the future.