Memory leak per execution for CompiledCode

[mattomopt]

IronPython seems to leak memory every time I set a variable via compiled Python scripts. The memory seems to get collected if I let the CompiledCode object go out of scope, but I want to be able to reuse one CompiledCode many times because of the speed benefits. The leak does not happen when I execute the ScriptSource instead of CompiledCode. It also doesn't happen when I set the variables in the scope via ScriptScope.SetVariable instead of setting them in Python.

Setting values this way:

d = dict()
d[1] = 1
d[2] = 1
d[3] = 1
# ......
d[100000] = 1

leaks much faster than:

d = dict()
for i in range(1, 100000):
    d[i] = 1

I have reproduced the issue using the below program in 64-bit .NET 4.5 with both IronPython 2.7.4 and 2.7.5. The Script.py I used is similar to the first Python script above: I set 100000 dictionary entries individually. Other variations of the C# program I have tried that do not help with the leak:

• Using LightweightScopes = false
• Using ExecuteAndWrap instead of Execute
• Creating a new ScriptScope for each execution
• Setting the dictionary variable to be a .NET dictionary in in the program instead of declaring it in Python
• Building the script from a string rather than a file

    public static class Program
    {
        public static void Main()
        {
            var engine = Python.CreateEngine(
                new Dictionary<string, object> { { "Debug", false }, { "LightweightScopes", true } });
            var scope = engine.CreateScope();
            var script = engine.CreateScriptSourceFromFile(@"Script.py").Compile();
            for (var i = 0; i < 100000; i++)
            {
                script.Execute(scope);
                GC.Collect();
            }

            Console.ReadKey();
        }
    }

If it helps, I ran the above leaking program in windbg and after many loop iterations, here are the biggest heap stats given by "!dumpheap -stat":

              MT    Count    TotalSize Class Name
000007fee85b0178      326         8627 System.Boolean[]
000007fee85b0a50      125        10000 System.Signature
000007fee859b0b8       20        11424 System.Collections.Hashtable+bucket[]
000007feefa481a0      299        11960 Microsoft.CSharp.RuntimeBinder.Syntax.NameTable+Entry
000007fee858c7e8       94        12032 System.Text.SBCSCodePageEncoding
000007fe8f7ee7a8      305        12200 IronPython.Runtime.Operations.StringOps+EncodingInfoWrapper
000007feefa49788      121        12584 Microsoft.CSharp.RuntimeBinder.Semantics.AggregateType
000007fee85bc370      226        12656 System.Reflection.Emit.OpCode
000007feefa49500       62        16368 Microsoft.CSharp.RuntimeBinder.Semantics.MethodSymbol
000007feefa48730       80        17920 Microsoft.CSharp.RuntimeBinder.Semantics.AggregateSymbol
000007fe8f7ef7d0        1        22080 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[IronPython.Runtime.Operations.StringOps+EncodingInfoWrapper, IronPython]][]
000007fee85b0e98      192        23040 System.Reflection.RuntimeParameterInfo
000007fee8592690     1467       164304 System.Reflection.RuntimeMethodInfo
000007fee859d610     2992       167552 System.RuntimeType
000007fee859b320     3764       193132 System.String
000007fee859e070      240       420368 System.Int32[]
000007fe8f8a55c8    99999      2399976 Microsoft.Scripting.Interpreter.DynamicInstruction`4[[System.Object, mscorlib],[System.Object, mscorlib],[System.Object, mscorlib],[System.Object, mscorlib]]
000007feed613088   100000      2400000 System.Linq.Expressions.ClearDebugInfoExpression
000007fee859d960   100051      2401224 System.RuntimeTypeHandle
000007fee85a1300   100064      2401536 System.RuntimeMethodHandle
000007fe8f867e58   100000      3200000 IronPython.Compiler.LookupGlobalInstruction
000007feed61e560   100003      3200096 System.Linq.Expressions.AssignBinaryExpression
000007feed61e208   100009      3200288 System.Linq.Expressions.ListArgumentProvider
000007fee85ef938   100030      3200960 System.Reflection.Emit.GenericMethodInfo
000007feed61bc18   100000      4000000 System.Linq.Expressions.SpanDebugInfoExpression
000007feed61f0a8   100002      4000080 System.Linq.Expressions.MethodCallExpression2
000007fe8f80f3c8   100008      4000320 IronPython.Compiler.LookupGlobalVariable
000007fe8f95d130    99999      4799952 System.Runtime.CompilerServices.CallSite`1[[System.Func`5[[System.Runtime.CompilerServices.CallSite, System.Core],[System.Object, mscorlib],[System.Int32, mscorlib],[System.Int32, mscorlib],[System.Object, mscorlib]], mscorlib]]
000007fe8f8a8720    99999      4799952 System.Runtime.CompilerServices.CallSite`1[[System.Func`5[[System.Runtime.CompilerServices.CallSite, System.Core],[System.Object, mscorlib],[System.Object, mscorlib],[System.Object, mscorlib],[System.Object, mscorlib]], mscorlib]]
000007fe8f82b058    99999      4799952 IronPython.Compiler.PythonDynamicExpressionN
000007fe8f80a5c0    99999      4799952 IronPython.Compiler.Ast.IndexExpression
000007fe8f80a998   100000      4800000 IronPython.Compiler.Ast.AssignmentStatement
000007feed60dbe8    99999      5599944 System.Linq.Expressions.DynamicExpression3
000007fe8f809aa0   100001      5600056 IronPython.Compiler.Ast.NameExpression
000007fe8f8a85d0   100001      6400064 System.Func`5[[System.Runtime.CompilerServices.CallSite, System.Core],[System.Object, mscorlib],[System.Object, mscorlib],[System.Object, mscorlib],[System.Object, mscorlib]]
000007fee85cade0   200004      6400128 System.Reflection.Emit.GenericFieldInfo
000007feed61e3a0   200008      6400256 System.Linq.Expressions.Block2
000007fee859e0d0   299405      7185720 System.Int32
000007fe8f84c958   299552      7189248 Microsoft.Scripting.Interpreter.LoadObjectInstruction
000007fe8f801ea8   199999      7999960 IronPython.Compiler.Ast.ConstantExpression
000007feed61e720   200000      8000000 System.Linq.Expressions.Block3
000007fe8f8428b0   200000      8000000 Microsoft.Scripting.Interpreter.DebugInfo
000007feed60c040   400022      9600528 System.Linq.Expressions.ConstantExpression
000007fe8f7712d0       14     10073568 IronPython.Runtime.CommonDictionaryStorage+Bucket[]
000007feed61e458   400013     12800416 System.Linq.Expressions.BlockExpressionList
000007fee859f498       99     14199148 System.Byte[]
000007feed2c99b0   500023     16000736 System.Collections.ObjectModel.ReadOnlyCollection`1[[System.Linq.Expressions.Expression, System.Core]]
0000000000736640    15294     19583260      Free
000007fee857f5e8   301116     35046696 System.Object[]

[jdhardy]

Long delay, for which I'm sorry - can you provide Script.py?

[mattomopt]

Thanks for the reply. I attached the two scripts described above here.
LeakingScript.txt
NonLeakingScript.txt

[slide]

@jdhardy Does #1250 fix this as well?

[jdhardy]

Haven't checked yet. The trigger seems to be a bit different than the other cases but I haven't compared it yet.

[mattomopt]

I tried pulling and testing that fix on my own. It does not seem to help with my case.

[slide]

This issue was moved to IronLanguages/ironpython2#114