ClrMD memory leak Thread.BlockingObjects

[Aleksei-Poliakov]

The code:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Globalization;
using System.Linq;
using System.Threading;
using Microsoft.Diagnostics.Runtime;

namespace ConsoleApplication1
{
    internal class MyClass
    {
        public static void Main()
        {
            var thread = new Thread(
                () =>
                {
                    while (true)
                    {
                        Console.WriteLine("123");
                    }
                });

            thread.Start();

            var pid = Process.GetCurrentProcess().Id;

            using (var dataTarget = DataTarget.AttachToProcess(pid, 5000, AttachFlag.Passive))
            {
                string dacLocation = dataTarget.ClrVersions[0].TryGetDacLocation();
                var runtime = dataTarget.CreateRuntime(dacLocation);

                while (true)
                {
                    Thread.Sleep(200);


                    foreach (var t in runtime.Threads)
                    {
                        var temp = t.BlockingObjects;
                    }

                    runtime.Flush();

                    GC.Collect();
                }
            }
        }
    }
}

When you run this code managed memory stays consitent, but native memory grows consistently. The detalization of most memory allocations below:

In my test run I got from 2.2 MB to 10.5 MB in 8 minutes, so I guess it is easily reproducible.

[goldshtn]

I suspect it might have to do with finalization -- can you add GC.WaitForPendingFinalizers(); GC.Collect(); after your GC.Collect(); call to verify? If the leak goes away, then it's just some COM stuff (RCWs) not getting reclaimed at the first GC.

[weltkante]

That explanation sounds wrong, he's using a loop so "not getting reclaimed at the first GC" makes no sense. There should be plenty of time for the finalizer thread to clean up stuff (unless he's producing it faster than it can be cleaned up).

Anyways, I've run his sample code he posted above and on VS 2015 / .NET 4.6 this does not leak any memory.

[update] Wasn't running it long enough here, the leak isn't large and the example may need to run a few minutes before the leak becomes noticeable.

[Aleksei-Poliakov]

What version of ClrMD are you using? I have tried it in .Net 4.6 as soon as Sasha posted his comment and got "This runtime is not initialized and contains no data." exception

Microsoft.Diagnostics.Runtime.RuntimeBase..ctor(DataTargetImpl dataTarget, DacLibrary lib)
Microsoft.Diagnostics.Runtime.Desktop.DesktopRuntimeBase..ctor(DataTargetImpl dt, DacLibrary lib)
Microsoft.Diagnostics.Runtime.Desktop.LegacyRuntime..ctor(DataTargetImpl dt, DacLibrary lib, DesktopVersion version, Int32 minor)
Microsoft.Diagnostics.Runtime.DataTargetImpl.CreateRuntime(String dacFilename)
ConsoleApplication1.MyClass.Main() Program.cs: line: 31
System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
System.Threading.ThreadHelper.ThreadStart()

ClrVersions collection of DataTarget objects contains only 1 item 4.6.81.00
I am using 0.8.27-beta version from NuGet

[weltkante]

@Darkwalker that's a known issue (#11) with .NET 4.6, it looks for a 4.0 runtime instead of the 4.5+ runtime - see PR #23 for a fix (it's a one liner to fix the version check which has been known for a while but unfortunately nobody is updating the repository or the nuget package)

[Aleksei-Poliakov]

Patched dll does not throw exception, but it still leaks about 1 MB per minute, even after addition of GC.WaitForPendingFinalizers();
Now code looks like this:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Globalization;
using System.Linq;
using System.Threading;
using Microsoft.Diagnostics.Runtime;

namespace ConsoleApplication1
{
    internal class MyClass
    {
        public static void Main()
        {
            var thread = new Thread(
                () =>
                {
                    while (true)
                    {
                        Console.WriteLine(456);
                    }
                });

            thread.Start();

            var pid = Process.GetCurrentProcess().Id;

            using (var dataTarget = DataTarget.AttachToProcess(pid, 5000, AttachFlag.Passive))
            {
                string dacLocation = dataTarget.ClrVersions[0].TryGetDacLocation();
                var runtime = dataTarget.CreateRuntime(dacLocation);

                while (true)
                {
                    Thread.Sleep(200);


                    foreach (var t in runtime.Threads)
                    {
                        var temp = t.BlockingObjects;
                    }

                    runtime.Flush();

                    GC.Collect();
                    GC.WaitForPendingFinalizers();
                    GC.Collect();
                }
            }
        }
    }
}

[weltkante]

Hm, ok I think I can see it now too, wasn't running it long enough previously. Seems to leak less for me than for you though.

I also see peaks in memory and across those peaks the leaks are a bit larger (might be the same as the spikes in your graph, but then you have a different resolution than me).

I'll take a closer look on it later or maybe next week, as time permits, maybe I can figure something out.

[goldshtn]

@Darkwalker Trying to debug this I'm seeing wildly different results on different OS and CLR versions. More importantly, I'm seeing different results between analyzing a dump file and attaching to a live process. Can you confirm your OS and CLR versions?

[goldshtn]

@Darkwalker Can you please confirm that when removing the .Flush() call, you don't experience a memory leak?

[Aleksei-Poliakov]

I am running on Windows 8.1 (6.3.9600) and .Net 4.6.81.00
Yes, without .Flush() call there is no memory leak.

[goldshtn]

@Darkwalker Well, here's one leak I found (I am sure there are more because this one is only responsible for about 10% of the leaking heap memory I can see).

This specific condition occurs when running your sample, i.e. when the process "attaches" to itself and tries to obtain the blocking objects for each thread.

Getting the thread's BlockingObject collection accesses its StackTrace property, which in turn calls IXCLRDataTask::CreateStackWalk. This method is implemented in mscordacwks!ClrDataTask::CreateStackWalk, and it in turn constructs a ClrDataStackWalk object and calls its Init method. That method initializes a StackFrameIterator object and calls its Init method. Finally, StackFrameIterator::Init throws an exception. This exception is caught by ClrDataTask::CreateStackWalk, but the memory allocated for a ClrDataStackWalk object and its corresponding objects is not reclaimed in that case.

The failure in StackFrameIterator::Init probably occurs because you're trying to obtain the stack trace for a thread that is currently running (it returns successfully for other threads). If StackFrameIterator::Init returned an error code, memory would be reclaimed; it's the exception that is unexpected and causes a leak.

Yesterday I was also able to reproduce this bug when attaching to another process (not the same process), so I will try looking into that as well.

[goldshtn]

Found some more tiny leaks like that. For example, calling ClrHeap.GetObjectType for some specific object address can cause a few dozen bytes to leak (for each call) due to an error inside mscordacwks!ClrDataAccess::GetMethodTableData. When the error occurs, an exception is thrown using the EX_THROW macro, which allocates the new exception object from the heap (!).

I have to conclude that it seems there was no serious stress-testing performed of the DAC interfaces (CLRMD is probably not at fault here), and we're looking at the results of that...

[weltkante]

Nice work. Are the coreclr guys aware of this? If not it might make sense to create an issue there, too.

[terrajobst]

We've moved CLR MD into its own GitHub repository. If this issue still applies, please file it there.

Sorry for the inconvenience but we never intended for this repo to contain the component itself.