Memory leak in file operations in mono 5.10

[andrewaggb]

Steps to Reproduce

1. Seems to be related to using a lot of FileInfos and/or deleting, copying, appending files. I also access LastWriteTimeUtc alot which I suspect most apps don't.
   Essentially I have approximately 20 ffmpeg processes creating files 5 or so files (per process) every second in a tmpfs mount.

I believe the relevant code is below. Depending on the type of file the file extension in the search may be different but otherwise it's doing the same thing. Finding files older than a certain time and either deleting them outright or combining them into a larger file on another drive and then deleting them.

DirectoryInfo cid = ...
var allfiles = cid.GetFiles("*.0.ts", SearchOption.TopDirectoryOnly).ToArray();
var nowMinus12 = DateTime.UtcNow.AddSeconds(-12);
var files = allfiles.Where(x => x.LastWriteTimeUtc < nowMinus12).OrderBy(x => x.Name).ToArray();
if (!files.Any() || (files[0].LastWriteTimeUtc.AddSeconds(42) >= DateTime.UtcNow)) return;

....
// Note: mfiles is an array of FileInfo

                var finfo = new List<TSFileInfo>();
                using (var fs = File.Open(fname + ".tmp", FileMode.Create))
                {
                    long offset = 0;
                    for (int k = first; k <= last; ++k)
                    {
                        var mfile = mfiles[k];
                        using (var fr = mfile.OpenRead())
                        {
                            CopyStream(fr, fs);
                            finfo.Add(new TSFileInfo() { Offset = offset, tStart = mfile.TSFnCreate() });
                            offset += mfile.Length;
                        }
                        mfile.Delete();
                    }
                    fs.Close();
                }
                File.Move(fname + ".tmp", fname);
                File.WriteAllText(fname + ".json", Newtonsoft.Json.JsonConvert.SerializeObject(finfo));

        public static void CopyStream(Stream input, Stream output)
        {
            byte[] buffer = new byte[65536];
            while (true)
            {
                int read = input.Read(buffer, 0, buffer.Length);
                if (read <= 0)
                    return;
                output.Write(buffer, 0, read);
            }
        }

Some files are copied to another drive and all deleted from tmpfs using the FileInfo Delete method
The actual file writes/copy are done by opening a FileStream on another drive and writing bytes to it.

I could build an app to reproduce the issue if necessary. I've been putting out fires everywhere as it's been crashing my systems (my bad letting yum and apt auto-update systems). My work around for now is to rollback mono on Centos 6 systems and use .net core on Centos 7 or Ubuntu systems.

Current Behavior

As of Mono 5.10 I have an application that leaks memory very fast. It can be using upwards of 80% of system memory according to top after running for a few hours. The same app on the first version of mono 5.8 or older or on .net core holds steady at 1-3% of memory depending on the system (hundreds of installs).

Expected Behavior

Memory usage holds steady instead of constantly increasing over time.

[ ] macOS
[ X] Linux, Centos 6, Centos 7, Ubuntu 16.04
[ ] Windows

Version Used:
Works on mono 4.6.2, 5.2.0, 5.4.0, 5.8.0.108 (unsure about the latest version of 5.8), doesn't work on mono 5.10.0.140 or 5.10.0.160

I'm currently testing to see if mono 5.8.0.127 is affected.

Stacktrace

Please paste the stack trace here if available.

[andrewaggb]

This is a simple app to reproduce issue. All this does is create small files, find them, and delete them.

Use a tmpfs mount so it goes faster and you don't wreck your ssd/disk/whatever.
mkdir /tmp/memleaktest
mount -t tmpfs -o defaults,size=64m,noatime tmpfs /tmp/memleaktest
mono FileSystemMemoryLeak.exe /tmp/memleaktest

This zip file has a Visual studio project file, the single source file, and a release build binary from VS 2017. By all means build it yourself. I didn't try building it with mono but I assume that doesn't matter as it only happens on mono 5.10
FileSystemMemoryLeak.zip

Leaks memory pretty fast on mono 5.10.0.160. By the time the loop counter has hit a few thousand it's already doubled it's memory usage. By loop 6000 it's tripled. Very easy to see after a few minutes.

Same test on mono 5.8.0.107 maintains steady memory usage the whole time. As I mentioned earlier in the ticket, earlier versions of mono such as 4.6.2,5.2,5.4 all work with constant memory usage. The same code compiled as a .net core app and run under dotnet 2.0.6 also works fine.

This code is included in the zip file.

using System;
using System.Linq;
using System.IO;
using System.Threading;

namespace FileSystemMemoryLeak
{
    class Program
    {
        static void Main(string[] args)
        {
            if (!args.Any())
            {
                Console.WriteLine(
@"Usage:
    FileSystemMemoryLeak PathToCreateFiles
eg  
Note: Use a tmpfs mount or something similiar as the 'PathToCreateFiles' target as this app will thrash the disk
");
                return;
            }

            if (!Directory.Exists(args[0]))
            {
                Console.WriteLine($"Error: Directory {args[0]} does not exist");
                return;
            }

            var sourceDir = Directory.CreateDirectory(Path.Combine(args[0], "source"));
            var data = new byte[8192];
            long counter = 0;
            while (true)
            {
                try
                {                   
                    //Create nonsense files
                    for (int i=0; i < 1000; ++i)
                    {
                        using (var fs = File.Open(Path.Combine(sourceDir.FullName, i.ToString() + ".tmp"), FileMode.Create))
                        {
                            fs.Write(data, 0, data.Length);
                        }
                    }
                    // find files
                    var files = sourceDir.GetFiles("*.tmp", SearchOption.TopDirectoryOnly);

                    // delete files
                    foreach (var f in files)
                    {
                        f.Delete();
                    }
                    Thread.Sleep(1);
                }
                catch(Exception e)
                {
                    Console.WriteLine(e.ToString());
                }
                if (counter++ % 1000 == 0)
                {
                    Console.WriteLine(DateTime.Now.ToString() + ", Loop Counter: " + counter);
                }
            }
        }
    }
}

[marek-safar]

I can reproduce the issue and I suspect GetFiles call.

@EgorBo could you try to track it down?

[vargaz]

findhandle_close () seems to be missing a
findhandle_unref (findhandle);
call.

[luhenry]

@vargaz at https://github.com/mono/mono/blob/master/mono/metadata/w32file-unix.c?utf8=%E2%9C%93#L4706 it should be the following:

static void
finds_remove (gpointer data)
{
	MonoFindHandle* findhandle;

	findhandle = (MonoFindHandle**) data;
	g_assert (findhandle);

	mono_refcount_dec (findhandle);
}

...

finds = g_hash_table_new_full (g_direct_hash, g_direct_equal, NULL, finds_remove);

[HeikoWeiss]

seems to be fixed with my test in
#7734

Please check it.
Thank You.

[pr0vieh]

@HeikoWeiss i think it's a good idea to open a Pull Req for your fix then you get feedback if your Fix is the right solution of this Bug

[marek-safar]

@luhenry could you turn that into PR?

[schtibb]

We have to use mono 5.12 in our production environment which is affected by this memory leak. If we use the nuget Mono Posix package and Mono.Unix.UnixDirectoryInfo.GetFileSystemEntries instead of System.IO.DirectoryInfo.GetFiles will that help? We are running mono on Linux

[adsb-related-code]

Still not fixed. Using 5.8.0.127 in production, any later mono leaks memory and crashes out application.

[avkhitry]

I have similar problem with memory leak during file operations, any chance it will be fixed?

[adsb-related-code]

This needs to be reopened and addressed.

[lewurm]

@adsbxchange the fix made it into 5.12 and later. can you try a newer mono?

[adsb-related-code]

I just tried 6.xx preview and current 5.xx stable, and all versions between by pinning the repo. Verified twice. Takes 13 minutes to crash the app with any version after 5.8.0.127 Uses all system ram then we have to force kill it before Linux starts killing processes.

…

________________________________ From: Bernhard Urban <notifications@github.com> Sent: Tuesday, April 23, 2019 8:00:10 AM To: mono/mono Cc: James Stanford; Mention Subject: Re: [mono/mono] Memory leak in file operations in mono 5.10 (#7655) @adsbxchange<https://github.com/adsbxchange> the fix made it into 5.12 and later. can you try a newer mono? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#7655 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AIHVZBV2M4RGRBOI2AJ5Z43PR4P7TANCNFSM4EVYCPXQ>.

[adsb-related-code]

vradarserver/vrs#11

VRS dev is salty about it. Not sure where the issue is. But being that is been an issue through multiple releases, there should be some fix.

[lewurm]

@adsbxchange in order to track this down we need a reproducer. The repro from #7655 (comment) works fine for me on 6.0.

[adsb-related-code]

@adsbxchange in order to track this down we need a reproducer. The repro from #7655 (comment) works fine for me on 6.0.

then it might not be the same issue or it's only related ... VRS (virtualradarserver.co.uk) is built in .NET and opens a lot of file descriptors and only leaks in mono after 5.8.0.127

Easily reproducible when using Mono.

Hopefully that helps ...

[emmanuelmathot]

using valgrind to perform deep memory leak check, I can confirm that huge memory leaks happen with mono > 5.8.0.108

When method uses a initialized FileInfo class with mono 5.20.1.19:

==28074== LEAK SUMMARY:
==28074==    definitely lost: 6,533,670 bytes in 62,818 blocks
==28074==    indirectly lost: 35,549 bytes in 364 blocks
==28074==      possibly lost: 42,287 bytes in 116 blocks
==28074==    still reachable: 68,151,840 bytes in 442,799 blocks
==28074==                       of which reachable via heuristic:
==28074==                         stdstring          : 19,308 bytes in 576 blocks
==28074==         suppressed: 0 bytes in 0 blocks
==28074== Reachable blocks (those to which a pointer was found) are not shown.
==28074== To see them, rerun with: --leak-check=full --show-leak-kinds=all

same program with mono 5.8.0.108:

==15143== LEAK SUMMARY:
==15143==    definitely lost: 95 bytes in 2 blocks
==15143==    indirectly lost: 0 bytes in 0 blocks
==15143==      possibly lost: 13,952 bytes in 22 blocks
==15143==    still reachable: 61,483,893 bytes in 294,702 blocks
==15143==                       of which reachable via heuristic:
==15143==                         stdstring          : 19,694 bytes in 581 blocks
==15143==         suppressed: 0 bytes in 0 blocks
==15143== Rerun with --leak-check=full to see details of leaked memory
==15143== 
==15143== For counts of detected and suppressed errors, rerun with: -v
==15143== ERROR SUMMARY: 3920 errors from 53 contexts (suppressed: 0 from 0)

[adsb-related-code]

Mono still leaks. Has anyone figured this out?

[adsb-related-code]

There's this issue reported a couple of days ago in github for mono - looks like a possible culprit, VRS creates a lot of FileInfo objects when it's caching up aircraft photographs, silhouettes and operator flags:

#7655

https://forum.virtualradarserver.co.uk/viewtopic.php?t=1554