Extreme memory leak #101

gavin-norman · 2012-05-14T09:40:21Z

Sorry to report this after bug #65 finally being fixed, but something similar (indeed worse) has started happening now :(

The IDE hangs, with high CPU usage. Looking at top, MonoDevelop is actually continually allocating memory -- which accounts for the high CPU usage I suppose! Within a second or two the process has allocated > 80% of my total RAM, and keeps growing. Obviously at this point the whole system grinds to a halt.

This has happened to me twice now since upgrading to version 0.3.9, and both times in different situations. In fact it happened so suddenly that I can't really remember exactly what I had just done which triggered it. I'll try to work out if there's some pattern to it, if it happens again.

gavin-norman · 2012-05-14T10:52:17Z

I originally titled this bug report "occasional extreme memory leak", but it's already happened 3 times in two hours today, which I don't think qualifies as occasional ;)

As far as I can recall, every time it's occurred when I was reorganising code using cut & paste, moving blocks around...

aBothe · 2012-05-14T16:17:33Z

always this 'occassional' - I hate it, because it's not even a hint that it's reproducable :D, but anyway, I'd say that you might download the mono-d sources, build them on your own and "debug" them (meaning getting the root of all evil code), only if you want to do so.

gavin-norman · 2012-05-21T15:21:29Z

Hm, ok I got as far as having built Mono-D (without errors), and trying to debug it, but the "Start Debugging" (F5) option is greyed out. Any ideas?

Still haven't worked out any pattern to this bug, but it's happening very often, like a few times an hour.

aBothe · 2012-05-21T15:28:10Z

I'm developing Mono-D on windows - so with VS instead of MD - I also have to copy all output files to the addins directory first (I created a .bat for me). You probably should debug the customly-built MonoDevelop.exe instead, so if it throws or hangs, you can set a breakpoint somewhere in Mono-D's code - anyway, haven't figured out yet how to do it on Linux :/

gavin-norman · 2012-05-22T11:23:08Z

Ok, I've made a bit of progress here.

I've managed to find a situation where the error occurs fairly reliably, and reduce it to a simple single-module case.

module RegistryFile;



/*******************************************************************************

    Imports

*******************************************************************************/

//private import IRegistry;



public class RegistryFile : IRegistry
{
    private ubyte[char[]] keys;

    private char[] file_content;

    public this ( char[] path )
    {
        scope file = new File(path, File.ReadExisting);
        scope ( exit ) file.close;

        this.file_content.length = file.length;

        file.read(this.file_content);

        enum State
        {
            LookForSep,
            LookForEOL
        }
        State state;

        foreach ( c; this.file_content )
        {
            switch ( state )
            {
                case State.LookForSep:
                    break;
                case State.LookForEOL:
                    break;
            }
        }
    }

    public ubyte[]* opIn_r ( char[] name )
    {
        return name in this.keys;
    }
}

The error occurs when typing

if ( c == ':' )
{
    separator = i;
    state = State.LookForEOL;
}

after the line case State.LookForSep:.

It doesn't always happen, and doesn't always happen at exactly the same point during typing the above lines, but seems to occur about 50% of the time in this situation.

Hopefully this might give you a hint as to what might be causing this... I'm very interested to see if you can reproduce the problem with these instructions!

(I've not looked any further into running MD in the debugger, btw.)

aBothe · 2012-05-22T16:00:50Z

Well, yay I was able to reproduce the error - thank you very much for that! :D
I'll start to figure out what it is exactly and how to handle it! :)

gavin-norman · 2012-05-22T16:26:11Z

I'm glad it's not just me then! :)

aBothe · 2012-05-22T17:08:02Z

This bug results of temporarily unfinished code - it doesn't know how to continue if there's only a if(..) { written after that case State.LookForSep: line... so well, lol, should take a couple of minutes, then it should be solved

gavin-norman · 2012-05-22T17:11:46Z

Ah that makes sense... It always seems to happen when I'm either writing completely new code (like in that example) or refactoring code, moving chunks around with cut & paste.

aBothe · 2012-05-22T17:22:09Z

Ok, fixed it - just reinstall and pray that it's working from now on :), if it's still not working, meh :/

[Edit] There has been even an other issue with "unfinished" code - but this should be solved now.

gavin-norman · 2012-05-23T09:34:23Z

Great news!

One question: Any idea how to set where MonoDevelop gets its addin updates from? I've compiled MD from source (version 3.1) and when I try to update the addins it says: "Could not get information from repository: http://addins.monodevelop.com/Beta/Linux/3.1.0/main.mrep."

I've had a look in the preferences, but can't find anything to determine where it looks for addin updates...

aBothe · 2012-05-23T17:28:12Z

You can enter even the beta channel url manually in the addins repo manager - just replace the 3.1.0 with 3.0.0 - because the one MD developer probably forgot to setup this directory online^^

gavin-norman · 2012-05-24T09:47:15Z

Oh no :(

I changed the repositories as suggested, uninstalled the addin, and now I can't reinstall it! It doesn't appear in the list of language bindings any more. I'm Mono-D-less :(

aBothe · 2012-05-24T10:41:18Z

Then just add the mono-d.alexanderbothe.com repository - it's also probably a further release issue.... really awful handling of addins - that's probably why there are so few ones available online^^

gavin-norman · 2012-05-24T11:06:15Z

Ok cool that worked, thanks. Version 0.3.9.5.

So the good news: the test case I described above now seems to be ok. I'll continue observation and let you know if it happens / doesn't happen again in other circumstances :)

The bad news: now MonoDevelop is constantly around 180% CPU usage when I open a large project. (This didn't happen before I updated to Version 0.3.9.5.) Possibly related to bug #96?

aBothe · 2012-05-24T11:13:17Z

gahh ok, so now define "large project" please - it's probably another "while(true)" something, but this happens in very few situations then..^^

gavin-norman · 2012-05-24T11:40:07Z

Well, for a start tango, which contains about 200K lines of code. Plus our own stuff which is also pretty big.

Is there anything I could try to see what might be triggering this?

aBothe · 2012-05-24T12:23:11Z

You probably could check if it's tango - or your project. By opening an other (or creating an empty) project. The main problem is that it's parsing all sources initially because it has to do so.

Is MD just computing something or is it massively allocating memory?

gavin-norman · 2012-05-24T15:03:49Z

No it's just CPU usage, no out of control memory allocation any more.

aBothe · 2012-05-24T15:11:06Z

I'd be really happy if you could debug or at least halt the process to see where it loops.. Hmm, do you have a windows pc somewhere with VS installed? Then you could check it :D

gavin-norman · 2012-05-24T15:57:29Z

Unfortunately not...

Looking at the MonoDevelop process with top -H, I can see that there are 9 threads which are contributing to this high CPU usage: one times mono and eight times MonoDevelop.

Connecting to the process in gdb yields the following information about the threads:

(gdb) info threads
  Id   Target Id         Frame 
  27   Thread 0x7fef30344700 (LWP 23671) "mono" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  26   Thread 0x7fef2f01b700 (LWP 23672) "mono" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  25   Thread 0x7fef23ea9700 (LWP 23673) "dconf worker" 0x00007fef306f7b03 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  24   Thread 0x7fef236a8700 (LWP 23674) "gdbus" 0x00007fef306f7b03 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  23   Thread 0x7fef21c05700 (LWP 23675) "MonoDevelop" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  22   Thread 0x7fef04b5c700 (LWP 23690) "MonoDevelop" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
  21   Thread 0x7fef0469f700 (LWP 23692) "MonoDevelop" 0x00007fef306f60bd in read () at ../sysdeps/unix/syscall-template.S:82
  20   Thread 0x7fef0449e700 (LWP 23693) "MonoDevelop" 0x00007fef309dd52d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
  19   Thread 0x7feeef9ef700 (LWP 23695) "MonoDevelop" sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:102
  18   Thread 0x7feeef43e700 (LWP 23697) "MonoDevelop" 0x00007fef30703b53 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
  17   Thread 0x7feeef3fd700 (LWP 23698) "MonoDevelop" sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:102
  16   Thread 0x7feeef3bc700 (LWP 23699) "MonoDevelop" sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:102
  11   Thread 0x7feeecf79700 (LWP 23715) "MonoDevelop" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  10   Thread 0x7feeec696700 (LWP 23716) "MonoDevelop" 0x000000004189fce9 in ?? ()
  9    Thread 0x7feeee70a700 (LWP 23717) "MonoDevelop" 0x00000000418a738b in ?? ()
  8    Thread 0x7feeedbc7700 (LWP 23718) "MonoDevelop" 0x0000000041d063e9 in ?? ()
  7    Thread 0x7feeed9c2700 (LWP 23719) "MonoDevelop" 0x0000000041d06390 in ?? ()
  6    Thread 0x7feeec479700 (LWP 23720) "MonoDevelop" 0x0000000040486007 in ?? ()
  5    Thread 0x7feeec274700 (LWP 23721) "MonoDevelop" 0x0000000041d063dc in ?? ()
  4    Thread 0x7feeedfd1700 (LWP 23722) "MonoDevelop" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  3    Thread 0x7feeed7c1700 (LWP 23723) "MonoDevelop" 0x00000000005d8d64 in ?? ()
  2    Thread 0x7feeed5bc700 (LWP 23724) "MonoDevelop" 0x0000000040485e6c in ?? ()
* 1    Thread 0x7fef314eb740 (LWP 23669) "MonoDevelop" 0x00007fef306f7b03 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysd

Not sure if any of that helps at all! Let me know if you'd like me to look into any thread in particular to try and get a backtrace or anything...

aBothe · 2012-05-24T16:14:22Z

All I could do now is to provide an extra version to check - yeah I guess this will do it - so all I do then is to let Mono-D note the last parsed document in a log file - to check in what document it's hanging.. how long will you be online today?

gavin-norman · 2012-05-24T16:16:03Z

Ok sounds cool! I'll be around for another 90 mins or so today, then back tomorrow :)

aBothe · 2012-05-24T23:56:08Z

So, I just added a 'hook' to the directory parser that hopefully will show you where it hangs..

You can download the package from http://mono-d.alexanderbothe.com/MonoDevelop.D_0.3.9.5_debug.mpack and install it directly, without any addins channel or whatever.
Then, when you start MD or open a D project, it'll put a parselog.log on your desktop (you have to remove it manually on every MD launch - it's 'appending' to the file only :) ) where you can see all files that are about to be parsed.

To make the results as transparent as possible, I recommend you to temporarily remove all includes - to exclusively focus on your project files.

I made the entire directory analysis procedure single-threaded - so it should be easier to follow what it's parsing currently :)

Anyway, by checking out the files that are parsed at latest, I really hope that we can find these error-causing language constructs :D

gavin-norman · 2012-05-25T09:45:15Z

Ok initial test results... The problem is actually caused by me including (in Options > Compiling > Includes) libraries by their root directory, rather than the subdirectory which contains the actual library code. So, for example, the tango root dir contains the following subdirs: build, doc, tango. The latter is where the actual source code lives, but I was giving the root dir in the include path (../tango).

If I change the include to ../tango/tango, then it's fine.

The weird thing is that this was never happening before version 0.3.9.5.

I'll have a look to see if I can narrow it down to which module is causing the problem when I include the root dirs.

gavin-norman · 2012-05-25T09:50:55Z

tango/doc/dil/data/xml_map.d seems to be the problem.

I notice it's not actually a valid D module, as it's lacking a module declaration. Interesting...

Thanks for the special debug version of the addin!

gavin-norman · 2012-05-25T10:32:56Z

Ok now I've found something interesting...

I'm now testing with a workspace which contains several solutions, one of which is tango. All the other solutions reference tango, and have ../tango/tango in their include paths. In the tango solution itself I've only included the tango subfolder in the project (with the right click "Include to Project" option). However, looking at the parselog.log, it's still parsing the contents of tango's doc subfolder, and still getting into the high CPU usage state.

I'm not sure if it should be parsing files which aren't included in a project... seems weird, or?

So I guess there are kind of two issues here.

aBothe · 2012-05-25T12:38:45Z

First off all: Great that you've been able to find out some hints where the parser is at its limit! :D,

Are you using tango D2 or D1? I guess doc/dil/data only exists in the first version? Then it could be a problem - I've also noticed some problems with T1 some time ago - anyway, Tango for D2 is being parsed flawlessly though.^^

gavin-norman · 2012-05-25T12:51:16Z

Yeah it's tango D1.

I just noticed that while changing the includes from ../tango to ../tango/tango prevents the CPU usage problem, it also stops the Go to Declaration from working. Not sure why this would be.

aBothe · 2012-05-25T12:58:18Z

It's probably because it's not finding a module package called "tango" anymore - it's directly "math.BigInt" instead "tango.math.BigInt" now.

gavin-norman · 2012-05-25T13:00:17Z

Ah true, that's probably why :)

aBothe · 2012-05-25T13:13:02Z

So it's mainly a problem related to tango.. but anyway I guess it'd be better to estimate if a file contains generically invalid data - like it's already skipping 'phobos.d' or 'index.d' in which only html-like stuff is written

aBothe · 2012-05-25T14:03:56Z

:D Just tried to open text.UnicodeData ... 0.1M Lines to parse -- who the hell wrote this!?

gavin-norman · 2012-05-25T14:09:41Z

Haha, ah that classic module ;)

gavin-norman · 2012-06-01T21:36:07Z

I'd say this is a fix... I've not had this problem at all since the update, and it was happening very regularly before.

gavin-norman closed this as completed Jun 1, 2012

gavin-norman mentioned this issue Jun 4, 2012

High CPU usage when parsing tango/doc/dil/data/xml_map.d #105

Closed

Extreme memory leak #101

Extreme memory leak #101

Comments

gavin-norman commented May 14, 2012

gavin-norman commented May 14, 2012

aBothe commented May 14, 2012

gavin-norman commented May 21, 2012

aBothe commented May 21, 2012

gavin-norman commented May 22, 2012

aBothe commented May 22, 2012

gavin-norman commented May 22, 2012

aBothe commented May 22, 2012

gavin-norman commented May 22, 2012

aBothe commented May 22, 2012

gavin-norman commented May 23, 2012

aBothe commented May 23, 2012

gavin-norman commented May 24, 2012

aBothe commented May 24, 2012

gavin-norman commented May 24, 2012

aBothe commented May 24, 2012

gavin-norman commented May 24, 2012

aBothe commented May 24, 2012

gavin-norman commented May 24, 2012

aBothe commented May 24, 2012

gavin-norman commented May 24, 2012

aBothe commented May 24, 2012

gavin-norman commented May 24, 2012

aBothe commented May 24, 2012

gavin-norman commented May 25, 2012

gavin-norman commented May 25, 2012

gavin-norman commented May 25, 2012

aBothe commented May 25, 2012

gavin-norman commented May 25, 2012

aBothe commented May 25, 2012

gavin-norman commented May 25, 2012

aBothe commented May 25, 2012

aBothe commented May 25, 2012

gavin-norman commented May 25, 2012

gavin-norman commented Jun 1, 2012