CodeIntel code inspector tree crashing

[AtmanActive]

Komodo IDE, version 9.2.0, build 87616, platform win32-x86. Built on Tue Aug 04 10:31:35 2015.

While I'm working on a pretty big (cca 12000 lines) Perl Module file (.pm), the code inspector in the right panel works correctly all of the time.
But, if the file is modified outside of Komodo, and Komodo detects the change and offers to reload the file, after the file is reloaded, the code inspector completely stops working and I need to restart Komodo to get it back to working state again.

[Naatan]

I'm unable to reproduce this with either Perl or Python. Can you share your error log (Help > Troubleshooting > View Log File) after reproducing the issue (without restarting)?

[AtmanActive]

I'm unable to artificially reproduce it either.
I just tried changing the file in another editor and then reloading in Komodo, but it works properly.

But from my workflow it happens every time. I do 'svn update' and then unison-sync to pull in the changes from my coworkers and if anyone touched any of the files that I have open, after a reload, code tree stops working on that file. Every time.
Now I discovered that if I close and reopen the file manually, then code tree starts working again, which is good, at least I don't have to restart the whole IDE just for that.

Here is the log, but unfortunately, I don't see anything in it when this happens (19:45:36).
Most probably it is not a crash but some loop gets stuck in the code tree.
Because that's how it looks, it looks like it will show the output just now (like every time), but then it just freezes there and no tree is shown.

    [2015-09-02 23:34:16,517] [INFO] Startup: Welcome to Komodo IDE 9.2.0 build 87616 (platform win32-x86, running on Windows 7 version 6.1.7601)
    [2015-09-02 23:34:16,517] [INFO] Startup: A:\AtmanActive\Programs\Development\Code\Edit\ActiveState Komodo IDE 9\lib\mozilla\komodo.exe built on Tue Aug 04 10:31:35 2015
    [2015-09-02 23:34:17,138] [WARNING] console-logger: mutating the [[Prototype]] of an object will cause your code to run very slowly; instead create the object with the correct initial [[Prototype]] value using Object.create (1) in resource://gre/modules/Preferences.jsm:381
    [2015-09-02 23:34:17,206] [WARNING] root: ko.logging has been converted to a CommonJS module; use require("ko/logging") instead (since Komodo 9.0.0a1).
        @chrome://komodo/content/komodo.js:15:1

    [2015-09-02 23:34:17,872] [ERROR] console-logger: bad script XDR magic number (0) in :0
    Traceback (most recent call last):
      File "", line 0, in 

    [2015-09-02 23:34:20,279] [DEBUG] editorconfig: Setting editorconfig: {}
    [2015-09-02 23:34:20,878] [WARNING] root: ko.findresults was already loaded, re-creating it.

    [2015-09-02 23:34:22,279] [WARNING] console-logger: Unknown property '-moz-border-radius'.  Declaration dropped. (1) in chrome://docksearch/skin/styles.css:6
    [2015-09-02 23:34:23,111] [WARNING] console-logger: mutating the [[Prototype]] of an object will cause your code to run very slowly; instead create the object with the correct initial [[Prototype]] value using Object.create (1) in file:///A:/AtmanActive/Programs/Development/Code/Edit/ActiveState%20Komodo%20IDE%209/lib/mozilla/components/koamAddonManager.js:138
    [2015-09-02 23:34:23,302] [INFO] komodospellchecker: Spell checker loaded.
    [2015-09-02 23:34:23,302] [INFO] komodospellchecker: Using dictionary en-US
    [2015-09-02 23:41:38,286] [WARNING] console-logger: Key event not available on GTK2: key="u" modifiers="control,shift" (1) in chrome://komodo/content/komodo.xul:0
    [2015-09-02 23:41:38,286] [WARNING] console-logger: Key event not available on some keyboard layouts: key="v" modifiers="control,alt" (1) in chrome://komodo/content/komodo.xul:0
    [2015-09-02 23:42:18,111] [DEBUG] editorconfig: Setting editorconfig: {}
    [2015-09-02 23:43:02,788] [DEBUG] editorconfig: Setting editorconfig: {}
    [2015-09-02 23:48:51,357] [WARNING] codeintel.perl: no Perl was found from which to determine the import path
    [2015-09-02 23:52:47,839] [DEBUG] editorconfig: Setting editorconfig: {}
    [2015-09-03 00:04:14,381] [DEBUG] editorconfig: Setting editorconfig: {}
    [2015-09-03 00:19:05,551] [WARNING] codeintel.perl: no Perl was found from which to determine the import path
    [2015-09-03 19:45:36,260] [DEBUG] editorconfig: Setting editorconfig: {}

[AtmanActive]

Oh, oh, oh, it doesn't have anything to do with file reloading.
The only relevant factor is how many open tabs you have.
Now the code tree is failing for me for every new file opened.
That is why we couldn't reproduce it, because when you want to test it with one or two files, the bug doesn't show it's ugly face.
But now, after working with many, many files for the whole day, without file reloading, the code tree does not work for any new file I load.

[Naatan]

Could you share the full log?

[AtmanActive]

Didn't I do just that?

How can I attach a file?

[AtmanActive]

Also, in this state, where CodeIntel code tree has 'crashed', even for previously loaded files, where there is some tree displayed, the updates are not working. The tree just sits there looking pretty, but no new entries are added and jump-to-subroutine on double click completely misses the line in question.

This time I have to restart the whole IDE, it became unusable.

[AtmanActive]

And it crashed today again:

Komodo_Log_2015-09-16.txt

[AtmanActive]

File close/open does not help in this case.
Restart of the whole IDE is required.

[AtmanActive]

Here it goes again.

This time there is something CRITICAL in logs, at [2015-09-24 20:56:34,755]:
Komodo_Log_2015-09-24.txt

[Defman21]

CodeIntel had reached the maximum of reserved/available RAM for itself and Komodo just killed the CI Driver. I suggest you're trying to use the code inspector in a very big file with a lot of functions, modules, variables, etc. but you don't have enough RAM to let CodeIntel scan your code without hitting the maximum. It could be an optimizing issue or just an use-case, I'm not sure. AFAIK Python ain't very gluttonous to RAM, but if it hits the memory limit, then something strange is going on there.

[AtmanActive]

That was my first thought too, but my RAM usage is nowhere near the maximum.

If only it was that simple.

    A:\>wmic OS get FreePhysicalMemory /Value

    FreePhysicalMemory=7846900

[AtmanActive]

    A:\>wmic OS get TotalVisibleMemorySize /Value

    TotalVisibleMemorySize=16757328

[Defman21]

There could be a limit in Python, CodeIntel or Komodo. I'm not sure if there's a memory limit for Python scripts.
Try to run the following script and show us the output.

import resource
print(resource.getrusage(resource.RUSAGE_SELF)[2])

[mitchell-as]

On Windows, codeintel is limited to about 1GB of memory.

[AtmanActive]

Try to run the following script and show us the output.

How do I do that?

[Defman21]

How do I do that?

You should run the code in the Python environment. But it's not very important now, see the @mitchell-as's answer. I don't know why there is 1GB limit for CodeIntel on Windows.

[AtmanActive]

On Windows, codeintel is limited to about 1GB of memory.

I understand.
Most probably there is a memory leak somewhere, as when I restart Komodo, with same amount of open files, everything continues working normally. Until some tome tomorrow or day after that when it crashes again.

So far I have established that it indeed depends on number of open files and complexity of those files, but it never ever crashes immediately so I can reproduce it, but always after few days of working in such an environment.

Looks like a memory leak to me.

[mitchell-as]

It may not be a leak. Your codeintel database may just be quite large, and require a lot of memory to utilize. If you don't mind, you can try clearing your codeintel cache. Keep in mind it may take some time to rebuild the database for your large files, so codeintel may not give completions right away.

http://community.activestate.com/faq/codeintel-database-fix

[AtmanActive]

Sure, will try that.

So, if I understood correctly, I should delete ALL files and folders under 'AppData/KomodoEdit/9.2/codeintel/db/', right?

[AtmanActive]

Current metrics:

7,789 Files, 68 Folders
Size: 79.3 MB ( 83,177,685 bytes )

from which in 'perl' subfolder:
446 Files, 26 Folders
Size: 5.97 MB ( 6,235,522 bytes )

[AtmanActive]

Done.

New metrics:
7,043 Files, 34 Folders
Size: 67.2 MB ( 70,525,500 bytes )

from which in 'perl' subfolder:
29 Files, 8 Folders
Size: 921 KB ( 943,331 bytes )

Only time will tell... :-)

[mitchell-as]

Thanks, please let us know if/when you see this again.

[AtmanActive]

...and again.

Memory usage as reported by Sysinternals Process Explorer:

---

Virutal Memory:
Private Bytes: 393,408 K
Peak Private Bytes: 504,460 K
Virtual Size: 948,820 K
Page Faults: 2,984,938

Physical Memory:
Working Set: 443,968 K
WS Private: 350,672 K
WS Shareable: 93,260 K
WS Shared: 60,464 K
Peak Working Set: 529,636 K

---

This time log looks weird.
Komodo_Log_2015-09-28.txt

[Naatan]

Any chance the project files are open-source, such that we could try to reproduce the issue with them? If not, any chance you could share a subset of files with us in private, strictly for the purpose of reproducing the issue?

By the way, github lets you attach txt files now.

[AtmanActive]

I'm afraid something like that is not possible.
It is closed source and I'm under an NDA.

The recipe to reproduce is to try to:

• leave Komodo running for as long as possible, don't restart it, don't restart the OS for days and weeks, if possible
• load and close around 20 files randomly each day, or if you really want to provoke it, try loading 100 files at once

I did notice that whenever I have this one big .pm file (cca 13000 lines) open at all times, then the bug happens more frequently, but still, even if I don't load that one for days, it happens sooner or later with other files. So, my impression is that the only relevant factor is how many files Komodo is processing per one session.

Firing up Komodo in the morning, working with few files throughout the day and then closing down Komodo at 5PM will never reproduce this bug.

I'm not complaining here, I can live with having to restart Komodo every few days, I'm just trying to help you guys solve it.

Memory leaks always hit hardest the most demanding use cases.
As always, the harder we push it, the more flaws surface up.

[AtmanActive]

By the way, github lets you attach txt files now.

I can't make that work. GitHub always gives me: "Something went really wrong, and we can't process that file".

[AtmanActive]

Sorry to be a party breaker here, but this issue is still haunting me.

To recap: after working in Komodo for several days and couple of dozen of files opened, edited and closed, CodeIntel code inspector tree just stops working and there is a '[CRITICAL] codeintel.komodo: Out-of-process ran out of memory - killing process' line in the logs

Below is the log file.
Komodo_Log_2015-11-17.txt

[Naatan]

Can you check in your task manager how much memory is being used by Komodo and CodeIntel? ie. check after a few hours of use to get a realistic picture.

Is it plausible that Komodo is in fact using up your system memory? In other words; do you on occasion run out of memory with your system?

[AtmanActive]

Nope.
Never.
16GiB of RAM onboard, current usage: 8.2GiB.

I will check and report Komodo's usage occasionally.

[AtmanActive]

2015-11-17 23:30:00 - komodo.exe - working set: 263,508 K

[AtmanActive]

... and crashed again

2015-11-27 00:33:30 - komodo.exe - working set: 384,000 K

Komodo_Log_2015-11-27.txt

Komodo_Log_2015-11-27_memory_usage.txt

[AtmanActive]

Immediately after the restart:

2015-11-27 00:37:50 - komodo.exe - working set: 229,100 K

[Naatan]

Sorry, I should have specified; codeintel runs in a different process. You were only looking at the komodo.exe process. Try looking for one with "codeintel" in the command. On windows the process name itself will probably just be python.exe.

Apologies for the inconvenience :\

[AtmanActive]

... and crashed again. After the crash, python.exe is nowhere to be found.

So, this will be tricky, I would need to install some reminder to remind me to get working set sizes periodically, as once crashed - it's too late.

Komodo_Log_2015-12-02.txt

[AtmanActive]

Right after the restart:

• komodo.exe: 243,504 K
• python.exe: 136,240 K

[Naatan]

Hmm interesting. You could use a Komodo macro to periodically run tasklist and dump the output to a timestamped file.

Note just checking manually once an hour could already tell you if the python.exe process is indicating some type of memory leak.

[AtmanActive]

• 2015-12-02 23:25:30 - komodo.exe: 277,592 K
• 2015-12-02 23:25:30 - python.exe: 150,996 K

[AtmanActive]

• 2015-12-03 10:55:05 - komodo.exe: 326,128 K
• 2015-12-03 10:55:05 - python.exe: 243,240 K

[Naatan]

That does seem suspicious, but doesn't necessarily mean a memory leak nor does it seem outside the norm. Were you working throughout the night or did this evolve during your computer being idle?

[AtmanActive]

Wouldn't know for sure.
I'll try to log punch in/out states so we can track does it grow on it's own or only when I'm working with it.

[AtmanActive]

Punching out...

• 2015-12-04 01:49:30 - komodo.exe: 385,008 K
• 2015-12-04 01:49:30 - python.exe: 332,032 K

[AtmanActive]

Punching in...

• 2015-12-04 12:30:00 - komodo.exe: 385,308 K
• 2015-12-04 12:30:00 - python.exe:332,032 K

[Naatan]

Well that answers that question. Thanks Atman :) If you dont mind could you check what it says every few hours today?

I really appreciate your willingness to help on us on this, unfortunately it's one of those problems that need a bit of trial and error .. :\

[AtmanActive]

Monday morning reboot.

• 2015-12-07 09:50:00 - komodo.exe: 255,220 K
• 2015-12-07 09:50:00 - python.exe: 87,432 K

[tonyattwood]

Hey!

I just wanted to share with you some pics of my favourite place, it;s so awesome! Here is the link for the pics http://effort.seanscarmardo.com/e4jkdn

In haste, tony.attwood

[tonyattwood]

Hi friend!

That is just a quick note to say thank you, you may find my whole message here http://vufropene.aaahbs.com/e4ticsxf

Yours sincerely, tony.attwood

[mitchell-as]

This should hopefully be fixed in Komodo 11 thanks to its new symbol browser.