Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Anaconda high memory usage #97

Closed
flomertens opened this issue Nov 5, 2013 · 30 comments
Closed

Anaconda high memory usage #97

flomertens opened this issue Nov 5, 2013 · 30 comments
Labels
Jedi os: Linux os: OS X os: Windows performance resolved

Comments

@flomertens
Copy link

I have found that lately anaconda jsonserver is eating memory pretty fast

I have logged the memory usage while working:

Memory usage: http://pastebin.com/XYBSHGvE
jsonserver log: http://pastebin.com/uSi7za36

Tell me if you need more information from me.
@DamnWidget
Copy link
Owner

Hi.

The JsonServer log is empty. Btw, this is too curious because the
jsonserver does not create any data structures in memory it just runs and
fire I never noticed this problem in my own machine.

Which version of the interpreter have you configured?
On Nov 5, 2013 8:52 AM, "flomertens" notifications@github.com wrote:

I have found that lately anaconda jsonserver is eating memory pretty fast

I have logged the memory usage while working:

Memory usage: http://pastebin.com/XYBSHGvE
jsonserver log: http://pastebin.com/index.php?e=1

Tell me if you need more information from me.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/97
.

@DamnWidget
Copy link
Owner

Hi again.

I am sorry but I am afraid that I can't replicate this problem:

http://bpaste.net/show/146925/

I'm marking this as "no replicable".

Some tips:

  1. Make sure you are monitoring the PID of the jsonserver process
  2. Tell us your configuration

Regards.

@flomertens
Copy link
Author

Forget the json log (it is a pain that you can't attach file to github issues!), I have got better. I did the same experiment again:

fmertens@vlb063:$ ps aux| grep json
fmertens 31230 0.0 0.3 50456 14868 ? R 14:14 0:00 python -B /homes/fmertens/.config/sublime-text-3/Packages/Anaconda/anaconda_lib/../anaconda_server/jsonserver.py -p data-analysis 59873 -e /homes/fmertens/Developpement/Workspace/playground,/homes/fmertens/Developpement/Workspace/pywave,/homes/fmertens/Developpement/Workspace/wds,/homes/fmertens/Developpement/Workspace/utils 31210
fmertens 31234 0.0 0.0 14636 944 pts/1 S+ 14:14 0:00 grep --color=auto json
fmertens@vlb063:$ pidstat -r -p 31230 2 1000
Linux 3.2.0-54-generic (vlb063) 05/11/13 x86_64 (2 CPU)

14:14:54 PID minflt/s majflt/s VSZ RSS %MEM Command
14:14:56 31230 546.00 0.00 138928 23368 0.62 python
14:14:58 31230 0.00 0.00 138928 23368 0.62 python
14:15:00 31230 0.00 0.00 138928 23368 0.62 python
14:15:02 31230 0.00 0.00 138928 23368 0.62 python
14:15:04 31230 0.00 0.00 138928 23368 0.62 python
14:15:06 31230 0.00 0.00 138928 23368 0.62 python
14:15:08 31230 0.00 0.00 138928 23368 0.62 python
14:15:10 31230 150.50 0.00 139996 24416 0.65 python
14:15:12 31230 0.00 0.00 139996 24416 0.65 python
14:15:14 31230 0.00 0.00 139996 24416 0.65 python
14:15:16 31230 8983.00 0.00 201300 85848 2.28 python
14:15:18 31230 22604.00 0.00 370004 254500 6.77 python
14:15:20 31230 24033.00 0.00 537612 422180 11.24 python
14:15:22 31230 108.00 0.00 489304 373764 9.95 python
14:15:24 31230 0.00 0.00 489304 373764 9.95 python
14:15:26 31230 23.50 0.00 489304 373764 9.95 python
14:15:28 31230 0.00 0.00 489304 373764 9.95 python
14:15:30 31230 0.00 0.00 489304 373764 9.95 python

the big increase in memory at 14:15:16 correspond to the time I was triggering anaconda_doc for gtk at the line:

import gtk

Python interpreter is the default one in this system, which is python 2.7.3

My user settings:
{
"anaconda_gutter_marks": true,
"anaconda_gutter_theme": "bright",
"pep8_max_line_length": 120,
}

@DamnWidget
Copy link
Owner

I just tried again using GTK and I got the same result than before.

I noticed you have several extra_paths maybe there is something adding rubbish to the global namespace but if you look at the JsonServer code nothing is referenced we just instantiate a new Doc command handler that fire the needed Jedi methods and call the callback (that is self.return_back in JsonServer that just send the data back through the socket).

Can you try to remove your extra paths and try again?

@flomertens
Copy link
Author

Retried on an empty workspace, same result.

First round:
I created a simple file test_anaconda.py, with only the following line:

import gtk

then "Anaconda > Show documentation" on 'gtk'

fmertens@vlb063:$ ps aux| grep json
fmertens 3286 1.1 0.5 134812 19284 ? Sl 15:45 0:00 python -B /homes/fmertens/.config/sublime-text-3/Packages/Anaconda/anaconda_lib/../anaconda_server/jsonserver.py -p anaconda-2 35911 3274
fmertens 3323 0.0 0.0 14636 944 pts/1 R+ 15:46 0:00 grep --color=auto json
fmertens@vlb063:$ 3286^C
fmertens@vlb063:~$ pidstat -r -p 3286 2 1000
Linux 3.2.0-54-generic (vlb063) 05/11/13 x86_64 (2 CPU)

15:46:34 PID minflt/s majflt/s VSZ RSS %MEM Command
15:46:36 3286 0.00 0.00 173760 58364 1.55 python
15:46:38 3286 0.00 0.00 173760 58364 1.55 python
15:46:40 3286 0.00 0.00 173760 58364 1.55 python
15:46:42 3286 0.00 0.00 173760 58364 1.55 python
15:46:44 3286 1433.50 0.00 444416 68984 1.84 python
15:46:46 3286 964.50 0.00 452152 76640 2.04 python
15:46:48 3286 26.00 0.00 452152 76904 2.05 python
15:46:50 3286 1349.00 0.00 462944 87728 2.33 python
15:46:52 3286 1527.00 0.00 475232 99872 2.66 python
15:46:54 3286 1386.00 0.00 486248 110960 2.95 python
15:46:56 3286 1916.00 0.00 501576 126272 3.36 python
15:46:58 3286 1751.50 0.00 515588 140264 3.73 python
15:47:00 3286 2040.50 0.00 531912 156632 4.17 python
15:47:02 3286 2812.50 0.00 554484 179072 4.77 python
15:47:04 3286 4439.00 0.00 589896 214712 5.71 python
15:47:06 3286 7066.00 0.00 657936 271284 7.22 python
15:47:08 3286 12111.50 0.00 731088 355896 9.47 python
15:47:10 3286 12624.50 0.00 807444 432300 11.50 python
15:47:12 3286 0.50 0.00 758160 383096 10.20 python
15:47:14 3286 0.00 0.00 758160 383096 10.20 python
15:47:16 3286 0.00 0.00 758160 383096 10.20 python
15:47:18 3286 0.00 0.00 758160 383096 10.20 python

15:47:18 PID minflt/s majflt/s VSZ RSS %MEM Command
15:47:20 3286 0.00 0.00 758160 383096 10.20 python
15:47:22 3286 0.50 0.00 758160 383128 10.20 python
15:47:24 3286 0.00 0.00 758160 383128 10.20 python
15:47:26 3286 0.00 0.00 758160 383128 10.20 python
15:47:28 3286 0.00 0.00 758160 383128 10.20 python
15:47:30 3286 0.00 0.00 758160 383128 10.20 python
15:47:32 3286 0.00 0.00 758160 383128 10.20 python
15:47:34 3286 0.00 0.00 758160 383128 10.20 python
15:47:36 3286 0.00 0.00 758160 383128 10.20 python
15:47:38 3286 0.00 0.00 758160 383128 10.20 python
15:47:40 3286 0.00 0.00 758160 383128 10.20 python
15:47:42 3286 0.00 0.00 758160 383128 10.20 python
15:47:44 3286 0.00 0.00 758160 383128 10.20 python
15:47:46 3286 0.00 0.00 758160 383128 10.20 python
15:47:48 3286 0.00 0.00 758160 383128 10.20 python
15:47:50 3286 10.00 0.00 758160 383128 10.20 python

I then closed sublime-text, restarted it, monitor again and asked again the doc of gtk:

fmertens@vlb063:$ ps aux| grep json
fmertens 3286 1.1 0.5 134812 19284 ? Sl 15:45 0:00 python -B /homes/fmertens/.config/sublime-text-3/Packages/Anaconda/anaconda_lib/../anaconda_server/jsonserver.py -p anaconda-2 35911 3274
fmertens 3323 0.0 0.0 14636 944 pts/1 R+ 15:46 0:00 grep --color=auto json
fmertens@vlb063:$ 3286^C
fmertens@vlb063:~$ pidstat -r -p 3286 2 1000
Linux 3.2.0-54-generic (vlb063) 05/11/13 x86_64 (2 CPU)

15:46:34 PID minflt/s majflt/s VSZ RSS %MEM Command
15:46:36 3286 0.00 0.00 173760 58364 1.55 python
15:46:38 3286 0.00 0.00 173760 58364 1.55 python
15:46:40 3286 0.00 0.00 173760 58364 1.55 python
15:46:42 3286 0.00 0.00 173760 58364 1.55 python
15:46:44 3286 1433.50 0.00 444416 68984 1.84 python
15:46:46 3286 964.50 0.00 452152 76640 2.04 python
15:46:48 3286 26.00 0.00 452152 76904 2.05 python
15:46:50 3286 1349.00 0.00 462944 87728 2.33 python
15:46:52 3286 1527.00 0.00 475232 99872 2.66 python
15:46:54 3286 1386.00 0.00 486248 110960 2.95 python
15:46:56 3286 1916.00 0.00 501576 126272 3.36 python
15:46:58 3286 1751.50 0.00 515588 140264 3.73 python
15:47:00 3286 2040.50 0.00 531912 156632 4.17 python
15:47:02 3286 2812.50 0.00 554484 179072 4.77 python
15:47:04 3286 4439.00 0.00 589896 214712 5.71 python
15:47:06 3286 7066.00 0.00 657936 271284 7.22 python
15:47:08 3286 12111.50 0.00 731088 355896 9.47 python
15:47:10 3286 12624.50 0.00 807444 432300 11.50 python
15:47:12 3286 0.50 0.00 758160 383096 10.20 python
15:47:14 3286 0.00 0.00 758160 383096 10.20 python
15:47:16 3286 0.00 0.00 758160 383096 10.20 python
15:47:18 3286 0.00 0.00 758160 383096 10.20 python

15:47:18 PID minflt/s majflt/s VSZ RSS %MEM Command
15:47:20 3286 0.00 0.00 758160 383096 10.20 python
15:47:22 3286 0.50 0.00 758160 383128 10.20 python
15:47:24 3286 0.00 0.00 758160 383128 10.20 python
15:47:26 3286 0.00 0.00 758160 383128 10.20 python
15:47:28 3286 0.00 0.00 758160 383128 10.20 python
15:47:30 3286 0.00 0.00 758160 383128 10.20 python
15:47:32 3286 0.00 0.00 758160 383128 10.20 python
15:47:34 3286 0.00 0.00 758160 383128 10.20 python
15:47:36 3286 0.00 0.00 758160 383128 10.20 python
15:47:38 3286 0.00 0.00 758160 383128 10.20 python
15:47:40 3286 0.00 0.00 758160 383128 10.20 python
15:47:42 3286 0.00 0.00 758160 383128 10.20 python
15:47:44 3286 0.00 0.00 758160 383128 10.20 python
15:47:46 3286 0.00 0.00 758160 383128 10.20 python
15:47:48 3286 0.00 0.00 758160 383128 10.20 python
15:47:50 3286 10.00 0.00 758160 383128 10.20 python

json server log for the two run:
2013-11-05 15:45:40,532: DEBUG : bind: address=('localhost', 35911)
2013-11-05 15:45:40,532: DEBUG : listen: backlog=5
2013-11-05 15:45:40,533: INFO : Anaconda Server started in port 35911 for PID 3274 with cache dir /homes/fmertens/.cache/jedi/anaconda-2
2013-11-05 15:45:40,537: INFO : Incomming connection from ('127.0.0.1', 57975)
2013-11-05 15:45:40,538: INFO : Incomming connection from ('127.0.0.1', 57976)
2013-11-05 15:45:40,726: INFO : client requests: run_linter
2013-11-05 15:45:41,571: INFO : client requests: run_linter
2013-11-05 15:46:17,585: INFO : client requests: run_linter
2013-11-05 15:46:18,755: INFO : client requests: autocomplete
2013-11-05 15:46:20,365: INFO : client requests: autocomplete
2013-11-05 15:46:20,368: INFO : client requests: autocomplete
2013-11-05 15:46:20,374: INFO : client requests: autocomplete
2013-11-05 15:46:20,380: INFO : client requests: autocomplete
2013-11-05 15:46:20,382: INFO : client requests: autocomplete
2013-11-05 15:46:22,778: INFO : client requests: autocomplete
2013-11-05 15:46:22,857: INFO : client requests: autocomplete
2013-11-05 15:46:22,928: INFO : client requests: autocomplete
2013-11-05 15:46:23,006: INFO : client requests: run_linter
2013-11-05 15:46:23,009: INFO : client requests: run_linter
2013-11-05 15:46:35,977: INFO : client requests: run_linter
2013-11-05 15:46:43,656: INFO : client requests: doc
2013-11-05 15:47:16,478: INFO : client requests: run_linter
2013-11-05 15:47:20,675: INFO : process 3274 does not exists stopping server...
2013-11-05 15:47:21,677: INFO : Closing the socket, server will be shutdown now...
2013-11-05 15:48:32,228: DEBUG : bind: address=('localhost', 37977)
2013-11-05 15:48:32,228: DEBUG : listen: backlog=5
2013-11-05 15:48:32,229: INFO : Anaconda Server started in port 37977 for PID 3435 with cache dir /homes/fmertens/.cache/jedi/anaconda-2
2013-11-05 15:48:32,230: INFO : Incomming connection from ('127.0.0.1', 46059)
2013-11-05 15:48:32,230: INFO : Incomming connection from ('127.0.0.1', 46060)
2013-11-05 15:48:32,664: INFO : client requests: run_linter
2013-11-05 15:49:07,985: INFO : client requests: run_linter
2013-11-05 15:49:10,851: INFO : client requests: run_linter
2013-11-05 15:49:19,750: INFO : client requests: doc
2013-11-05 15:49:24,621: INFO : client requests: run_linter
2013-11-05 15:49:34,375: INFO : process 3435 does not exists stopping server...
2013-11-05 15:49:35,383: INFO : Closing the socket, server will be shutdown now...

Version of pygtk (but I don't think it has anything to do with this issue)

gtk.pygtk_version
(2, 24, 0)

@DamnWidget
Copy link
Owner

Ok, lets try something else.

Edit your anaconda configuration with this:

"jsonserver_debug": true, "jsonserver_debug_port: 9999

Go to $HOME/.config/sublime-text-3/Packages/Anaconda and start the JsonServer manually with:

python -B ./anaconda_server/jsonserver.py -p debug 9999 DEBUG

Now restart your sublime text 3 and anaconda will now connect to the DEBUG server, hopefully we will get more information about what is happening.

@flomertens
Copy link
Author

I did that on my home system which is a bit different (ubuntu 13.10, python 2.7.5), with same result.
Note that my sublime text configuration are sync between the 2 system.

  1. Run

    python -B ./anaconda_server/jsonserver.py -p debug 9999 DEBUG &> ~/jsonserver.log

  2. Monitor

    ps aux |grep json
    flo 7225 0.6 0.3 57936 19176 pts/2 S+ 18:16 0:00 python -B ./anaconda_server/jsonserver.py -p debug 9999 DEBUG
    pidstat -r -p 7225 2 1000

  3. Start sublime text on empty project with single file containing single line (import gtk)

  4. Call doc on gtk

Results:

Memory monitor:
Linux 3.11.0-12-generic (ubuntu) 05/11/13 x86_64 (8 CPU)

18:17:24 UID PID minflt/s majflt/s VSZ RSS %MEM Command
18:17:26 1000 7225 0.00 0.00 57936 19176 0.33 python
18:17:28 1000 7225 0.00 0.00 57936 19176 0.33 python
18:17:30 1000 7225 0.00 0.00 57936 19176 0.33 python
18:17:32 1000 7225 0.00 0.00 57936 19176 0.33 python
18:17:34 1000 7225 0.00 0.00 57936 19176 0.33 python
18:17:36 1000 7225 5.00 0.00 57936 19176 0.33 python
18:17:38 1000 7225 0.00 0.00 57936 19176 0.33 python
18:17:40 1000 7225 37.00 0.00 58096 19356 0.33 python
18:17:42 1000 7225 0.00 0.00 58096 19356 0.33 python
18:17:44 1000 7225 0.00 0.00 58096 19356 0.33 python
18:17:46 1000 7225 0.00 0.00 58096 19356 0.33 python
18:17:48 1000 7225 0.00 0.00 58096 19356 0.33 python
18:17:50 1000 7225 0.00 0.00 58096 19356 0.33 python
18:17:52 1000 7225 2088.50 0.00 70744 32132 0.55 python
18:17:54 1000 7225 7606.00 0.50 412768 82280 1.40 python
18:17:56 1000 7225 1116.50 0.00 421520 91256 1.55 python
18:17:58 1000 7225 3162.00 0.00 446952 116336 1.98 python
18:18:00 1000 7225 3614.00 0.00 475964 145376 2.48 python
18:18:02 1000 7225 4474.00 0.00 511520 181280 3.09 python
18:18:04 1000 7225 4775.50 0.00 549960 219296 3.73 python
18:18:06 1000 7225 19612.00 0.00 694284 364012 6.20 python
18:18:08 1000 7225 11832.00 0.00 715220 385000 6.56 python
18:18:10 1000 7225 0.00 0.00 715220 385000 6.56 python
18:18:12 1000 7225 0.00 0.00 715220 385000 6.56 python
18:18:14 1000 7225 0.00 0.00 715220 385000 6.56 python
18:18:16 1000 7225 0.00 0.00 715220 385000 6.56 python
18:18:18 1000 7225 0.00 0.00 715220 385000 6.56 python
18:18:20 1000 7225 0.00 0.00 715220 385000 6.56 python
18:18:22 1000 7225 0.00 0.00 715220 385000 6.56 python
18:18:24 1000 7225 0.00 0.00 715220 385000 6.56 python
18:18:26 1000 7225 0.00 0.00 715220 385000 6.56 python
18:18:28 1000 7225 0.00 0.00 715220 385000 6.56 python
18:18:30 1000 7225 0.00 0.00 715220 385000 6.56 python
18:18:32 1000 7225 0.00 0.00 715220 385000 6.56 python
18:18:34 1000 7225 0.00 0.00 715220 385000 6.56 python
18:18:36 1000 7225 0.00 0.00 715220 385000 6.56 python
18:18:38 1000 7225 0.00 0.00 715220 385000 6.56 python

(note that I have 6Go of RAM on my home system vs 2Go on my work system)

Jsonserver log:
DEBUG MODE
About push back to ST3: {"errors": [{"regex": "(^|\s+|,\s_|as\s+)(?P[\w]gtk[\w]*)", "linematch": "(from\s+[\w.]+\s+)?import\s+(?P[^#;]+)", "level": "W", "raw_error": "'gtk' imported but unused", "pep8": false, "lineno": 1, "offset": 0, "message": "%r imported but unused"}], "uid": "d1f5ff362364433e9d3e9134e98af2ed", "success": true}

About push back to ST3: {"doc": "No docstring for \n-------------------------------------------------------------------------------\nNo docstring for ", "uid": "8ce7bd22ecab4f3eb061cdfb8d530e8b", "success": true}

About push back to ST3: {"errors": [{"regex": "(^|\s+|,\s_|as\s+)(?P[\w]gtk[\w]*)", "linematch": "(from\s+[\w.]+\s+)?import\s+(?P[^#;]+)", "level": "W", "raw_error": "'gtk' imported but unused", "pep8": false, "lineno": 1, "offset": 0, "message": "%r imported but unused"}], "uid": "727435c072834b22a40e6f707dee1b53", "success": true}

About push back to ST3: {"errors": [{"regex": "(^|\s+|,\s_|as\s+)(?P[\w]gtk[\w]*)", "linematch": "(from\s+[\w.]+\s+)?import\s+(?P[^#;]+)", "level": "W", "raw_error": "'gtk' imported but unused", "pep8": false, "lineno": 1, "offset": 0, "message": "%r imported but unused"}], "uid": "1806b3baf1604f28bb27cf091dca5f7c", "success": true}

About push back to ST3: {"errors": [{"regex": "(^|\s+|,\s_|as\s+)(?P[\w]gtk[\w]*)", "linematch": "(from\s+[\w.]+\s+)?import\s+(?P[^#;]+)", "level": "W", "raw_error": "'gtk' imported but unused", "pep8": false, "lineno": 1, "offset": 0, "message": "%r imported but unused"}], "uid": "2d620bd56ea044a1afb483ff55e85aaf", "success": true}

@DamnWidget
Copy link
Owner

I still can't reproduce that.

Even in your case, that call to the doc function on GTK returns nothing:

About push back to ST3: {"doc": "No docstring for \n-------------------------------------------------------------------------------\nNo docstring for ", "uid": "8ce7bd22ecab4f3eb061cdfb8d530e8b", "success": true}

In my case it returns nothing and the memory consumption remains the same.

I will let you know if I am able to replicate the problem in my work machine tomorrow (Ubuntu 13.10)

Regards.

@DamnWidget
Copy link
Owner

Good news @flomertens finally I reproduced it (not with this huge amount of memory of yours).

Will be solved in the next commit. Thank you for your feedback this was a really hidden bug in the JsonServer.

Regards.

@DamnWidget
Copy link
Owner

I have good and bad news.

First the good one: There is no memory leak in anaconda
The bad one: Is the normal behavior in Jedi

When you import gtk and run any jedi related feature (autocompletion, goto, find usages, documentation), Jedi load the whole library and cache it on memory. Seems that can't be configured at all.

I added a call to the Jedi clean cache library after each command is executed, it will delete any "unused" cache, seems like the impact is not too much but I see better numbers in my pidstat.

The really good new is it doesn't grow more (really seems like gtk import everything in the library when you just do import gtk, that is a really bad design, PySide for instance does not have this behavior).

I'm sorry to can't be more helpful here.

DamnWidget added a commit that referenced this issue Nov 5, 2013
@DamnWidget
workaround to try to mitigate #97
c94119d
@highend
Copy link

highend commented Jan 2, 2014

I'm using Build 3061 (x86) on Windows Server 2012 R2 Datacenter.
I use many plugins and I wondered for months what causes the constant memory "drain".
Today I decided to take (at least) a look, so I've disabled a few plugins (the bigger ones) to see if it's more a ST issue or really a plugin one.

I've disabled four at the beginning (Anaconda, Trailing Spaces, Bracket Highlighter and Vintagous) and watched the plugin_host.exe in the taskmanager closely. Memory consumption wasn't increasing any more...
Firing up ST leads to about ~16MB for this process. Now I've reenabled the other three one by one to see if the memory consumption begins to grow. Nope, still staying at 16,9Mb, even after 10 minutes.

But reenabling Anaconda let's the plugin_host begin to grow. Not every second but about every 12 seconds the process claims 300Kb more. It never stops as long Anaconda is running. So even if I don't use ST at all (but it's running in the background) the plugin_host process will consume more and more memory. It easily reaches 2GB after 24 hours.

The curious thing... This happens even if there are only 3 open files (and all of them are .txt). No .php, nothing else in that direction...

I'll deliver as much information as I can if you like to describe how I get them (under Windows) :)

Regards,
Highend

@DamnWidget
Copy link
Owner

Hi @highend

Maybe there is some memory leak on anaconda that only affects Windows cos I have running it for weeks on my Linux boxes with no problems.

Let's try to get some information. Anaconda uses external process that do all the hard work. The process should be identified in Windows as a regular python.exe process, this is the process that lint autocomplete and do everything else.

The plugin host should be really just doing really light stuff.

Can you monitor the anaconda python process as well?
Can you disable/uninstall everything else and check again?
Can you take a look at %APP_DATA%/jedi/jedi/your_project_name/anaconda_jsobserver.log?
Can you take a look at your ST3 console logs?

Regards

@highend
Copy link

highend commented Jan 3, 2014

Hi @DamnWidget

  • Under Windows all plugins are encapsulated in the plugin_host process and because there are only .txt files no python.exe process is running / started.
  • I've downloaded a fresh 3061 x86 portable version and used it only with Anaconda (ofc latest commit), no configurations were made so far -> Default settings ST + Anaconda.
  • There is no jedi folder inside the %APPDATA% one ;(
    Do I have to enable the debug option for that?
  • There are no further entries in the ST3 console log after initally loading the the plugins

Last lines

reloading plugin Anaconda.anaconda
loaded 837 snippets
Persisten list does not exists, skiping load...
plugins loaded

The "Persisten list" is the only one that belongs to Anaconda that looks "unusual". I get the same message on my normal installation, too.

Regards

@DamnWidget
Copy link
Owner

Did you noticed the same memory drain on this portable version with just anaconda installed on it?

@highend
Copy link

highend commented Jan 3, 2014

Yepp, that's what I wanted to say with the second list item :)

So there is definitely a memory "leak" inside Anaconda even if it's only present under a MS Windows system. Btw, whenever Anaconda allocates a new ~300KB block the process (plugin_host) activity jumps from 0,0% to 0,4-0,7%. It's doing something but I don't know what ;)

Time for bed now, I'll answer any new questions in about 8 hours.

@DamnWidget DamnWidget mentioned this issue Jan 4, 2014
Closed
@DamnWidget
Copy link
Owner

Hi guys.

I've been working on that but I am not able to replicate this problem myself, I am using Windows 7 x64 with ST3 3059 and the last version of Anaconda and my plugin_host.exe process does not grow at all.

I test it with both portable and installable versions and both 32 and 64 bits. I am running Package Control and Anaconda with no more plugins.

I have to be able to replicate this behaviour in order to fix it but I am not so I need more information.

Is the memory usage growing in a consistent way?
What are you doing when memory usage grows?
It grows even if you are not using the ST3 editor at all?
Anything else that you can think about?

@cloudysapa
Copy link

  • Is the memory usage growing in a consistent way?
    Yes. On fresh start, the memory of plugin_host.exe grows pretty fast with an increment of 64 or 128KB for every second. After a while (around 1 minute), it steadily increases with an increment of 256KB for every 10 seconds.
  • What are you doing when memory usage grows?
    Nothing. Just open sublime text with no tab... no other plugins installed.
  • It grows even if you are not using the ST3 editor at all?
    Right.
  • Anything else that you can think about?
    I have no idea. However, with this pattern of memory growth, I think the leaking may occur in a loop.

@highend
Copy link

highend commented Jan 5, 2014

I have absolutely the same behavior like cloudysapa.
256kb / ~10 seconds

Only ST3, only Anaconda, no Package Control or any other additional addons.

ST3 has no open buffers, TaskManager in front of ST3 (no other apps are running).

ST3 3061 x86 portable
My main OS: Windows Server 2012 R2 (x64, de-DE)

I've now tested it in a virtual machine: Windows 7 Enterprise, x86 de-DE
to be sure it's not some kind of problem between a 64-bit OS and the 32-bit application.
It's doing absolutely the same (plugin_host.exe is growing without any files open or any other plugins (apart from Anaconda) installed)

I guess it wouldn't help if I record a video of it, right?

@DamnWidget
Copy link
Owner

Hi

I detected some sort of memory growing on plugin_host related with the linting part of the plugin (that is in big part inherited code from Sublime Linter itself) but I am not able to replicate the memory growing with no open views.

I will work on this problem until it get fixed, I will come here eventually to ask you to update anaconda version and test the bug that I can't replicate myself.

Regards.

@highend
Copy link

highend commented Jan 5, 2014

Whenever there is an update, I'll be there to report if it changes anything...

Regards,
Highend

@DamnWidget
Copy link
Owner

Hi again.

I have two news, one is good and the another one is really bad.

Let's start with the good one. I found the memory leak.
The bad one is that the memory leak is in Sublime Text itself so there is nothing that we can do to solve it (at least without a lot of pain and suffering).

Please, follow the following steps to confirm that I am right:

  1. Remove any plugin in your ST3
  2. Close ST3
  3. Open ST3
  4. Start monitoring the RAM used by plugin_host
  5. Open the ST3 console
  6. Write the following line and press enter: [sublime.active_window().active_view() for i in range(100000)]
  7. Your memory usage will grow like 30M
  8. Let me know if I am right (I know I am)

Regards

@highend
Copy link

highend commented Jan 5, 2014

8). You're right...

About 20 MB on my system but I got the point. Will you forward the bug to jps?

@DamnWidget
Copy link
Owner

Yes, but I am investigating deeper about the problem, seems like if you run the garbage collector manually, it get collected almost all the new allocated memory but you have to collect it manually (and it doesn't collect all the memory, some kilobytes are still leaked)

DamnWidget added a commit that referenced this issue Jan 6, 2014
@DamnWidget
Workaround to mitigate the severe memory leaks that plugin_host suffe…
47ce3dd 
…r when objects are shared betwen threads. In the API doc of ST3 they said those _async methods are thread safe, but the objects that are created in local scopes into that functions are not being collected by the ST3 embedded python garbage collector and severe memory leaks appears much more often if your plugin is hardly based on asynchronous events (as anaconda is). This meant to solve (or at least strongly mitigate) the #97 issue
@DamnWidget
Copy link
Owner

Hi guys.

I just refactored critical parts of the plugin. Seems like ST3 leaks memory when the "thread safe" _async API methods are used or you just run code in a separate thread created with the standard threading library. The garbage collector of the embedded ST3 Python interpreter is not able to collect variables that are out of scope in local frames created on those threads.

That affected anaconda quite seriously as the plugin is totally event driven and asynchronous, I had to modify really critical parts of the plugin that are related with the speed and performance of the linting process. I still feel the plugin much more faster than other plugins out there but we lost some speed in the process.

Please, take a look at the last commit and report anything that doesn't work (I use auto-completion and linting only so I didn't tested every anaconda feature).

I also tested the changes on my Linux box only so is quite possible other implementations are partially break.

Let me know how you feel the plugin now.

Regards.

@cloudysapa
Copy link

Yeahhhh, I confirm that the memory leak issue has been fixed. Code completion and linting work great. I'm not sure how "rename object under cursor" works. It seems not working. I tried to rename a method of a class but nothing happened.

@DamnWidget
Copy link
Owner

Rename feature never worked quite well so nevermind it

@highend
Copy link

highend commented Jan 6, 2014

The latest commit fixes the memory leak for me as well :)

Thanks a lot, Oscar!

@xuyu
Copy link

xuyu commented Jan 13, 2014

I'm so happy to read this comments. I have the same issue before.
i update the code just now, it's been fixed!!!

@DamnWidget DamnWidget mentioned this issue Mar 23, 2014
Closed
@rahul-ramadas rahul-ramadas mentioned this issue May 4, 2014
Closed
@SmartManoj
Copy link

Still

@SmartManoj
Copy link

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Jedi os: Linux os: OS X os: Windows performance resolved
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@DamnWidget @highend @xuyu @flomertens @cloudysapa @SmartManoj