Fabric very high memory usage #800
Comments
|
A few things:
Anyway -- given the unusual use case and my own admittedly weak skills in diagnosing RAM utilization, I'd have to ask you to do a few things before I can take any action:
Thanks for the report & your understanding :) Interested to see what you find. |
|
I'm suffering from this problem as well. I use fabric to assist me with development and debug on an embedded device, and I'm currently running long, verbose test runs and redirecting fab output to file. Thismorning, the size of the file is 621MB, and the RSS of the fab process is 5.5GB. Bitprohpet, could you give me some background on why fab currently stores all of stdout+stderr? |
|
@dunkpa It's a core feature -- check the API docs for run/sudo. An enormous amount of Fabric usage revolves around doing things with returned stdout/stderr text blocks. Thanks for the note re: your on-disk output + process usage. I'm assuming you aren't doing any output hiding (using If so, then that definitely sounds like a leak of some kind; if you have the time to poke around and see if there's anything that isn't being freed/ Your use case isn't the most common so I can't prioritize my own time for this right now :( though I will absolutely expedite merging of fixes should any be identified & proven. Thanks! |
|
I'm having a similar issue. I'm using fabric to start/stop scripts on many servers. Each server outputs a lot of information. Any ideas or suggestions? |
|
I'd +1 a PR that implements some sort of circular buffer (instead of the current |
|
After reading #1195 and thinking about ramifications, I think what we need is the following:
I will write a patchset for this momentarily. |
|
Looks like we already wrote a So reusing that class for our capture buffers ought to work well. (And in Invoke/Fab 2, I can just use |
|
Glad I decided to explicitly test the memory usage myself...even with this implemented we were still chewing crazy memory. Why? Organic code growth left us with an unused extra buffer object :( used in the past to track things like newlines & prefix printing, and never deleted by whomever overhauled that state-keeping functionality. That would have been exacerbating this problem for users (i.e. by about 2x). With that also removed, using this new To wit:
To get this working I had to enhance |
I have a fabric task I often use to reindex my solr index from a 4 GB mysql database. The process takes around 10 hours to complete and is very verbose. I invoke the fab task via Jenkins so I can capture the output nicely. I noticed today while running the reindex that the fab process took 2.7 GB of my system memory!
Fabric 1.4.3
ssh (library) 1.7.14
Python 2.7.3
Linux jenkins 2.6.32-24-server #43-Ubuntu SMP Thu Sep 16 16:05:42 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.1 LTS
From top
cat /proc/16816/status
cat /proc/16816/pmap
cat /proc/16816/smaps
The text was updated successfully, but these errors were encountered: