Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
salt-minion, memory and oom-killer with salt-master 2014.7.0 #19999
Since i upgraded my salt-master server (Linux Centos 6.5) from salt 2014.1.13 to 2014.7.0, my minions (Linux, 2014.1.13 and 2014.7.0) are eating memory randomly (10 servers in a few days on up to 200 minions). Windows minions aren't affected.
A few days ago, i downgraded salt-master from 2014.7.0 to 2014.1.13 and the problem disappeared. I didn't see useful information in logs. What kind of information could be useful for you to help debugging ?
Now with the downgraded master:
With the latest master:
On the minion:
I'm using Centos 6.x with EPEL repo.
Fixed Pending Verification
Jan 26, 2015
I upgraded to ZMQ 4 today on my master with this package http://www.itsprite.com/centos-linux-how-to-upgrade-zmq2-x-to-zmq-4-x/
Stay tuned ;)
Feb 15 00:44:43 sisib05 kernel: salt-minion invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
I have ~1800 process salt-minion in stack during the oom-killer call !
I think i will try to disable multiprocessing support on minion.
i found that some of my windows minions are concerned by #19350
It seems that those minions are eating threads on master. After some time, master is out of order.
I upgraded my windows minions from 2014.7.0 to 2014.7.1 to see if it can resolve my linux issues with oom-killer.
A few more oom-killer this night. I saw this log on the master
2015-02-16 20:29:23,114 [salt.client ][ERROR ] Salt request timed out. If this error persists, worker_threads may need to be increased.
I have this conf: