Fast memory leak on ctrl-c out of salt '*' state.highstate

[edgan]

Description of Issue/Question

I run state.highstate and ctrl-c out. There are hosts that are in the list, but down. So they take time to time out. When I ctrl-c memory usage of one of the salt-master processes starts to shoot up very fast, and will out of memory the box.

Versions Report

Salt Version:
Salt: 2016.3.1

Dependency Versions:
cffi: 1.5.2
cherrypy: 6.0.1
dateutil: 2.4.2
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: Not Installed
Jinja2: 2.8
libgit2: 0.24.0
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pygit2: 0.24.0
Python: 2.7.11+ (default, Apr 17 2016, 14:00:29)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: Ubuntu 16.04 xenial
machine: x86_64
release: 4.4.0-28-generic
system: Linux
version: Ubuntu 16.04 xenial

[Ch3LL]

I am able to replicate this behavior. I believe this is related to #33616

Just to clarify this is what I am seeing and please clarify if this is the same behavior.

I run a state.highstate with minions intentionally not running and I have to hit Ctrl+C multiple times to get the highstate to stop. And after every time I select Ctrl+C the memory increases and stays there until I hit Ctrl+C about 3-5 times. Although I should note I have not run into an issue where the box runs out of memory entirely on the box. How much memory is on your box? I have 4Gb .

And here is an example of my memory usage for the salt process:

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                                                                    
13307 root      20   0 1332208 836912   7208 R 98.3 21.6   0:16.88 salt                                                                                               

I did a git bisect and this is shown as the first commit:

200af076db9754fc10a6bc4bdaf9f3ca32d12a3d is the first bad commit
commit 200af076db9754fc10a6bc4bdaf9f3ca32d12a3d
Author: Mike Place <mp@saltstack.com>
Date:   Thu May 5 09:31:09 2016 -0700

    Catch IOLoop RuntimeError on CLI exit (#33070)

    Closes 32899

    Fises

:040000 040000 25e67a0ade3b26887903f4a5a7b3fb56eaace39d f4b3ad5dd8d0341e0aa2ddf39f5e333c7c8fa6a4 M      salt

[Ch3LL]

I also want to note that after I do select Ctrl+C 3-5 times then the memory does go back down though. @edgan do you see hte same behavior?

[edgan]

Yes, ctrl-c more stops it completely. After one ctrl-c it continues to grow continiously till OOM.

I last reproduced it with just "salt '*' cmd.run 'ls'".

My salt master has 7982mb, AWS EC2 instance type t2.large.

[Ch3LL]

@edgan thanks for the additional information. We definitely need to get this fixed up. Thanks

[cachedout]

This is a reactor loop combined with a known issue with a leak in the event system. Please try #34490 for a fix.

[edgan]

Confirmed, that fixes it for me.