Salt master stating every single file in files root and follows symlinks

[ryan-lane]

It seems my salt master is stating every single file in my files root even when nothing is referencing them. It does this for every highstate and every also for sync_all. It also follows symlinks, going into recursive hell. I can't run a highstate on more than a few nodes at a time without consuming all CPU on the master and even simple calls take 10-15 minutes.

[ryan-lane]

Using version 0.16.3-1precise on ubuntu precise for the master and mix of lucid and precise for the minions.

[ryan-lane]

     Salt: 0.16.3
     Python: 2.7.3 (default, Apr 10 2013, 06:20:15)
     Jinja2: 2.6
     M2Crypto: 0.21.1
     msgpack-python: 0.1.10
     msgpack-pure: Not Installed
     pycrypto: 2.4.1
     PyYAML: 3.10
     PyZMQ: 13.0.0
     ZMQ: 3.2.2

[basepi]

Is this a change from previous versions of salt or have you recently added many more files to your file roots?

If it's a change, from what version did you upgrade?

[ryan-lane]

This must be a change, as highstate runs didn't make my master fallover previously. I'd imagine it happened in the 0.16 series, but I'm not completely sure.

[basepi]

OK, thanks. We'll get this fixed.

[cachedout]

OK, I've done a lot of digging into this and I'm going to brain-dump my findings here.

Basically, this problem traces back to saltutil.py:L60 where we copy the entire directory structure for a given file root on every call to saltutil's _sync() method (which happens a lot).

At the end of the day, this problem is influenced by the design decision to intermix Salt configuration files with user-files under the same file_root.

As best I've been able to work out, there isn't an easy way to avoid recursing these directories on sync, since we don't artificially limit or track the potential locations of state files.

After talking this over with @basepi, we believe that one possible approach might be to allow a separate directory structure for non-Salt file delivery that wouldn't be evaluated on each sync and each high state.

Regardless, this is a fairly major design decision that needs a larger discussion amongst the Salt developers. We have this listed as a high-prirority issue and hope to come to a decision very soon.

[cachedout]

@thatch45 Ping (We'll try and find you this week or next if you don't see this.)

[cachedout]

@ryan-lane The above PR will allow you to set a flag that should stop the fileserver from recursing into directories when searching for states. Note that you'll have to actually go in and set a flag in the config for this to work.

There are other places inside Salt where we do traverse an entire fileserver root, so let me know to what extent this might solve the issue you're having so we can evaluate whether or not to continue optimizations in this area. Thanks!

[cachedout]

We've had this open for a while now. I'm going to close it for the time being and if it turns out that this issue is truly not resolved, let's re-visit it. Feel free to leave a comment and we'll be happy to re-open. Thanks!