Cookbook loading is inefficient with large ignored directories

[acoulton]

Description

The CookbookVersionLoader recurses into every directory and loads a full list of files before iterating back over to remove any that match the chefignore.

This causes very slow cookbook loading when there are entirely-ignored directories with large numbers of files. For example, the presence of build-time gems with a vendors/* ignore pattern produces very slow chefspec performance.

It would be better if directories and files were tested against the ignore during recursion, so that the vendors directory could be skipped outright if it matches a pattern that excludes the whole directory. Note that fixing this won't take effect until #2994 is also fixed, as at present cookbook-level chefignores are completely ignored by the cookbook loader.

Chef Version

Verified in Chef 12.21.1 and 13.2.20

Platform Version

Ubuntu 14.04

Replication Case

I have created a self-executing replication case in this gist - download to a path with at least one empty parent directory to avoid chef attempting to load siblings as cookbooks and execute.

• Create a simple cookbook with chefspec setup and a single empty recipe
• Create a chefignore containing vendor/* in the cookbook root directory
• Create a minimal spec with two specs, each just triggering converge on the empty recipe
• Run chefspec and observe the runtime is very short
• Populate a large number (10k or so) of recursive files / directories into {cookbook}/vendor equivalent to running bundle install --deployment with a chef / chefspec gemfile
• Run chefspec and observe that each spec now takes multiple seconds to run

Output

From my replication case - observe that once vendor is present the run is around 180 times slower at 11.6 rather than 0.06 seconds. The presence of a chefignore has no effect.

Installing gems
With no local vendors
..

Finished in 0.06462 seconds (files took 0.96274 seconds to load)
2 examples, 0 failures


real    0m1.307s
user    0m1.104s
sys     0m0.163s

With local vendors and no chefignore - expected to be slow
..

Finished in 11.59 seconds (files took 0.93044 seconds to load)
2 examples, 0 failures


real    0m12.797s
user    0m11.916s
sys     0m0.710s

With local vendors chefignored - should be faster
..

Finished in 11.58 seconds (files took 0.98431 seconds to load)
2 examples, 0 failures


real    0m12.841s
user    0m12.017s
sys     0m0.598s

[lamont-granquist]

#7794 and #7820 were some major cleanup done recently for Chef-15.

[lamont-granquist]

looks like this is still unfixed after the cleanup