Slow/sub-optimal file lookup

[pch]

Hi,

First of all I'm not sure it's the right place to post this. I'm not even sure if it's actually a bug or simply the way asset pipeline works. Anyway, here's the problem:

In the project I'm working on, asset compilation has been taking a huge amount of time (1-2mins). Since we're using @import 'compass' rather than sprockets' require, whenever an SCSS file changes, rails recompiles everything - including compass. It's a fairly large projects with a bunch of dependencies, but still 1-2minutes is quite a significant amount of time. I've been trying to track down why it's taking so long - so far with little success.

However, when I attached dtruss to the rails process, it revealed that a considerable chunk of that time is spent on looking for compass files. It basically checks for compass files in every possible asset directory in load_path.

To give you an idea how it looks:

93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/images/_compass.css.sass\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/images/compass.css.sass\0", 0x7FFF595C4768, 0x7FFF595C17F0)       = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/images/_compass.css.sass.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/images/compass.css.sass.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)       = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/images/_compass.css.scss\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/images/compass.css.scss\0", 0x7FFF595C4768, 0x7FFF595C17F0)       = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/images/_compass.css.scss.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/images/compass.css.scss.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)       = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/images/_compass.sass\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/images/compass.sass\0", 0x7FFF595C4768, 0x7FFF595C17F0)       = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/images/_compass.sass.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/images/compass.sass.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)       = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/images/_compass.scss\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/images/compass.scss\0", 0x7FFF595C4768, 0x7FFF595C17F0)       = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/images/_compass.scss.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/images/compass.scss.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)       = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/_compass.css\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/compass.css\0", 0x7FFF595C4768, 0x7FFF595C17F0)       = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/_compass.css.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/compass.css.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)       = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/_compass.css.sass\0", 0x7FFF595C4768, 0x7FFF595C17F0)         = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/compass.css.sass\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/_compass.css.sass.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)         = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/compass.css.sass.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/_compass.css.scss\0", 0x7FFF595C4768, 0x7FFF595C17F0)         = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/compass.css.scss\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/_compass.css.scss.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)         = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/compass.css.scss.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/_compass.sass\0", 0x7FFF595C4768, 0x7FFF595C17F0)         = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/compass.sass\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/_compass.sass.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)         = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/compass.sass.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/_compass.scss\0", 0x7FFF595C4768, 0x7FFF595C17F0)         = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/compass.scss\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/_compass.scss.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)         = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/javascripts/compass.scss.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/plugins/_compass.css\0", 0x7FFF595C4768, 0x7FFF595C17F0)      = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/plugins/compass.css\0", 0x7FFF595C4768, 0x7FFF595C17F0)       = -1 Err#2
93963/0xab888:  lstat64("/Users/piotrek/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/mediaelement_rails-0.8.0/app/assets/plugins/_compass.css.erb\0", 0x7FFF595C4768, 0x7FFF595C17F0)

Here's the whole (filtered) output from dtruss (around 2.8mb, 15215 calls to lstat64 just for compass files): https://gist.github.com/pch/f41bb68316339877faad

Is this a bug/something that can be improved?

We're running Rails 4.2 (recently upgraded from 4.0.x). I have yet to do a comparison with the old version, but the problem got bigger after the upgrade.

[mleszcz]

+1