Simple Guard Script Running at 100%+ CPU usage

[jwhitcraft]

I have this guard script

require 'pp'

guard :shell do
    watch /sugarcrm\/.+$/ do |file|
        pp file
    end
end

And when I run this command:
bundle exec Guard -w /Users/jwhitcraft/Projects/Mango

the ruby process takes up 100% on a CPU in my activity monitor.

The folder i'm watching has 18531 files in it, but if i change it to one that doesn't contain that many files, the issue goes away.

My gem file looks like this

# Gemfile
source 'https://rubygems.org'

gem 'rake'

gem 'guard', '~> 2.3'
gem 'guard-shell', '~> 0.6'
gem 'listen', '~> 2.0'

I've tracked it down to being listen as the problem as when i just run listen, it does the same thing

listener = Listen.to('/Users/jwhitcraft/Projects/Mango/sugarcrm') do |modified, added, removed|
  puts "modified absolute path: #{modified}"
  puts "added absolute path: #{added}"
  puts "removed absolute path: #{removed}"
end
listener.start # not blocking
sleep

[thibaudgg]

Yeah Listen doesn't support watching that amount of files, try to ignore as much as you can with the ignore method of Guard.

[jwhitcraft]

@thibaudgg, thanks for the info, but i can't exclude that many files, as they are 90% php and 10% JS and Less files.

I need them all to be built out, it's for our software that the company i work for builds.

Just some general FYI, we have a version of listen running just find on our webapp. Here is the gem file from it.

# Gemfile

source 'https://rubygems.org'

gem 'rake'

group :development do
  platforms :ruby do
    gem 'rb-fsevent', '~> 0.9.3', :require => false
  end

  gem 'terminal-notifier-guard', :require => false
  gem 'growl', :require => false

  gem 'guard', '1.4.0'
  gem 'guard-shell', '0.5.1'
  gem 'listen', '0.7.0'
  gem 'rainbow'
  gem 'git', :git => 'git@github.com:jwhitcraft/ruby-git.git'
  gem 'addressable'
  gem 'configliere'
end

Is there any reason why the older version of guard + listen works just fine? If I go higher than version 0.7.0 it just spikes the CPU usage.

[thibaudgg]

How many php & js files do you have?

[jwhitcraft]

jwhitcraft@jwhitcraft-mbpro ~/Projects/Mango/sugarcrm
↪ find . | grep '.php' | wc -l
6432

jwhitcraft@jwhitcraft-mbpro ~/Projects/Mango/sugarcrm
↪ find . | grep '.js' | wc -l
2176

jwhitcraft@jwhitcraft-mbpro ~/Projects/Mango/sugarcrm
↪ find . | grep '.less' | wc -l
191

jwhitcraft@jwhitcraft-mbpro ~/Projects/Mango/sugarcrm
↪ find . | grep '.hbs' | wc -l
438

jwhitcraft@jwhitcraft-mbpro ~/Projects/Mango/sugarcrm
↪ find . | grep '.json' | wc -l
32

There is a better break down of all the files types that i need to watch. Is there a away with listen to say just look for php, js, less, hbs and json files?

[thibaudgg]

Even if you listen only these files, that's still a lot, and because Listen keeps the md5 checksum of each modified files it quite slow with so much files.
Listen would need some improvements on that side, but I've not much planned time for it right now.

[jwhitcraft]

@thibaudgg, thanks for the info. Maybe if I get some time I can dig in and hack away at it to see if i can't figure out how to increase performance for larger file sets.

Thanks!

[thibaudgg]

Would be awesome! You can try to comment that line: https://github.com/guard/listen/blob/master/lib/listen/file.rb#L104, maybe it'll boost everything.

[jwhitcraft]

@thibaudgg, In my limited testing, that did boost everything, there is still an initial delay of about 1 and half minutes where the cpu is at 100+%, I assume where it's indexing all the files.

I'm gonna play around with it more to try and figure out a better way to handle that many files as IMO it should be supported.

[thibaudgg]

Interesting, normally timestamp and checksum should be eager loaded to avoid such initial delay, there's maybe something to fix there.

[thibaudgg]

@jwhitcraft could you try if release 2.6.0 fix it, I improve md5 checksum logic: db6dcc2

[jwhitcraft]

@thibaudgg, 2.6.0 did show a lot of improvement, it only ran at 100% for about 30 seconds

I know when dealing with the amount of files that I'm dealing with it's not going to be instantaneous, but this is a far cry better thant was it was at 2.4.x

But something I just noticed while using the new listener cli tool is that when i start modifying files the cpu jumps again and it takes a while for the event to come though.

I sampled the Process when it's spiked and attached it here:
https://gist.github.com/jwhitcraft/9171802

[thibaudgg]

Ok good news! Thanks for your feedback.

On Darwin, we still need to keep every files timestamp & mode (both read from File.lstat) and load them on boot, so with a big amount of files it'll still be slow. Too bad.

The new listener cli is just for tcp forwarding, so maybe it comes from there too. I'm sorry but I'm not sure I'll be able to found any useful info from the Process you give me.

Can we close it for now?

[jwhitcraft]

@thibaudgg, ya this can be closed. I'll keep playing around with it to see if i can find more speed improvements once it's running.

Thank!