Running guards in parallel #52

Closed
ryanb opened this Issue May 2, 2011 · 17 comments

Comments

Projects
None yet
5 participants

ryanb commented May 2, 2011

In the latest Railscasts episode I ran into the issue that LiveReload changes were not shown until after the specs run. Moving them first helps solve this problem, but if you're making changes quickly the tests will still be running and you won't see the changes.

I would love it if there was some way to specify a guard to run in a separate thread so it won't interfere with other guards. This can also address issue #19 where changes to files are not detected while a guard is running. IIRC other similar tools such as autotest detect changes while the tests are running, why can't guard? It's frustrating to have to re-save a file for Guard to pick up on a change because it happened to be running before.

Update: Maybe this can be combined with the groups feature (issue #26), because often you want some guards to run in sequence (cucumber with rspec) but some in parallel. Placing guards in a group could make it run in sequence.

Contributor

netzpirat commented May 4, 2011

This was exactly one of the reasons why I implemented the group feature: It enables you to selectively start different guard groups in separate threads.

I admit it would be very nice if each guard group would be automatically run in a separate thread when no option --group is passed on the command line, but implementing this would increase code complexity a lot due to the design of Guard (the setup of the different guards and the listener are done statically in the Guard module).

So if you have a Guardfile flike this:

group 'backend' do
  guard 'rspec', :cli => '--color --format doc' do
    watch(%r{^spec/.+_spec\.rb})
    watch(%r{^lib/(.+)\.rb})         { |m| "spec/lib/#{m[1]}_spec.rb" }
  end
end

group 'frontend' do
  guard 'coffeescript', :input => 'app/coffeescripts', :output => 'public/javascripts/compiled'
  guard 'livereload' do
    watch(%r{^app/.+\.(erb|haml)})
  end
end

Just open a terminal and run guard -g fronend and another terminal to run guard -g backend. It's not as simple as just running guard and have the groups separated, but it works and it's not bad to have the outputs of the groups separated.

ryanb commented May 4, 2011

@netzpirat, good idea. At least that is a work-around. It would be nice if we didn't have to start two processes though.

There's still the other problem mentioned in #19 where files saved while the tests are running won't be picked up to run the tests again which requires me to save the file again for it to notice. I think that would be an easy feature to add once guards are split into multiple threads. The longer the tests take to run the more critical this is. I love Guard, but have a hard time using it on those kinds of projects.

Contributor

netzpirat commented May 4, 2011

I'd also prefer to start Guard only once. I've spent some time today to test some ideas how this can be done, but haven't found a simple solution yet. #19 has been solved with a workaround within guard-coffeescript, but it should be handled in common by Guard. I've some more ideas how to achieve this and will play some more with Guard the next days.

rymai was assigned Jul 21, 2011

Owner

rymai commented Sep 22, 2011

Hi @netzpirat, are you still interested in developing this feature (now that we store the list of groups)?

If yes, feel free to assign yourself to the issue instead of me! ;)

Contributor

netzpirat commented Sep 22, 2011

Not yet, because I have no need for it.

The reason why I wouldn't use it currently is simple: I like to have each Guard group in its own terminal window because I don't want to have my RSpec results mixed with CoffeeScript validations and the like. For this reason I use Terminitor in each project and I can perfectly configure my terminal setup to have my iTerm split with different Guard groups.

For the example in the issue description this issue makes perfectly sense, but I don't have the problem right now and I have spent (again) to much time on my toolkit (guard-jasmine lately) instead of my product, so someone who will profit from the solution should also implement it ;)

But this may change in the future...

Owner

rymai commented Sep 22, 2011

I understand and I agree!

@ryanb, are you still interested? If yes, feel free to make a pull-request.

Contributor

netzpirat commented Sep 22, 2011

I remember that we discussed the locking issue once, but I'm no longer sure what @thibaudgg said why locking is needed at all. Could it be an issue with inotify? In this case, I could simply add an option :locking that defaults to true, so that at least on OS X and Windows we could set it to false and enjoy parallel Guards with a very low effort?

Owner

thibaudgg commented Sep 23, 2011

Not sure if there's again an issue with inotify, with the new listening process introduced in Guard 0.7.0 it should be possible to add this feature without too much problems.
I think parallelizing Guard groups is the way to go, we just need to take care of not having the same group running twice in the same time (a group should wait for itself before running again).

Contributor

hedgehog commented Nov 5, 2011

I think the way to do this is to use, or at least copy, Aruba's process management, which uses child process, so is cross platform.
Anyway, you'd index the processes using the shell command that started them, duplicates can easily be prevented by searching for a process before starting.

I used this approach to manage SSH multiplexing connections, and in various tests it seems to be working very well.
The addiotnal code would be small if you depend on Aruba, but even if you just 'copy' Aruba's Process and API files it won't be much work.

HTH

Owner

thibaudgg commented Nov 5, 2011

Please can you share some links about this Aruba's process management.

Contributor

hedgehog commented Nov 5, 2011

Not sure if this[0] suffices (up/down that page there may be more relevant illustrations)? You'll likely need a get_name(process) but get_process(launch_string) shows how trivial that is. The whole API isn't much more complicated, nor is the whole process management code[1]

HTH

[0] https://github.com/cucumber/aruba/blob/master/lib/aruba/api.rb#L207
[1] https://github.com/cucumber/aruba/blob/master/lib/aruba/process.rb

Owner

thibaudgg commented Nov 13, 2011

Aruba's process management seems interesting, sadly I don't have the time to trying to add it on Guard at the moment. But it would be great if you give it a try! Thanks

Contributor

hedgehog commented Nov 13, 2011

Not familiar with Gaurd's internals, so to flatten the learning curve...
Can to suggest the best point at which to insert such a 'launch'.
Can you suggest the most appropriate module/class to contain such a launch method?
Finally, configuration. How to indicate parallel launch, maybe:

group 'frontend', :parallel => true do
  guard 'coffeescript', :input => 'app/coffeescripts', :output => 'public/javascripts/compiled'
  guard 'livereload' do
    watch(%r{^app/.+\.(erb|haml)})
  end
end
Owner

rymai commented Nov 14, 2011

Maybe this can help you:

Guards are all started (regardless of their group) here so you should probably replace it with a loop on the groups, like here where – if the group has the :parallel option – you would start a process (with #launch) and start guards using this new process.
I think you could add the launch method directly inside https://github.com/guard/guard/blob/dev/lib/guard.rb for now.

Does it make sense?

Contributor

hedgehog commented Nov 14, 2011

@rymai, thanks for that I'll try get to this in the next week or so. Am I right in understanding that both group and guard should accept a :parallel option? Specifically if guard takes a :parallel option then we'd need to loop over the watched and #launch each of them.

Contributor

netzpirat commented Oct 18, 2012

Oh my dear GitHub issue, I've looked at you many times, because you're still in open state. I've also read your completely at least three times, which takes ages... But nobody cares about you, not you, dear reader and recipient of the notification, not the creator of the issue, not anyone of the Guard maintainers, just nobody. There's no discussion since 11 months and nobody want to spend some time for you. In short: we don't need you. I'm sorry about you, but I'm going to close you now...

netzpirat closed this Oct 18, 2012

Owner

thibaudgg commented Oct 19, 2012

That was poetry. 😢

thibaudgg referenced this issue in guard/guard-livereload Mar 9, 2013

Closed

run order #67

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment