[Fix #117] Add --parallel option #4272
Conversation
This change tries to solve the tricky parallel execution problem by spawning off a number of processes/threads to do file inspection, sharing the work between them, without collecting any output. When all processes are finished, the original process runs the full inspection again, taking advantage of result caching.
| it 'prints a warning' do | ||
| cli.run ['-P'] | ||
| expect($stderr.string) | ||
| .to include('Process.fork is not supported by this Ruby') |
There was a problem hiding this comment.
Won't this happen using JRuby as well?
There was a problem hiding this comment.
No on JRuby the parallel gem uses threads instead of processes.
Can we apply a similar concept to running the cops in parallel and then merge the result when each one finishes to avoid having to run rely on the cache and running checks twice? |
|
👍 Great work, @jonas054! I love the simplicity or your solution of the problem! |
Perhaps, but I think that coming up with a solution that does everything we wish for is exceedingly difficult. Otherwise we wouldn't be closing in on the four year anniversary of the issue saying that we'd like to run RuboCop in parallel -- without any successful resolution. 😄 What I'm trying to do here is to set the bar a bit lower, in order to get something that works. Glad to see @bbatsov agreeing. |
|
Sorry if this was mentioned somewhere else: Why is |
|
The error message says A more detailed explanation is that it might be possible to get parallel configuration generation to work, but it would require some extra logic, and I didn't think it worth the effort when I made the implementation. I wanted something simple. Anyone who's interested is most welcome to give the matter some thought and let the rest of us know what you came up with. 😄 |
|
Awesome, thanks for the contribution! We were super blown away when we found it. I'm just glad to know that |
Another stab at an old problem.
This change tries to solve the tricky parallel execution problem by spawning off a number of processes/threads to do file inspection, sharing the work between them, without collecting any output. When all processes are finished, the original process runs the full inspection again, taking advantage of result caching.
There's not a lot of specs added for parallel execution. I've run on RuboCop's own source with
--force-default-configto see that I get the same offenses with and without-P.With MRI there seems to be a speed gain of around 3 times when running on an 8 core machine. With JRuby and Rubinius, it's about 2 times.
@bbatsov said in #3794
With the solution proposed here, this would not matter. Files are inspected in parallel, but cops are being run in sequence on a given file.
I have looked but have not found any evidence for tricky cops like that, and I have no memory of them being added. Let's hope I'm right about that, because otherwise I don't think there's any easy way to parallelize the execution.