The new option: "--jobs number (-j)" specifies the maximum number of
concurrent tasks. The suggested value is equal to the number of CPUs.
default: unlimited concurrent tasks (standard rake behavior)
1: one task at a time
The code consists of two major edits, the first is a change to
`application.rb` to support the parsing of the option.
The second is a more substantial change to `multi_task.rb` which
replaces the multi-task scheduling algorithm. Instead of spawning a new
thread for every pre-requisite that needs to be executed, a block is
created which calls the pre-requisite and added to a Queue.
Additionally, a thread-pool is created to pull the blocks off the queue
and execute them. Finally, the MultiTask queueing up its prerequisites
will itself participate in the block-processing while waiting for its
prerequisites to finish processing.
It can tell when its prerequisites are finished by enveloping the
queued blocks in another block that adds a little bookkeeping.
Rake ruby version requirements remain unchanged.
This is because I left in the original code which just passed through
to spawn unlimited threads.
Now the code always uses the thread pool, and sets the initial
limit to be the maximum Fixnum (which means virtually unlimited)
This is nice because it means the rake MultiTask tests are now
stressing the thread pool implementation.
Additionally, the thread pool size can now be changed dynamically
to adjust to load by changing 'application.options.thread_pool_size'
while rake is running.
There is no code that observes load, but it certainly could and
adjust as it saw fit.
While the threads are in the their processing loop, they add
other threads need to be added to the pool to meet the limit.
Additionally, if the thread pool size is larger than the
application preference the thread exits.
The previous "add blocks to queue" method worked, but had the
unnecessary side-effect of blowing the stack for large amounts of
This is a new thread pool implementation which retains all the
advantages of the original, but keeps the stack size the same as the
It's closer to the pre-thread-pool rake implementation with the
addition of checking the thread pool size before spawning a new thread.