The new option: "--jobs number (-j)" specifies the maximum number of
concurrent tasks. The suggested value is equal to the number of CPUs.
default: unlimited concurrent tasks (standard rake behavior)
1: one task at a time
The code consists of two major edits, the first is a change to
`application.rb` to support the parsing of the option.
The second is a more substantial change to `multi_task.rb` which
replaces the multi-task scheduling algorithm. Instead of spawning a new
thread for every pre-requisite that needs to be executed, a block is
created which calls the pre-requisite and added to a Queue.
Additionally, a thread-pool is created to pull the blocks off the queue
and execute them. Finally, the MultiTask queueing up its prerequisites
will itself participate in the block-processing while waiting for its
prerequisites to finish processing.
It can tell when its prerequisites are finished by enveloping the
queued blocks in another block that adds a little bookkeeping.
Rake ruby version requirements remain unchanged.
This is because I left in the original code which just passed through
to spawn unlimited threads.
Now the code always uses the thread pool, and sets the initial
limit to be the maximum Fixnum (which means virtually unlimited)
This is nice because it means the rake MultiTask tests are now
stressing the thread pool implementation.
Additionally, the thread pool size can now be changed dynamically
to adjust to load by changing 'application.options.thread_pool_size'
while rake is running.
There is no code that observes load, but it certainly could and
adjust as it saw fit.
While the threads are in the their processing loop, they add
other threads need to be added to the pool to meet the limit.
Additionally, if the thread pool size is larger than the
application preference the thread exits.
The previous "add blocks to queue" method worked, but had the
unnecessary side-effect of blowing the stack for large amounts of
This is a new thread pool implementation which retains all the
advantages of the original, but keeps the stack size the same as the
It's closer to the pre-thread-pool rake implementation with the
addition of checking the thread pool size before spawning a new thread.
This change pulls the concurrent implementation of
'invoke_prerequisites' out of MultiTask into Task while changing the
method name to 'invoke_prerequisites_concurrently'
Then, if -m is passed, Task calls 'invoke_prerequisites_concurrently'
so everything is multithreaded.
MultiTask always calls 'invoke_prerequisites_concurrently'
This is implemented by adding Rake::WorkerPool which provides a way for
callers to synchronously execute blocks by a thread pool.
What has not changed (and is still problematic) is that Rake creates a
new thread for each and every MultiTask prerequisite.
NOTES: Passes all rake tests
- now has the ability to execute an array of blocks and wait for them
all to execute.
- only adds a new thread when there is no thread waiting for action.
This slows the ramp up and threads are better reused
- has a minimum and maximum size. By default, minimum is 1 and maximum
is the maximum fixnum
- removed unused #wait call
- Now uses WorkerPool#execte_blocks to execute its prerequisites
…ongst a thread pool.
Since the WorkerPool was only needed in MultiTask, I changed Task back to simply executing it's actions and calling its prerequisites.
Merge branch 'master' into everything_is_a_multitask
* The default WorkerPool maximum size was changed to FIXNUM_MAX. This
is ok because only enough blocks are added to the thread pool to
cover the requested number of blocks (but not past the maximum).
* Added a #join call which clears the thread pool of all threads. This
is called inside #execute_blocks when there are no more threads
waiting on the thread pool.
* Suppressed "multithreads" output when specifying -m
* Removed unnecessary exception backtrace concatenating
* -j now is kinder when receiving bad input. If it has no parameter
or the parameter can't be parsed, it defaults to 2.
* Added WorkerPool test
* added tests for -j and -m to the application options test.