Skip to content
This repository

Add -j <max_jobs> and -m (all tasks multitasks) -- solution included #113

Closed
wants to merge 10 commits into from

9 participants

Michael Bishop Paweł Kondzior Malachi Griffie Jai Singh amcbride AKing Nick Maiorsky Lars Burgess Jim Weirich
Michael Bishop

PROBLEM SUMMARY (THE CASE FOR -j and -m)

Rake can be unusable for builds invoking large numbers of concurrent external processes.

PROBLEM DESCRIPTION

Rake makes it easy to maximize concurrency in builds with its "multitask" function. When using rake to build non-ruby projects quite often rake needs to execute shell tasks to process files. Unfortunately, when executing multitasks, rake spawns a new thread for each task-prerequisite. This shouldn't cause problems when the build code is pure ruby (for green threads), but when the tasks are executing external processes, the sheer number of spawned processes can cause the machine to thrash. Additionally ruby can reach the maximum number of open files (presumably because it's reading stdout for all those processes).

SOLUTION SUMMARY

This request includes the code to add support for a --jobs NUMBER (-j) command-line option to specify the number of simultaneous tasks to execute.

  • To maintain backward compatibility, not passing -j reverts to the old behavior of unlimited concurrent tasks.

As a nod to drake, a --multitask (-m) flag is also included which when supplied, changes tasks into multitasks.

SOLUTION

Rather than spawning a new thread per prerequisite MultiTask now sends its prerequisites to a WorkerPool object. WorkerPool.new(n).execute_blocks has the same semantics as Thread.new...join but caps the thread count at n.

Core Change

threads = @prerequisites.collect { |p|
  Thread.new(p) { |r| application[r, @scope].invoke_with_call_chain(args, invocation_chain) }
}
threads.each { |t| t.join }

...becomes...

@@wp ||= WorkerPool.new(application.options.thread_pool_size)

blocks = @prerequisites.collect { |r|
  lambda { application[r, @scope].invoke_with_call_chain(args, invocation_chain) }
}
@@wp.execute_blocks blocks

To support -m, the MultiTask implementation has moved to Task#invoke_prerequisites_concurrently and is called from MultiTask#invoke_prerequisites. This enables concurrent behavior for Task when -m is used.

Details

WorkerPool#execute_blocks adds the passed-in blocks to a queue, ensures there are enough threads to execute them (under the maximum), and sleeps the current thread until the blocks are processed.

This creates a few potential problems:

What if all of the blocks then called #execute_blocks? Wouldn't that sleep all the threads?

Yes it would. This is solved as #execute_blocks removes the current thread from the thread pool just before it sleeps and creates a new one in its place. When all the blocks are processed, the current thread is added back to the pool (adjusting for the max-size). There are always enough available threads in the thread pool for processing.

When do the threads shutdown?

WorkerPool#execute_blocks knows how many threads are waiting for their blocks to be processed. If, upon its awakening, it notices there are no threads waiting on blocks, it shuts down the thread pool.

Statistics

 ---LINES--         ----LOC---
  old   new  diff    old   new  diff  File Name
 ----------  ----   ----------  ----  ----------
  598   605    +7    477   484    +7  lib/rake/application.rb
   16    13    -3     11     8    -3  lib/rake/multi_task.rb
  327   341   +14    210   222   +12  lib/rake/task.rb
        111                 80        lib/rake/worker_pool.rb
 4264  4393         2696  2792        TOTAL
 ------------------------------------------------
             +129                +96  SUMMARY

TESTS

Tests are included for all new functionality

REQUIREMENTS

The Ruby version requirements remain the same. lib/rake/worker_pool.rb adds two new requirements: thread and set

added some commits April 18, 2012
Michael Bishop Rake now supports a --jobs <n> command-line option.
DESCRIPTION
-----------
The new option: "--jobs number (-j)" specifies the maximum number of
concurrent tasks. The suggested value is equal to the number of CPUs.

Sample values:
  default: unlimited concurrent tasks (standard rake behavior)
  1: one task at a time

The code consists of two major edits, the first is a change to
`application.rb` to support the parsing of the option.

The second is a more substantial change to `multi_task.rb` which
replaces the multi-task scheduling algorithm. Instead of spawning a new
thread for every pre-requisite that needs to be executed, a block is
created which calls the pre-requisite and added to a Queue.

Additionally, a thread-pool is created to pull the blocks off the queue
and execute them. Finally, the MultiTask queueing up its prerequisites
will itself participate in the block-processing while waiting for its
prerequisites to finish processing.

It can tell when its prerequisites are finished by enveloping the
queued blocks in another block that adds a little bookkeeping.

VERSION REQUIREMENTS
--------------------
Rake ruby version requirements remain unchanged.
295c7a4
Michael Bishop Fixed a bug where the MultiTask tests were not testing the thread pool
This is because I left in the original code which just passed through
to spawn unlimited threads.

Now the code always uses the thread pool, and sets the initial
limit to be the maximum Fixnum (which means virtually unlimited)

This is nice because it means the rake MultiTask tests are now
stressing the thread pool implementation.

Additionally, the thread pool size can now be changed dynamically
to adjust to load by changing 'application.options.thread_pool_size'
while rake is running.

There is no code that observes load, but it certainly could and
adjust as it saw fit.

Notes:
  While the threads are in the their processing loop, they add
  other threads need to be added to the pool to meet the limit.
  Additionally, if the thread pool size is larger than the
  application preference the thread exits.
d1229f0
Michael Bishop Fixed bug where the stack was being blown
The previous "add blocks to queue" method worked, but had the
unnecessary side-effect of blowing the stack for large amounts of
prerequisites.

This is a new thread pool implementation which retains all the
advantages of the original, but keeps the stack size the same as the
pre-thread-pool rake.

It's closer to the pre-thread-pool rake implementation with the
addition of checking the thread pool size before spawning a new thread.
578b637
Michael Bishop Added -m option which forces every task to be a multitask
This change pulls the concurrent implementation of
'invoke_prerequisites' out of MultiTask into Task while changing the
method name to 'invoke_prerequisites_concurrently'

Then, if -m is passed, Task calls 'invoke_prerequisites_concurrently'
so everything is multithreaded.

MultiTask always calls 'invoke_prerequisites_concurrently'
0ee43db
Michael Bishop Forgot to update the rdoc for the -m option c3f5260
Michael Bishop Rake now has a throttle on simultaneous task count
This is implemented by adding Rake::WorkerPool which provides a way for
callers to synchronously execute blocks by a thread pool.

What has not changed (and is still problematic) is that Rake creates a
new thread for each and every MultiTask prerequisite.

NOTES: Passes all rake tests
faa1ff1
Michael Bishop MultiTask no longer spawns a thread for each prerequisite
WorkerPool:
- now has the ability to execute an array of blocks and wait for them
all to execute.
- only adds a new thread when there is no thread waiting for action.
This slows the ramp up and threads are better reused
- has a minimum and maximum size. By default, minimum is 1 and maximum
is the maximum fixnum
- removed unused #wait call

MultiTask:
- Now uses WorkerPool#execte_blocks to execute its prerequisites
7fa886d
Michael Bishop Fixed WorkerPool so it can synchronously execute a group of blocks am…
…ongst a thread pool.

Since the WorkerPool was only needed in MultiTask, I changed Task back to simply executing it's actions and calling its prerequisites.

Merge branch 'master' into everything_is_a_multitask

Conflicts:
	lib/rake/application.rb
	lib/rake/multi_task.rb
2a2bb2f
Michael Bishop Added tests. tidied up command-line ops. WorkerPool default size is F…
…IXNUM_MAX.

* The default WorkerPool maximum size was changed to FIXNUM_MAX. This
  is ok  because only enough blocks are added to the thread pool to
  cover the requested number of blocks (but not past the maximum).
* Added a #join call which clears the thread pool of all threads. This
  is called inside #execute_blocks when there are no more threads
  waiting on the thread pool.
* Suppressed "multithreads" output when specifying -m
* Removed unnecessary exception backtrace concatenating
* -j now is kinder when receiving bad input. If it has no parameter
  or the parameter can't be parsed, it defaults to 2.
* Added WorkerPool test
* added tests for -j and -m to the application options test.
f40087d
Michael Bishop -m hooked up. test included
Added a test for -m in task.rb. Hooked it up.
Concised code in application.rb
Added documentation for -j
b828747
Paweł Kondzior

+1, is this going to be merged?

Michael Bishop
Malachi Griffie

Any movement on this? This is incredibly useful, however we depend on the distributed rake gem so we can't directly use @michaeljbishop's changes until they are integrated.

Jai Singh

this would be really helpful and give us a comparable feature to make

amcbride

Please merge! This would be awesome

AKing
aking0 commented June 29, 2012

This feature would be very useful, hopefully it can be merged in shortly.

Nick Maiorsky
nickma commented June 29, 2012

This will be really helpful for us.

Lars Burgess

+1, this would be awesome!

Jim Weirich
Owner

First, sorry for the delay in responding to this... I'm going through the backlog of Rake work and its taking some time.

Second, I like the functionality of the patch, but I find the worker pool logic a bit difficult to work through. I'm reluctant to merge this until I have a better handle on what's going on.

Michael Bishop
Jim Weirich
Owner

Excellent. Quick question: If multitask is waiting for the futures I'm assuming it won't be available for job processing. It is possible to get in a state where all the threads in the pool are waiting for futures and nothing is available to actually do the work?

This is the point I reached in my own simple-minded implementation of a thread pool. It seems your initial version side-stepped this problem, but I wasn't clear on how it did it.

Michael Bishop
Michael Bishop
Michael Bishop

Hi Jim. As promised, I have a new implementation that I feel is a much improved version of this pull-request (and a tidier git history to boot), I have removed this repository, created a new one and added a pull request for that implementation here: #131

Michael Bishop michaeljbishop closed this October 18, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 10 unique commits by 1 author.

Apr 18, 2012
Michael Bishop Rake now supports a --jobs <n> command-line option.
DESCRIPTION
-----------
The new option: "--jobs number (-j)" specifies the maximum number of
concurrent tasks. The suggested value is equal to the number of CPUs.

Sample values:
  default: unlimited concurrent tasks (standard rake behavior)
  1: one task at a time

The code consists of two major edits, the first is a change to
`application.rb` to support the parsing of the option.

The second is a more substantial change to `multi_task.rb` which
replaces the multi-task scheduling algorithm. Instead of spawning a new
thread for every pre-requisite that needs to be executed, a block is
created which calls the pre-requisite and added to a Queue.

Additionally, a thread-pool is created to pull the blocks off the queue
and execute them. Finally, the MultiTask queueing up its prerequisites
will itself participate in the block-processing while waiting for its
prerequisites to finish processing.

It can tell when its prerequisites are finished by enveloping the
queued blocks in another block that adds a little bookkeeping.

VERSION REQUIREMENTS
--------------------
Rake ruby version requirements remain unchanged.
295c7a4
Apr 20, 2012
Michael Bishop Fixed a bug where the MultiTask tests were not testing the thread pool
This is because I left in the original code which just passed through
to spawn unlimited threads.

Now the code always uses the thread pool, and sets the initial
limit to be the maximum Fixnum (which means virtually unlimited)

This is nice because it means the rake MultiTask tests are now
stressing the thread pool implementation.

Additionally, the thread pool size can now be changed dynamically
to adjust to load by changing 'application.options.thread_pool_size'
while rake is running.

There is no code that observes load, but it certainly could and
adjust as it saw fit.

Notes:
  While the threads are in the their processing loop, they add
  other threads need to be added to the pool to meet the limit.
  Additionally, if the thread pool size is larger than the
  application preference the thread exits.
d1229f0
Michael Bishop Fixed bug where the stack was being blown
The previous "add blocks to queue" method worked, but had the
unnecessary side-effect of blowing the stack for large amounts of
prerequisites.

This is a new thread pool implementation which retains all the
advantages of the original, but keeps the stack size the same as the
pre-thread-pool rake.

It's closer to the pre-thread-pool rake implementation with the
addition of checking the thread pool size before spawning a new thread.
578b637
Apr 21, 2012
Michael Bishop Added -m option which forces every task to be a multitask
This change pulls the concurrent implementation of
'invoke_prerequisites' out of MultiTask into Task while changing the
method name to 'invoke_prerequisites_concurrently'

Then, if -m is passed, Task calls 'invoke_prerequisites_concurrently'
so everything is multithreaded.

MultiTask always calls 'invoke_prerequisites_concurrently'
0ee43db
Michael Bishop Forgot to update the rdoc for the -m option c3f5260
Apr 23, 2012
Michael Bishop Rake now has a throttle on simultaneous task count
This is implemented by adding Rake::WorkerPool which provides a way for
callers to synchronously execute blocks by a thread pool.

What has not changed (and is still problematic) is that Rake creates a
new thread for each and every MultiTask prerequisite.

NOTES: Passes all rake tests
faa1ff1
Apr 24, 2012
Michael Bishop MultiTask no longer spawns a thread for each prerequisite
WorkerPool:
- now has the ability to execute an array of blocks and wait for them
all to execute.
- only adds a new thread when there is no thread waiting for action.
This slows the ramp up and threads are better reused
- has a minimum and maximum size. By default, minimum is 1 and maximum
is the maximum fixnum
- removed unused #wait call

MultiTask:
- Now uses WorkerPool#execte_blocks to execute its prerequisites
7fa886d
Michael Bishop Fixed WorkerPool so it can synchronously execute a group of blocks am…
…ongst a thread pool.

Since the WorkerPool was only needed in MultiTask, I changed Task back to simply executing it's actions and calling its prerequisites.

Merge branch 'master' into everything_is_a_multitask

Conflicts:
	lib/rake/application.rb
	lib/rake/multi_task.rb
2a2bb2f
Apr 26, 2012
Michael Bishop Added tests. tidied up command-line ops. WorkerPool default size is F…
…IXNUM_MAX.

* The default WorkerPool maximum size was changed to FIXNUM_MAX. This
  is ok  because only enough blocks are added to the thread pool to
  cover the requested number of blocks (but not past the maximum).
* Added a #join call which clears the thread pool of all threads. This
  is called inside #execute_blocks when there are no more threads
  waiting on the thread pool.
* Suppressed "multithreads" output when specifying -m
* Removed unnecessary exception backtrace concatenating
* -j now is kinder when receiving bad input. If it has no parameter
  or the parameter can't be parsed, it defaults to 2.
* Added WorkerPool test
* added tests for -j and -m to the application options test.
f40087d
Apr 27, 2012
Michael Bishop -m hooked up. test included
Added a test for -m in task.rb. Hooked it up.
Concised code in application.rb
Added documentation for -j
b828747
This page is out of date. Refresh to see the latest.
14  doc/command_line_usage.rdoc
Source Rendered
@@ -31,15 +31,27 @@ Options are:
31 31
 [<tt>--execute-print</tt> _code_ (-p)]
32 32
     Execute some Ruby code, print the result, and exit.
33 33
 
34  
-[<tt>--execute-continue</tt> _code_ (-p)]
  34
+[<tt>--execute-continue</tt> _code_ (-E)]
35 35
     Execute some Ruby code, then continue with normal task processing.
36 36
 
37 37
 [<tt>--help</tt>  (-H)]
38 38
     Display some help text and exit.
39 39
 
  40
+[<tt>--jobs</tt> _number_  (-j)]
  41
+    Specifies the maximum number of concurrent tasks. The suggested
  42
+    value is equal to the number of CPUs.
  43
+    
  44
+    Sample values:
  45
+     no -j   : unlimited concurrent tasks (standard rake behavior)
  46
+     only -j : 2 concurrent tasks
  47
+     -j 16   : 16 concurrent tasks
  48
+
40 49
 [<tt>--libdir</tt> _directory_  (-I)]
41 50
     Add _directory_ to the list of directories searched for require.
42 51
 
  52
+[<tt>--multitask</tt> (-m)]
  53
+    Treat all tasks as multitasks. ('make/drake' semantics)
  54
+
43 55
 [<tt>--nosearch</tt>  (-N)]
44 56
     Do not search for a Rakefile in parent directories.
45 57
 
7  lib/rake/application.rb
@@ -325,9 +325,16 @@ def standard_rake_options
325 325
           "Execute some Ruby code, then continue with normal task processing.",
326 326
           lambda { |value| eval(value) }
327 327
         ],
  328
+        ['--jobs',  '-j [NUMBER]',
  329
+          "Specifies the maximum number of tasks to execute in parallel. (default:2)",
  330
+          lambda { |value| options.thread_pool_size = [(value || 2).to_i,2].max }
  331
+        ],
328 332
         ['--libdir', '-I LIBDIR', "Include LIBDIR in the search path for required modules.",
329 333
           lambda { |value| $:.push(value) }
330 334
         ],
  335
+        ['--multitask', '-m', "Treat all tasks as multitasks.",
  336
+          lambda { |value| options.always_multitask = true }
  337
+        ],
331 338
         ['--no-search', '--nosearch', '-N', "Do not search parent directories for the Rakefile.",
332 339
           lambda { |value| options.nosearch = true }
333 340
         ],
9  lib/rake/multi_task.rb
@@ -5,12 +5,9 @@ module Rake
5 5
   #
6 6
   class MultiTask < Task
7 7
     private
8  
-    def invoke_prerequisites(args, invocation_chain)
9  
-      threads = @prerequisites.collect { |p|
10  
-        Thread.new(p) { |r| application[r, @scope].invoke_with_call_chain(args, invocation_chain) }
11  
-      }
12  
-      threads.each { |t| t.join }
  8
+    def invoke_prerequisites(task_args, invocation_chain) # :nodoc:
  9
+      invoke_prerequisites_concurrently(task_args, invocation_chain)
13 10
     end
14  
-  end
15 11
 
  12
+  end
16 13
 end
24  lib/rake/task.rb
... ...
@@ -1,4 +1,5 @@
1 1
 require 'rake/invocation_exception_mixin'
  2
+require 'rake/worker_pool'
2 3
 
3 4
 module Rake
4 5
 
@@ -171,10 +172,23 @@ def add_chain_to(exception, new_chain)
171 172
 
172 173
     # Invoke all the prerequisites of a task.
173 174
     def invoke_prerequisites(task_args, invocation_chain) # :nodoc:
174  
-      prerequisite_tasks.each { |prereq|
175  
-        prereq_args = task_args.new_scope(prereq.arg_names)
176  
-        prereq.invoke_with_call_chain(prereq_args, invocation_chain)
  175
+      if application.options.always_multitask
  176
+        invoke_prerequisites_concurrently(task_args, invocation_chain)
  177
+      else
  178
+        prerequisite_tasks.each { |prereq|
  179
+          prereq_args = task_args.new_scope(prereq.arg_names)
  180
+          prereq.invoke_with_call_chain(prereq_args, invocation_chain)
  181
+        }
  182
+      end
  183
+    end
  184
+
  185
+    def invoke_prerequisites_concurrently(args, invocation_chain)
  186
+      @@wp ||= WorkerPool.new(application.options.thread_pool_size)
  187
+    
  188
+      blocks = @prerequisites.collect { |r|
  189
+        lambda { application[r, @scope].invoke_with_call_chain(args, invocation_chain) }
177 190
       }
  191
+      @@wp.execute_blocks blocks
178 192
     end
179 193
 
180 194
     # Format the trace flags for display.
@@ -200,9 +214,9 @@ def execute(args=nil)
200 214
       @actions.each do |act|
201 215
         case act.arity
202 216
         when 1
203  
-          act.call(self)
  217
+           act.call(self)
204 218
         else
205  
-          act.call(self, args)
  219
+           act.call(self, args)
206 220
         end
207 221
       end
208 222
     end
111  lib/rake/worker_pool.rb
... ...
@@ -0,0 +1,111 @@
  1
+require 'thread'
  2
+require 'set'
  3
+
  4
+module Rake
  5
+  class WorkerPool
  6
+    attr_accessor :maximum_size   # this is the maximum size of the pool
  7
+
  8
+    def initialize(max = nil)
  9
+      @threads = Set.new          # this holds the set of threads in the pool
  10
+      @waiting_threads = Set.new  # set of threads waiting in #execute_blocks
  11
+      @threads_mutex = Mutex.new  # use this whenever r/w @threads
  12
+      @queue = Queue.new          # this holds blocks to be executed
  13
+      @join_cv = ConditionVariable.new # alerts threads sleeping from calling #join
  14
+      if (max && max > 0)
  15
+        @maximum_size = max
  16
+      else
  17
+        @maximum_size = (2**(0.size * 8 - 2) - 1) # FIXNUM_MAX
  18
+      end
  19
+    end
  20
+
  21
+    def execute_blocks(blocks)
  22
+      mutex = Mutex.new
  23
+      cv = ConditionVariable.new
  24
+      exception = nil
  25
+      unprocessed_block_count = blocks.count
  26
+      mutex.synchronize {
  27
+        blocks.each { |block|
  28
+          @queue.enq lambda {
  29
+            begin
  30
+              block.call
  31
+            rescue Exception => e
  32
+              exception = e
  33
+            ensure
  34
+              # we *have* to have this 'ensure' because we *have* to
  35
+              # call cv.signal to wake up WorkerPool#execute_blocks
  36
+              # which is asleep because it called cv.wait(mutex)
  37
+              mutex.synchronize { unprocessed_block_count -= 1; cv.signal }
  38
+            end
  39
+          }
  40
+        }
  41
+        was_in_set = @threads_mutex.synchronize {
  42
+          @waiting_threads.add(Thread.current)
  43
+          @threads.delete? Thread.current
  44
+        }
  45
+        ensure_thread_count(blocks.count)
  46
+        cv.wait(mutex) until unprocessed_block_count == 0
  47
+        @threads_mutex.synchronize {
  48
+          @waiting_threads.delete(Thread.current)
  49
+          @threads.add(Thread.current) if was_in_set
  50
+          # shutdown the thread pool if we were the last thread
  51
+          # waiting on the thread pool to process blocks
  52
+          join if @waiting_threads.count == 0
  53
+        }
  54
+      }
  55
+      # raise any exceptions that arose in the block (the last
  56
+      # exception won)
  57
+      raise exception if exception
  58
+    end
  59
+
  60
+    def ensure_thread_count(count)
  61
+      # here, we need to somehow make sure to add as many threads as
  62
+      # are needed and no more. So (blocks.size - ready threads)
  63
+      @threads_mutex.synchronize {
  64
+        threads_needed = [[@maximum_size,count].min - @threads.size, 0].max
  65
+        threads_needed.times do
  66
+          t = Thread.new do
  67
+            begin
  68
+              while @threads.size <= @maximum_size
  69
+                @queue.deq.call
  70
+              end
  71
+            ensure
  72
+              @threads_mutex.synchronize {
  73
+                @threads.delete(Thread.current)
  74
+                @join_cv.signal
  75
+              }
  76
+            end
  77
+          end
  78
+          @threads.add t
  79
+        end
  80
+      }
  81
+    end
  82
+    private :ensure_thread_count
  83
+    
  84
+    def join
  85
+      # *** MUST BE CALLED inside @threads_mutex.synchronize{}
  86
+      # because we don't want any threads added while we wait, only
  87
+      # removed we set the maximum size to 0 and then add enough blocks
  88
+      # to the queue so any sleeping threads will wake up and notice
  89
+      # there are more threads than the limit and exit
  90
+      saved_maximum_size, @maximum_size = @maximum_size, 0
  91
+      @threads.each { @queue.enq lambda { ; } } # wake them all up
  92
+
  93
+      # here, we sleep and wait for a signal off @join_cv
  94
+      # we will get it once for each sleeping thread so we watch the
  95
+      # thread count
  96
+      #
  97
+      # avoid the temptation to change this to
  98
+      # "<code> until <condition>". The condition needs to checked
  99
+      # first or you will deadlock.
  100
+      while (@threads.size > 0)
  101
+        @join_cv.wait(@threads_mutex)
  102
+      end
  103
+      
  104
+      # now everything has been executed and we are ready to
  105
+      # start accepting more work so we raise the limit back
  106
+      @maximum_size = saved_maximum_size
  107
+    end
  108
+    private :join
  109
+
  110
+  end
  111
+end
20  test/test_rake_application_options.rb
@@ -33,6 +33,7 @@ def test_default_options
33 33
     assert_nil opts.dryrun
34 34
     assert_nil opts.ignore_system
35 35
     assert_nil opts.load_system
  36
+    assert_nil opts.always_multitask
36 37
     assert_nil opts.nosearch
37 38
     assert_equal ['rakelib'], opts.rakelib
38 39
     assert_nil opts.show_prereqs
@@ -40,6 +41,7 @@ def test_default_options
40 41
     assert_nil opts.show_tasks
41 42
     assert_nil opts.silent
42 43
     assert_nil opts.trace
  44
+    assert_nil opts.thread_pool_size
43 45
     assert_equal ['rakelib'], opts.rakelib
44 46
     assert ! Rake::FileUtilsExt.verbose_flag
45 47
     assert ! Rake::FileUtilsExt.nowrite_flag
@@ -110,6 +112,18 @@ def test_help
110 112
     assert_equal :exit, @exit
111 113
   end
112 114
 
  115
+  def test_jobs
  116
+    flags(['--jobs', '4'], ['-j', '4']) do |opts|
  117
+      assert_equal 4, opts.thread_pool_size
  118
+    end
  119
+    flags(['--jobs', 'asdas'], ['-j', 'asdas']) do |opts|
  120
+      assert_equal 2, opts.thread_pool_size
  121
+    end
  122
+    flags('--jobs', '-j') do |opts|
  123
+      assert_equal 2, opts.thread_pool_size
  124
+    end
  125
+  end
  126
+
113 127
   def test_libdir
114 128
     flags(['--libdir', 'xx'], ['-I', 'xx'], ['-Ixx']) do |opts|
115 129
       $:.include?('xx')
@@ -118,6 +132,12 @@ def test_libdir
118 132
     $:.delete('xx')
119 133
   end
120 134
 
  135
+  def test_multitask
  136
+    flags('--multitask', '-m') do |opts|
  137
+      assert_equal opts.always_multitask, true
  138
+    end
  139
+  end
  140
+
121 141
   def test_rakefile
122 142
     flags(['--rakefile', 'RF'], ['--rakefile=RF'], ['-f', 'RF'], ['-fRF']) do |opts|
123 143
       assert_equal ['RF'], @app.instance_eval { @rakefiles }
19  test/test_rake_task.rb
@@ -223,6 +223,25 @@ def c.timestamp() Time.now + 5 end
223 223
     assert_in_delta now + 10, a.timestamp, 0.1, 'computer too slow?'
224 224
   end
225 225
 
  226
+  def test_all_multitask
  227
+    mx = Mutex.new
  228
+    result = ""
  229
+    root = task :root
  230
+    ('aa'..'zz').each do |c|
  231
+      task(c.to_sym) { mx.synchronize{ result << c } }
  232
+      task(:root => c.to_sym)
  233
+    end
  234
+    root.invoke
  235
+
  236
+    root.prerequisite_tasks.each { |p| p.reenable };
  237
+    root.reenable
  238
+    task_result = result.dup; result.clear
  239
+    
  240
+    Rake.application.options.always_multitask = true
  241
+    root.invoke
  242
+    refute_equal task_result, result
  243
+  end
  244
+
226 245
   def test_investigation_output
227 246
     t1 = task(:t1 => [:t2, :t3]) { |t| runlist << t.name; 3321 }
228 247
     task(:t2)
90  test/test_rake_test_worker_pool.rb
... ...
@@ -0,0 +1,90 @@
  1
+require File.expand_path('../helper', __FILE__)
  2
+require 'rake/worker_pool'
  3
+require 'test/unit/assertions'
  4
+
  5
+class TestRakeTestWorkerPool < Rake::TestCase
  6
+  include Rake
  7
+  
  8
+  def test_block_order
  9
+    mx = Mutex.new
  10
+    block = lambda {|executor,count=20,result=""|
  11
+      return if (count < 1)
  12
+      mx.synchronize{ result << count.to_s }
  13
+      sleep(rand * 0.01)
  14
+      executor.call( [lambda {block.call(executor,count-1,result)}] )
  15
+      result
  16
+    }
  17
+
  18
+    old = lambda {|b|
  19
+        threads = b.collect {|c| Thread.new(c) {|d| d.call } }
  20
+        threads.each {|t| t.join }
  21
+    }
  22
+
  23
+    wp = WorkerPool.new(4)
  24
+    new = lambda {|b| wp.execute_blocks b }
  25
+    
  26
+    assert_equal(block.call(old), block.call(new))
  27
+  end
  28
+
  29
+  # test that there are no deadlocks within the worker pool itself
  30
+  def test_deadlocks
  31
+    wp = WorkerPool.new(10)
  32
+    blocks = []
  33
+    10.times {
  34
+       inner_block = lambda {|count=5|
  35
+        return if (count < 1)
  36
+        sleep(rand * 0.000001)
  37
+        inner_blocks = []
  38
+        3.times {  inner_blocks << lambda {inner_block.call(count-1)} }
  39
+        wp.execute_blocks inner_blocks
  40
+      }
  41
+      blocks << inner_block
  42
+    }
  43
+    wp.execute_blocks blocks
  44
+  end
  45
+
  46
+  # test that throwing an exception way down in the blocks propagates
  47
+  # to the top
  48
+  def test_exceptions
  49
+    wp = WorkerPool.new(4)
  50
+    deep_exception_block = lambda {|count=3|
  51
+      raise Exception.new if ( count < 1 )
  52
+      deep_exception_block.call(count-1)
  53
+    }
  54
+    assert_raises(Exception) do
  55
+      wp.execute_blocks [deep_exception_block]
  56
+    end
  57
+  end
  58
+
  59
+  def test_thread_count
  60
+    mutex = Mutex.new
  61
+    expected_thread_count = 2
  62
+    wp = WorkerPool.new(expected_thread_count)
  63
+
  64
+    # the lambda code will determine how many threads are running in
  65
+    # the pool
  66
+    blocks = []
  67
+    thread_count = 0
  68
+    should_sleep = true
  69
+    (expected_thread_count*2).times do
  70
+      blocks << lambda {
  71
+        mutex.synchronize do; stack_prefix = "#{__FILE__}:#{__LINE__}" # this synchronize will be on the stack
  72
+          sleep 1 if should_sleep # this lets all the threads wait on the mutex
  73
+          threads = Thread.list
  74
+          backtraces = threads.collect {|t| t.backtrace}
  75
+          # sometimes a thread doesn't return a thread count
  76
+          if ( threads.count == backtraces.count )
  77
+            should_sleep = false
  78
+            # finds all the backtraces that contain our mutex.synchronize call
  79
+            our_bt = backtraces.find_all{|bt| bt && bt.index{|tr| tr.start_with? stack_prefix}!=nil }
  80
+            thread_count = [thread_count, our_bt.count].max
  81
+          end
  82
+        end
  83
+      }
  84
+    end
  85
+    
  86
+    wp.execute_blocks blocks
  87
+    assert_equal(expected_thread_count, thread_count)
  88
+  end
  89
+end
  90
+
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.