Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Updated version of spawn to work with newer Rails version

  • Loading branch information...
commit cdf812f106223f6bcd999c2d937f57dba45774fd 1 parent 779d591
@jpallen jpallen authored
View
4 vendor/plugins/spawn/CHANGELOG
@@ -49,3 +49,7 @@ v0.8 - 2008/05/02
--------------------------------------------------
v0.9 - 2008/05/11
* added ability to set nice level for child process
+
+--------------------------------------------------
+v1.0 - 2010/10/09
+* merged edged to master, let's call this version 1.0
View
121 vendor/plugins/spawn/README
@@ -1,121 +0,0 @@
-Spawn
-=====
-
-This plugin provides a 'spawn' method to easily fork OR thread long-running sections of
-code so that your application can return results to your users more quickly.
-This plugin works by creating new database connections in ActiveRecord::Base for the
-spawned block.
-
-The plugin also patches ActiveRecord::Base to handle some known bugs when using
-threads (see lib/patches.rb).
-
-Usage
------
-
-Here's a simple example of how to demonstrate the spawn plugin.
-In one of your controllers, insert this code (after installing the plugin of course):
-
- spawn do
- logger.info("I feel sleepy...")
- sleep 11
- logger.info("Time to wake up!")
- end
-
-If everything is working correctly, your controller should finish quickly then you'll see
-the last log message several seconds later.
-
-If you need to wait for the spawned processes/threads, then pass the objects returned by
-spawn to Spawn::wait(), like this:
-
- N.times do |i|
- # spawn N blocks of code
- spawn_ids[i] = spawn do
- something(i)
- end
- end
- # wait for all N blocks of code to finish running
- wait(spawn_ids)
-
-If you want your forked child to run at a lower priority than the parent process, pass in
-the :nice option like this:
-
- spawn(:nice => 7) do
- do_something_nicely
- end
-
-By default, spawn will use the fork to spawn child processes. You can configure it to
-do threading either by telling the spawn method when you call it or by configuring your
-environment.
-For example, this is how you can tell spawn to use threading on the call,
-
- spawn(:method => :thread) do
- something
- end
-
-When using the :thread setting, spawn will check to make sure that you have set
-allow_concurrency=true in your configuration. If you want this setting then
-put this line in one of your environment config files:
-
- config.active_record.allow_concurrency = true
-
-If it is not set, then spawn will raise an exception.
-
-To (optionally) configure the spawn method in your configuration, add a line to
-your configuration file(s) like this:
-
- Spawn::method :thread
-
-If you don't set any configuration, the :method will default to :fork. To
-specify different values for different environments, pass the environment as
-the 2nd argument:
-
- Spawn::method :fork, 'production'
- Spawn::method :yield, 'test'
-
-This allows you to set your production and development environments to use different
-methods according to your needs.
-
-Forking vs. Threading
----------------------
-
-There are several tradeoffs for using threading vs. forking. Forking was chosen as the
-default primarily because it requires no configuration to get it working out of the box.
-
-Forking advantages:
- - more reliable? - the ActiveRecord code is generally not deemed to be thread-safe.
- Even though spawn attempts to patch known problems with the threaded implementation,
- there are no guarantees. Forking is heavier but should be fairly reliable.
- - keep running - this could also be a disadvantage, but you may find you want to fork
- off a process that could have a life longer than its parent. For example, maybe you
- want to restart your server without killing the spawned processes.
- We don't necessarily condone this (i.e. haven't tried it) but it's technically possible.
- - easier - forking works out of the box with spawn, threading requires you set
- allow_concurrency=true. Also, beware of automatic reloading of classes in development
- mode (config.cache_classes = false).
-
-Threading advantages:
- - less filling - threads take less resources... how much less? it depends. Some
- flavors of Unix are pretty efficient at forking so the threading advantage may not
- be as big as you think... but then again, maybe it's more than you think. ;-)
- - debugging - you can set breakpoints in your threads
-
-Acknowledgements
-----------------
-
-This plugin was initially inspired by Scott Persinger's blog post on how to use fork
-in rails for background processing.
- http://geekblog.vodpod.com/?p=26
-
-Further inspiration for the threading implementation came from Jonathon Rochkind's
-blog post on threading in rails.
- http://bibwild.wordpress.com/2007/08/28/threading-in-rails/
-
-Also thanks to all who have helped debug problems and suggest improvements including:
- Ahmed Adam, Tristan Schneiter, Scott Haug, Andrew Garfield, Eugene Otto, Dan Sharp,
- Olivier Ruffin
- Garry Tan, Matt Jankowski (Rails 2.2.x fixes)
- Tim Kadom, Mauricio Marcon Zaffari, Danial Pearce, Hongli Lai, Scott Wadden (passenger fixes)
- Adrian Duyzer, Cyrille Labesse
- <your name here>
-
-Copyright (c) 2007-present Tom Anderson (tom@squeat.com), see LICENSE
View
193 vendor/plugins/spawn/README.markdown
@@ -0,0 +1,193 @@
+# Spawn
+
+This plugin provides a 'spawn' method to easily fork OR thread long-running sections of
+code so that your application can return results to your users more quickly.
+This plugin works by creating new database connections in ActiveRecord::Base for the
+spawned block.
+
+The plugin also patches ActiveRecord::Base to handle some known bugs when using
+threads (see lib/patches.rb).
+
+## Installation
+
+To install the plugin from the master branch (recommended).
+
+ script/plugin install git://github.com/tra/spawn.git
+
+If you want to install the plugin from the 'edge' branch (latest development):
+
+ script/plugin install git://github.com/tra/spawn.git -r edge
+
+If you are unfortunate enough to be stuck on Rails 1.x, then it is recommended you
+stick with v1.0 of this plugin (Rails 1.x won't be supported in future versions but
+it might still work if you're lucky). To install this version:
+
+ script/plugin install git://github.com/tra/spawn.git -r master:v1.0
+
+## Usage
+
+Here's a simple example of how to demonstrate the spawn plugin.
+In one of your controllers, insert this code (after installing the plugin of course):
+
+ spawn do
+ logger.info("I feel sleepy...")
+ sleep 11
+ logger.info("Time to wake up!")
+ end
+
+If everything is working correctly, your controller should finish quickly then you'll see
+the last log message several seconds later.
+
+If you need to wait for the spawned processes/threads, then pass the objects returned by
+spawn to Spawn::wait(), like this:
+
+ N.times do |i|
+ # spawn N blocks of code
+ spawn_ids[i] = spawn do
+ something(i)
+ end
+ end
+ # wait for all N blocks of code to finish running
+ wait(spawn_ids)
+
+## Options
+
+The options you can pass to spawn are:
+
+<table>
+ <tr><th>Option</th><th>Values</th></tr>
+ <tr><td>:method</td><td>:fork, :thread, :yield</td></tr>
+ <tr><td>:nice</td><td>integer value 0-19, 19 = really nice</td></tr>
+ <tr><td>:kill</td><td>boolean value indicating whether the parent should kill the spawned process
+ when it exits (only valid when :method => :fork)</td></tr>
+ <tr><td>:argv</td><td>string to override the process name</td></tr>
+</table>
+
+Any option to spawn can be set as a default so that you don't have to pass them in
+to every call of spawn. To configure the spawn default options, add a line to
+your configuration file(s) like this:
+
+ Spawn::default_options {:method => :thread}
+
+If you don't set any default options, the :method will default to :fork. To
+specify different values for different environments, add the default_options call to
+he appropriate environment file (development.rb, test.rb). For testing you can set
+the default :method to :yield so that the code is run inline.
+
+ # in environment.rb
+ Spawn::method :method => :fork, :nice => 7
+ # in test.rb, will override the environment.rb setting
+ Spawn::method :method => :yield
+
+This allows you to set your production and development environments to use different
+methods according to your needs.
+
+### be nice
+
+If you want your forked child to run at a lower priority than the parent process, pass in
+the :nice option like this:
+
+ spawn(:nice => 7) do
+ do_something_nicely
+ end
+
+### fork me
+
+By default, spawn will use the fork to spawn child processes. You can configure it to
+do threading either by telling the spawn method when you call it or by configuring your
+environment.
+For example, this is how you can tell spawn to use threading on the call,
+
+ spawn(:method => :thread) do
+ something
+ end
+
+For older versions of Rails (1.x), when using the :thread setting, spawn will check to
+make sure that you have set allow_concurrency=true in your configuration. If you
+want this setting then put this line in one of your environment config files:
+
+ config.active_record.allow_concurrency = true
+
+If it is not set, then spawn will raise an exception.
+
+### kill or be killed
+
+Depending on your application, you may want the children processes to go away when
+the parent process exits. By default spawn lets the children live after the
+parent dies. But you can tell it to kill the children by setting the :kill option
+to true.
+
+### a process by any other name
+
+If you'd like to be able to identify which processes are spawned by looking at the
+output of ps then set the :argv option with a string of your choice.
+You should then be able to see this string as the process name when
+listing the running processes (ps).
+
+For example, if you do something like this,
+
+ 3.times do |i|
+ spawn(:argv => "spawn -#{i}-") do
+ something(i)
+ end
+ end
+
+then in the shell,
+
+ $ ps -ef | grep spawn
+ 502 2645 2642 0 0:00.01 ttys002 0:00.02 spawn -0-
+ 502 2646 2642 0 0:00.02 ttys002 0:00.02 spawn -1-
+ 502 2647 2642 0 0:00.02 ttys002 0:00.03 spawn -2-
+
+The length of the process name may be limited by your OS so you might want to experiment
+to see how long it can be (it may be limited by the length of the original process name).
+
+## Forking vs. Threading
+
+There are several tradeoffs for using threading vs. forking. Forking was chosen as the
+default primarily because it requires no configuration to get it working out of the box.
+
+Forking advantages:
+
+- more reliable? - the ActiveRecord code is generally not deemed to be thread-safe.
+ Even though spawn attempts to patch known problems with the threaded implementation,
+ there are no guarantees. Forking is heavier but should be fairly reliable.
+- keep running - this could also be a disadvantage, but you may find you want to fork
+ off a process that could have a life longer than its parent. For example, maybe you
+ want to restart your server without killing the spawned processes.
+ We don't necessarily condone this (i.e. haven't tried it) but it's technically possible.
+- easier - forking works out of the box with spawn, threading requires you set
+ allow_concurrency=true (for older versions of Rails).
+ Also, beware of automatic reloading of classes in development
+ mode (config.cache_classes = false).
+
+Threading advantages:
+- less filling - threads take less resources... how much less? it depends. Some
+ flavors of Unix are pretty efficient at forking so the threading advantage may not
+ be as big as you think... but then again, maybe it's more than you think. ;-)
+- debugging - you can set breakpoints in your threads
+
+## Acknowledgements
+
+This plugin was initially inspired by Scott Persinger's blog post on how to use fork
+in rails for background processing.
+ http://geekblog.vodpod.com/?p=26
+
+Further inspiration for the threading implementation came from Jonathon Rochkind's
+blog post on threading in rails.
+ http://bibwild.wordpress.com/2007/08/28/threading-in-rails/
+
+Also thanks to all who have helped debug problems and suggest improvements
+including:
+
+- Ahmed Adam, Tristan Schneiter, Scott Haug, Andrew Garfield, Eugene Otto, Dan Sharp,
+ Olivier Ruffin, Adrian Duyzer, Cyrille Labesse
+
+- Garry Tan, Matt Jankowski (Rails 2.2.x fixes), Mina Naguib (Rails 2.3.6 fix)
+
+- Tim Kadom, Mauricio Marcon Zaffari, Danial Pearce, Hongli Lai, Scott Wadden
+ (passenger fixes)
+
+- &lt;your name here&gt;
+
+Copyright (c) 2007-present Tom Anderson (tom@squeat.com), see LICENSE
View
8 vendor/plugins/spawn/lib/patches.rb
@@ -3,9 +3,11 @@ class ActiveRecord::Base
# reconnect without disconnecting
if Spawn::RAILS_2_2
def self.spawn_reconnect(klass=self)
- # keep ancestors' connection_handlers around to avoid them being garbage collected
- (@@ancestor_connection_handlers ||= []) << @@connection_handler
- @@connection_handler = ActiveRecord::ConnectionAdapters::ConnectionHandler.new
+ # keep ancestors' connection_handlers around to avoid them being garbage collected in the forked child
+ @@ancestor_connection_handlers ||= []
+ @@ancestor_connection_handlers << self.connection_handler
+ self.connection_handler = ActiveRecord::ConnectionAdapters::ConnectionHandler.new
+
establish_connection
end
else
View
86 vendor/plugins/spawn/lib/spawn.rb
@@ -2,22 +2,47 @@ module Spawn
RAILS_1_x = (::Rails::VERSION::MAJOR == 1) unless defined?(RAILS_1_x)
RAILS_2_2 = (::Rails::VERSION::MAJOR > 2 || (::Rails::VERSION::MAJOR == 2 && ::Rails::VERSION::MINOR >= 2)) unless defined?(RAILS_2_2)
- # default to forking (unless windows or jruby)
- @@method = (RUBY_PLATFORM =~ /(win32|java)/) ? :thread : :fork
+ @@default_options = {
+ # default to forking (unless windows or jruby)
+ :method => ((RUBY_PLATFORM =~ /(win32|java)/) ? :thread : :fork),
+ :nice => nil,
+ :kill => false,
+ :argv => nil
+ }
+
# things to close in child process
@@resources = []
# in some environments, logger isn't defined
@@logger = defined?(RAILS_DEFAULT_LOGGER) ? RAILS_DEFAULT_LOGGER : Logger.new(STDERR)
+ # forked children to kill on exit
+ @@punks = []
+
+ # Set the options to use every time spawn is called unless specified
+ # otherwise. For example, in your environment, do something like
+ # this:
+ # Spawn::default_options = {:nice => 5}
+ # to default to using the :nice option with a value of 5 on every call.
+ # Valid options are:
+ # :method => (:thread | :fork | :yield)
+ # :nice => nice value of the forked process
+ # :kill => whether or not the parent process will kill the
+ # spawned child process when the parent exits
+ # :argv => changes name of the spawned process as seen in ps
+ def self.default_options(options = {})
+ @@default_options.merge!(options)
+ @@logger.info "spawn> default options = #{options.inspect}"
+ end
+ # @deprecated - please use Spawn::default_options(:method => ) instead
# add calls to this in your environment.rb to set your configuration, for example,
# to use forking everywhere except your 'development' environment:
# Spawn::method :fork
# Spawn::method :thread, 'development'
def self.method(method, env = nil)
+ @@logger.warn "spawn> please use Spawn::default_options(:method => #{method}) instead of Spawn::method"
if !env || env == RAILS_ENV
- @@method = method
+ default_options :method => method
end
- @@logger.debug "spawn> method = #{@@method}" if defined? RAILS_DEFAULT_LOGGER
end
# set the resources to disconnect from in the child process (when forking)
@@ -33,17 +58,42 @@ def self.close_resources
# in case somebody spawns recursively
@@resources.clear
end
+
+ def self.alive?(pid)
+ begin
+ Process::kill 0, pid
+ # if the process is alive then kill won't throw an exception
+ true
+ rescue Errno::ESRCH
+ false
+ end
+ end
+
+ def self.kill_punks
+ @@punks.each do |punk|
+ if alive?(punk)
+ @@logger.info "spawn> parent(#{Process.pid}) killing child(#{punk})"
+ begin
+ Process.kill("TERM", punk)
+ rescue
+ end
+ end
+ end
+ @@punks = []
+ end
+ # register to kill marked children when parent exits
+ at_exit {kill_punks}
# Spawns a long-running section of code and returns the ID of the spawned process.
# By default the process will be a forked process. To use threading, pass
# :method => :thread or override the default behavior in the environment by setting
# 'Spawn::method :thread'.
- def spawn(options = {})
- options.symbolize_keys!
- # setting options[:method] will override configured value in @@method
- if options[:method] == :yield || (options[:method] == nil && @@method == :yield)
+ def spawn(opts = {})
+ options = @@default_options.merge(opts.symbolize_keys)
+ # setting options[:method] will override configured value in default_options[:method]
+ if options[:method] == :yield
yield
- elsif options[:method] == :thread || (options[:method] == nil && @@method == :thread)
+ elsif options[:method] == :thread
# for versions before 2.2, check for allow_concurrency
if RAILS_2_2 || ActiveRecord::Base.allow_concurrency
thread_it(options) { yield }
@@ -92,6 +142,9 @@ def fork_it(options)
start = Time.now
@@logger.debug "spawn> child PID = #{Process.pid}"
+ # this child has no children of it's own to kill (yet)
+ @@punks = []
+
# set the nice priority if needed
Process.setpriority(Process::PRIO_PROCESS, 0, options[:nice]) if options[:nice]
@@ -99,6 +152,9 @@ def fork_it(options)
Spawn.close_resources
# get a new connection so the parent can keep the original one
ActiveRecord::Base.spawn_reconnect
+
+ # set the process name
+ $0 = options[:argv] if options[:argv]
# run the block of code that takes so long
yield
@@ -118,6 +174,8 @@ def fork_it(options)
@@logger.info "spawn> child[#{Process.pid}] took #{Time.now - start} sec"
# ensure log is flushed since we are using exit!
@@logger.flush if @@logger.respond_to?(:flush)
+ # this child might also have children to kill if it called spawn
+ Spawn::kill_punks
# this form of exit doesn't call at_exit handlers
exit!(0)
end
@@ -127,6 +185,15 @@ def fork_it(options)
# detach from child process (parent may still wait for detached process if they wish)
Process.detach(child)
+ # remove dead children from the target list to avoid memory leaks
+ @@punks.delete_if {|punk| !Spawn::alive?(punk)}
+
+ # mark this child for death when this process dies
+ if options[:kill]
+ @@punks << child
+ @@logger.debug "spawn> death row = #{@@punks.inspect}"
+ end
+
return SpawnId.new(:fork, child)
end
@@ -137,6 +204,7 @@ def thread_it(options)
# run the long-running code block
yield
end
+ thr.priority = -options[:nice] if options[:nice]
return SpawnId.new(:thread, thr)
end
Please sign in to comment.
Something went wrong with that request. Please try again.