Permalink
Browse files

move everything to spawn subdir to make it easy to './script/plugin i…

…nstall'

git-svn-id: svn+ssh://rubyforge.org/var/svn/spawn/spawn@2 2aabfe31-cc40-457c-aec7-21ad9ce7fb7d
  • Loading branch information...
tra
tra committed Oct 16, 2007
0 parents commit 92ffed27c9c802f7c2ff1219846bd437bab1a25b
Showing with 279 additions and 0 deletions.
  1. +21 −0 CHANGELOG
  2. +22 −0 LICENSE
  3. +100 −0 README
  4. +4 −0 init.rb
  5. +38 −0 lib/patches.rb
  6. +94 −0 lib/spawn.rb
@@ -0,0 +1,21 @@
+v0.1 - 2007/09/13
+
+initial version
+
+--------------------------------------------------
+v0.2 - 2007/09/28
+
+* return PID of the child process
+* added ":detach => false" option
+
+--------------------------------------------------
+v0.3 - 2007/10/15
+
+* added ':method => :thread' for threaded spawns
+* removed ':detach => false' option in favor of more generic implementation
+* added ability to set configuration of the form 'Spawn::method :thread'
+* added patch to ActiveRecord::Base to allow for more efficient reconnect in child processes
+* added monkey patch for http://dev.rubyonrails.org/ticket/7579
+* added wait() method to wait for spawned code blocks
+* don't allow threading if allow_concurrency=false
+
22 LICENSE
@@ -0,0 +1,22 @@
+Copyright (c) 2007 Tom Anderson (tom@squeat.com)
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
100 README
@@ -0,0 +1,100 @@
+Spawn
+=====
+
+This plugin provides a 'spawn' method to easily fork OR thread long-running sections of
+code so that your application can return results to your users more quickly.
+This plugin works by creating new database connections in ActiveRecord::Base for the
+spawned block.
+
+The plugin also patches ActiveRecord::Base to handle some known bugs when using
+threads (see lib/patches.rb).
+
+Usage
+-----
+
+Here's a simple example of how to demonstrate the spawn plugin.
+In one of your controllers, insert this code (after installing the plugin of course):
+
+ spawn do
+ logger.info("I feel sleepy...")
+ sleep 11
+ logger.info("Time to wake up!")
+ end
+
+If everything is working correctly, your controller should finish quickly then you'll see
+the last log message several seconds later.
+
+If you need to wait for the spawned processes/threads, then pass the objects returned by
+spawn to Spawn::wait(), like this:
+
+ N.times do |i|
+ # spawn N blocks of code
+ spawn_ids[i] = spawn do
+ something(i)
+ end
+ end
+ # wait for all N blocks of code to finish running
+ wait(spawn_ids)
+
+By default, spawn will use the fork to spawn child processes. You can configure it to
+do threading either by telling the spawn method when you call it or by configuring your
+environment.
+For example, this is how you can tell spawn to use threading on the call,
+
+ spawn(:method => :thread) do
+ something
+ end
+
+When using the :thread setting, spawn will check to make sure that you have set allow_concurrency=true in your configuration. If it is not set, then spawn will
+issue a warning in your log and revert to using :fork.
+
+To (optionally) configure the spawn method in your configuration, add a line to
+your configuration file(s) like this:
+
+Spawn::method :thread
+
+If you don't set any configuration, the :method will default to :fork. To specify different values for different environments, pass the environment as the 2nd
+argument:
+
+Spawn::method :fork, 'production'
+
+This allows you to set your production and development environments to use different
+methods according to your needs.
+
+Forking vs. Threading
+---------------------
+
+There are several tradeoffs for using threading vs. forking. Forking was chosen as the
+default primarily because it requires no configuration to get it working out of the box.
+
+Forking advantages:
+ - more reliable? - the ActiveRecord code is generally not deemed to be thread-safe.
+ Even though spawn attempts to patch known problems with the threaded implementation,
+ there are no guarantees. Forking is heavier but should be fairly reliable.
+ - keep running - this could also be a disadvantage, but you may find you want to fork
+ off a process that could have a life longer than its parent. For example, maybe you
+ want to restart your server without killing the spawned processes.
+ We don't necessarily condone this (i.e. haven't tried it) but it's technically possible.
+ - easier - forking works out of the box with spawn, threading requires you set.
+ allow_concurrency=true. Also, beware of automatic reloading of classes in development
+ mode (config.cache_classes = false).
+
+Threading advantages:
+ - less filling - threads take less resources... how much less? it depends. Some
+ flavors of Unix are pretty efficient at forking so the threading advantage may not
+ be as big as you think... but then again, maybe it's more than you think. ;-)
+ - debugging - you can set breakpoints in your threads
+
+Acknowledgement
+---------------
+
+This plugin was initially inspired by Scott Persinger's blog post on how to use fork
+in rails for background processing.
+ http://geekblog.vodpod.com/?p=26
+
+Further inspiration for the threading implementation came from Jonathon Rochkind's
+blog post on threading in rails.
+ http://bibwild.wordpress.com/2007/08/28/threading-in-rails/
+
+
+Copyright (c) 2007 Tom Anderson (tom@squeat.com), see LICENSE
@@ -0,0 +1,4 @@
+require 'patches'
+
+ActiveRecord::Base.send :include, Spawn
+ActionController::Base.send :include, Spawn
@@ -0,0 +1,38 @@
+class ActiveRecord::Base
+
+ # method to allow a child process to establish a new connection while letting
+ # the parent retain the original connection
+ def self.reconnect_in_child(klass=self)
+ spec = @@defined_connections[klass.name]
+ konn = active_connections[klass.name]
+ @@defined_connections.delete_if { |key, value| value == spec }
+ active_connections.delete_if { |key, value| value == konn }
+ establish_connection(spec ? spec.config : nil)
+ end
+
+ # monkey patch to fix threading problems,
+ # see: http://dev.rubyonrails.org/ticket/7579
+ def self.clear_reloadable_connections!
+ puts "spawn's version of clear_reloadable_connections!"
+ if @@allow_concurrency
+ # Hash keyed by thread_id in @@active_connections. Hash of hashes.
+ @@active_connections.each do |thread_id, conns|
+ conns.each do |name, conn|
+ if conn.requires_reloading?
+ conn.disconnect!
+ @@active_connections[thread_id].delete(name)
+ end
+ end
+ end
+ else
+ # Just one level hash, no concurrency.
+ @@active_connections.each do |name, conn|
+ if conn.requires_reloading?
+ conn.disconnect!
+ @@active_connections.delete(name)
+ end
+ end
+ end
+ end
+
+end
@@ -0,0 +1,94 @@
+module Spawn
+
+ # default to forking
+ @@method = :fork
+
+ # add calls to this in your environment.rb to set your configuration, for example,
+ # to use forking everywhere except your 'development' environment:
+ # Spawn::method :fork
+ # Spawn::method :thread, 'development'
+ def self.method(method, env = nil)
+ if !env || env == RAILS_ENV
+ @@method = method
+ end
+ end
+
+ # Spawns a long-running section of code and returns the ID of the spawned process.
+ # By default the process will be a forked process. To use threading, pass
+ # :method => :thread or override the default behavior in the environment by setting
+ # 'Spawn::method :thread'.
+ def spawn(options = {})
+ options.symbolize_keys!
+ # setting options[:method] will override configured value in @@method
+ if options[:method] == :thread || (options[:method] == nil && @@method == :thread)
+ if ActiveRecord::Base.allow_concurrency
+ thread_it(options) { yield }
+ else
+ logger.warn("spawn(:method=>:thread) only allowed when allow_concurrency=true, reverting to :method=>:fork")
+ fork_it(options) { yield }
+ end
+ else
+ fork_it(options) { yield }
+ end
+ end
+
+ def wait(sids = [])
+ # wait for all threads and/or forks
+ sids.to_a.each do |sid|
+ if sid.type == :thread
+ sid.handle.join()
+ else
+ begin
+ Process.wait(sid.handle)
+ rescue
+ # if the process is already done, ignore the error
+ end
+ end
+ end
+ # clean up connections from expired threads
+ ActiveRecord::Base.verify_active_connections!()
+ end
+
+ class SpawnId
+ attr_accessor :type
+ attr_accessor :handle
+ def initialize(t, h)
+ self.type = t
+ self.handle = h
+ end
+ end
+
+ protected
+ def fork_it(options)
+ # The problem with rails is that it only has one connection (per class),
+ # so when we fork a new process, we need to reconnect.
+ child = fork do
+ # call the method we added in patches.rb to allow the child to get a new connection
+ # without messing with the parent's connection
+ ActiveRecord::Base.reconnect_in_child
+ begin
+ # run the block of code that takes so long
+ yield
+ ensure
+ ActiveRecord::Base.remove_connection
+ end
+ # this form of exit doesn't call at_exit handlers
+ exit!
+ end
+ # detach from child process (parent may still wait for detached process if they wish)
+ Process.detach(child)
+ # reconnect in the parent process
+ return SpawnId.new(:fork, child)
+ end
+
+ def thread_it(options)
+ # clean up stale connections from previous threads
+ ActiveRecord::Base.verify_active_connections!()
+ thr = Thread.new do
+ # run the long-running code block
+ yield
+ end
+ return SpawnId.new(:thread, thr)
+ end
+
+end

0 comments on commit 92ffed2

Please sign in to comment.