Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

move everything to spawn subdir to make it easy to './script/plugin i…

…nstall'

git-svn-id: svn+ssh://rubyforge.org/var/svn/spawn/spawn@2 2aabfe31-cc40-457c-aec7-21ad9ce7fb7d
  • Loading branch information...
commit 92ffed27c9c802f7c2ff1219846bd437bab1a25b 0 parents
tra authored
21 CHANGELOG
@@ -0,0 +1,21 @@
+v0.1 - 2007/09/13
+
+initial version
+
+--------------------------------------------------
+v0.2 - 2007/09/28
+
+* return PID of the child process
+* added ":detach => false" option
+
+--------------------------------------------------
+v0.3 - 2007/10/15
+
+* added ':method => :thread' for threaded spawns
+* removed ':detach => false' option in favor of more generic implementation
+* added ability to set configuration of the form 'Spawn::method :thread'
+* added patch to ActiveRecord::Base to allow for more efficient reconnect in child processes
+* added monkey patch for http://dev.rubyonrails.org/ticket/7579
+* added wait() method to wait for spawned code blocks
+* don't allow threading if allow_concurrency=false
+
22 LICENSE
@@ -0,0 +1,22 @@
+Copyright (c) 2007 Tom Anderson (tom@squeat.com)
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
100 README
@@ -0,0 +1,100 @@
+Spawn
+=====
+
+This plugin provides a 'spawn' method to easily fork OR thread long-running sections of
+code so that your application can return results to your users more quickly.
+This plugin works by creating new database connections in ActiveRecord::Base for the
+spawned block.
+
+The plugin also patches ActiveRecord::Base to handle some known bugs when using
+threads (see lib/patches.rb).
+
+Usage
+-----
+
+Here's a simple example of how to demonstrate the spawn plugin.
+In one of your controllers, insert this code (after installing the plugin of course):
+
+ spawn do
+ logger.info("I feel sleepy...")
+ sleep 11
+ logger.info("Time to wake up!")
+ end
+
+If everything is working correctly, your controller should finish quickly then you'll see
+the last log message several seconds later.
+
+If you need to wait for the spawned processes/threads, then pass the objects returned by
+spawn to Spawn::wait(), like this:
+
+ N.times do |i|
+ # spawn N blocks of code
+ spawn_ids[i] = spawn do
+ something(i)
+ end
+ end
+ # wait for all N blocks of code to finish running
+ wait(spawn_ids)
+
+By default, spawn will use the fork to spawn child processes. You can configure it to
+do threading either by telling the spawn method when you call it or by configuring your
+environment.
+For example, this is how you can tell spawn to use threading on the call,
+
+ spawn(:method => :thread) do
+ something
+ end
+
+When using the :thread setting, spawn will check to make sure that you have set allow_concurrency=true in your configuration. If it is not set, then spawn will
+issue a warning in your log and revert to using :fork.
+
+To (optionally) configure the spawn method in your configuration, add a line to
+your configuration file(s) like this:
+
+Spawn::method :thread
+
+If you don't set any configuration, the :method will default to :fork. To specify different values for different environments, pass the environment as the 2nd
+argument:
+
+Spawn::method :fork, 'production'
+
+This allows you to set your production and development environments to use different
+methods according to your needs.
+
+Forking vs. Threading
+---------------------
+
+There are several tradeoffs for using threading vs. forking. Forking was chosen as the
+default primarily because it requires no configuration to get it working out of the box.
+
+Forking advantages:
+ - more reliable? - the ActiveRecord code is generally not deemed to be thread-safe.
+ Even though spawn attempts to patch known problems with the threaded implementation,
+ there are no guarantees. Forking is heavier but should be fairly reliable.
+ - keep running - this could also be a disadvantage, but you may find you want to fork
+ off a process that could have a life longer than its parent. For example, maybe you
+ want to restart your server without killing the spawned processes.
+ We don't necessarily condone this (i.e. haven't tried it) but it's technically possible.
+ - easier - forking works out of the box with spawn, threading requires you set.
+ allow_concurrency=true. Also, beware of automatic reloading of classes in development
+ mode (config.cache_classes = false).
+
+Threading advantages:
+ - less filling - threads take less resources... how much less? it depends. Some
+ flavors of Unix are pretty efficient at forking so the threading advantage may not
+ be as big as you think... but then again, maybe it's more than you think. ;-)
+ - debugging - you can set breakpoints in your threads
+
+Acknowledgement
+---------------
+
+This plugin was initially inspired by Scott Persinger's blog post on how to use fork
+in rails for background processing.
+ http://geekblog.vodpod.com/?p=26
+
+Further inspiration for the threading implementation came from Jonathon Rochkind's
+blog post on threading in rails.
+ http://bibwild.wordpress.com/2007/08/28/threading-in-rails/
+
+
+Copyright (c) 2007 Tom Anderson (tom@squeat.com), see LICENSE
4 init.rb
@@ -0,0 +1,4 @@
+require 'patches'
+
+ActiveRecord::Base.send :include, Spawn
+ActionController::Base.send :include, Spawn
38 lib/patches.rb
@@ -0,0 +1,38 @@
+class ActiveRecord::Base
+
+ # method to allow a child process to establish a new connection while letting
+ # the parent retain the original connection
+ def self.reconnect_in_child(klass=self)
+ spec = @@defined_connections[klass.name]
+ konn = active_connections[klass.name]
+ @@defined_connections.delete_if { |key, value| value == spec }
+ active_connections.delete_if { |key, value| value == konn }
+ establish_connection(spec ? spec.config : nil)
+ end
+
+ # monkey patch to fix threading problems,
+ # see: http://dev.rubyonrails.org/ticket/7579
+ def self.clear_reloadable_connections!
+ puts "spawn's version of clear_reloadable_connections!"
+ if @@allow_concurrency
+ # Hash keyed by thread_id in @@active_connections. Hash of hashes.
+ @@active_connections.each do |thread_id, conns|
+ conns.each do |name, conn|
+ if conn.requires_reloading?
+ conn.disconnect!
+ @@active_connections[thread_id].delete(name)
+ end
+ end
+ end
+ else
+ # Just one level hash, no concurrency.
+ @@active_connections.each do |name, conn|
+ if conn.requires_reloading?
+ conn.disconnect!
+ @@active_connections.delete(name)
+ end
+ end
+ end
+ end
+
+end
94 lib/spawn.rb
@@ -0,0 +1,94 @@
+module Spawn
+
+ # default to forking
+ @@method = :fork
+
+ # add calls to this in your environment.rb to set your configuration, for example,
+ # to use forking everywhere except your 'development' environment:
+ # Spawn::method :fork
+ # Spawn::method :thread, 'development'
+ def self.method(method, env = nil)
+ if !env || env == RAILS_ENV
+ @@method = method
+ end
+ end
+
+ # Spawns a long-running section of code and returns the ID of the spawned process.
+ # By default the process will be a forked process. To use threading, pass
+ # :method => :thread or override the default behavior in the environment by setting
+ # 'Spawn::method :thread'.
+ def spawn(options = {})
+ options.symbolize_keys!
+ # setting options[:method] will override configured value in @@method
+ if options[:method] == :thread || (options[:method] == nil && @@method == :thread)
+ if ActiveRecord::Base.allow_concurrency
+ thread_it(options) { yield }
+ else
+ logger.warn("spawn(:method=>:thread) only allowed when allow_concurrency=true, reverting to :method=>:fork")
+ fork_it(options) { yield }
+ end
+ else
+ fork_it(options) { yield }
+ end
+ end
+
+ def wait(sids = [])
+ # wait for all threads and/or forks
+ sids.to_a.each do |sid|
+ if sid.type == :thread
+ sid.handle.join()
+ else
+ begin
+ Process.wait(sid.handle)
+ rescue
+ # if the process is already done, ignore the error
+ end
+ end
+ end
+ # clean up connections from expired threads
+ ActiveRecord::Base.verify_active_connections!()
+ end
+
+ class SpawnId
+ attr_accessor :type
+ attr_accessor :handle
+ def initialize(t, h)
+ self.type = t
+ self.handle = h
+ end
+ end
+
+ protected
+ def fork_it(options)
+ # The problem with rails is that it only has one connection (per class),
+ # so when we fork a new process, we need to reconnect.
+ child = fork do
+ # call the method we added in patches.rb to allow the child to get a new connection
+ # without messing with the parent's connection
+ ActiveRecord::Base.reconnect_in_child
+ begin
+ # run the block of code that takes so long
+ yield
+ ensure
+ ActiveRecord::Base.remove_connection
+ end
+ # this form of exit doesn't call at_exit handlers
+ exit!
+ end
+ # detach from child process (parent may still wait for detached process if they wish)
+ Process.detach(child)
+ # reconnect in the parent process
+ return SpawnId.new(:fork, child)
+ end
+
+ def thread_it(options)
+ # clean up stale connections from previous threads
+ ActiveRecord::Base.verify_active_connections!()
+ thr = Thread.new do
+ # run the long-running code block
+ yield
+ end
+ return SpawnId.new(:thread, thr)
+ end
+
+end
Please sign in to comment.
Something went wrong with that request. Please try again.